Tag

#pretraining

#agents
#aggregation
#ai
#ai-agents
#ai-architecture
#ai-tooling
#anthropic-skills
#appsec
#architecture
#artifacts
#attention
#automation
#banking
#batch
#bean-validation
#bert
#binding
#blackboard
#branching
#branding
#broadcasting
#calibration
#case-classes
#causal-attention
#cgroups
#chatgpt
#chatgpt-sites
#ci-cd
#classpath
#claude
#claude-code
#cli
#codex
#collections
#configuration
#cosine-similarity
#cross-entropy
#csp
#data-modeling
#dataviz
#daybreak
#ddd
#deep-learning
#deepseek
#design-system
#design-systems
#developer-experience
#developer-tooling
#developer-tools
#devex
#devops
#dns
#docker
#dropout
#email
#embabel
#embeddings
#events
#fable
#fail2ban
#fairness
#fastapi
#fastmail
#fine-tuning
#flyway
#for-comprehensions
#from-java-to-python
#frontend
#functional-programming
#fundamentals
#gdpr
#gelu
#git
#github-actions
#goap
#gpt
#grafana
#guardrails
#hibernate-validator
#honeypot
#hooks
#http
#human-in-the-loop
#ideas
#identity-brokering
#image-generation
#intellij
#java
#java-to-python
#jev
#jinja2
#junit
#jvm
#kafka
#keycloak
#knowledge-base
#layer-normalization
#learning
#linkedin
#linux
#llm
#llms
#llms-from-scratch
#llms-txt
#loki
#loops
#lora
#machine-learning
#maths-for-llms
#maven
#mcp
#mockito
#mongodb
#multi-agent
#multi-head-attention
#networking
#nlp
#oauth
#oauth2
#observability
#oidc
#okf
#ollama
#openkb
#opentelemetry
#ops
#option
#otel
#outbox
#parking
#pattern-matching
#patterns
#pgvector
#play-framework
#positional-encoding
#postgres
#pretraining
#print-css
#privacy
#probability
#prompt-audit
#prompt-injection
#pydantic
#python
#pytorch
#rag
#react
#refactoring
#relu
#rocm
#sbt
#scala
#scala-play
#search-console
#secrets
#security
#self-attention
#seo
#shipping
#skills
#softmax
#spring
#spring-ai
#spring-batch
#spring-boot
#spring-data
#spring-data-jdbc
#sse
#ssh
#state
#svg
#systemd
#templating
#tempo
#tensors
#testing
#tokenization
#tools
#traceql
#tracing
#traefik
#training-data
#transactions
#transformers
#tts
#typescript
#unsloth
#uv
#vectors
#vite
#waitfor
#web-design
#web-speech-api
#workflow
#worktree

Training GPT from Scratch: Loss, Learning, and the Next Token

Chapter 5 connects next-token loss to real parameter updates, then separates training from decoding. My notes on batches, perplexity, validation, sampling, checkpoints, and pretrained GPT-2 weights.

20 Sep 2026 · 6 min read

#llms-from-scratch
#pytorch
#gpt
#llm
#pretraining