Tag
LLMs, the whole thing · Part 11
Chapter 5 connects next-token loss to real parameter updates, then separates training from decoding. My notes on batches, perplexity, validation, sampling, checkpoints, and pretrained GPT-2 weights.
LLMs, the whole thing · Part 10
Chapter 4 is the moment attention becomes a complete GPT. I build the model one inspectable component at a time, count its parameters, and trace the gloriously untrained first token.
LLMs, the whole thing · Part 9
Part 9 of LLMs, the whole thing: two heads, one token, and the difference between computing attention and organizing its tensors. A concrete PyTorch walkthrough of view, transpose, concatenation and the output projection.
LLMs, the whole thing · Part 8
Section 3.5 of Raschka's book on seven real GPT-2 tokens: why masking after softmax breaks the row sums, why -inf on the scores fixes it in one pass, and how dropout is a second, random mask that doubles its survivors.
LLMs, the whole thing · Part 7
Section 3.4 of Raschka's book on six tokens with real PyTorch output: three shared learned matrices, eighteen private q/k/v vectors, why d_in is forced, why scores are divided by the square root of d_k, and why 'weights' now means two different things.
Most 'chat with your docs' tools retrieve fragments at query time. OpenKB compiles the synthesis once, at ingest - into a Markdown wiki you can read, diff, and ship as an installable Claude Code skill.
I ran a full LoRA fine-tune of Gemma 3 270M on my AMD Strix Halo iGPU. It froze the whole machine twice before three lines of config fixed it — here's the run, the fix, and what it taught a backend dev about how LLMs actually work.
LLMs, the whole thing · Part 6
The first attention mechanism in Raschka's LLM book, explained the way it finally clicked for me, a Java dev who bounces off dense maths: self-attention is just three operations, a dot product, a softmax, and a weighted average.
LLMs, the whole thing · Part 3
An LLM never sees your text — it sees integers. Building a tokenizer from a regex and a dictionary, hitting the out-of-vocabulary wall, and why GPT’s byte-pair encoding is the fix that needs no unknown token.
I built a local instrument to look at embedding space, and most of what I assumed turned out to be wrong: nothing is orthogonal, no individual dimension means anything, and king minus man plus woman does not give you queen.