Series
Working through large language models from the ground up — turning text into tokens, embeddings and the geometry of meaning, the transformer architecture, attention, how these models are trained, and what separates one generation from the next.
Part 1
The opening part of a series building LLMs from first principles. We start where every model starts: turning language into vectors and meaning into geometry, with a runnable script and its real numbers.
Part 2
Every modern LLM is a transformer, yet BERT, GPT and T5 behave very differently. The first article in a ground-up series on LLMs: the single distinction — the attention mask — that separates an encoder from a decoder.
Part 3
An LLM never sees your text — it sees integers. Building a tokenizer from a regex and a dictionary, hitting the out-of-vocabulary wall, and why GPT’s byte-pair encoding is the fix that needs no unknown token.
Part 4
I printed my first training batch, saw shape [8, 4], and could not say what the two numbers counted. Untangling it - one short story sliced into 1,286 windows, grabbed 8 at a time - made the whole GPT input pipeline finally go transparent.
Part 5
The finale of chapter 2: how meaningless token IDs become the tensor attention actually consumes. It comes down to two lookup tables and one addition - a learnable embedding table for meaning, and a second, position-indexed table for order.
Part 6
The first attention mechanism in Raschka's LLM book, explained the way it finally clicked for me, a Java dev who bounces off dense maths: self-attention is just three operations, a dot product, a softmax, and a weighted average.
Part 7
Section 3.4 of Raschka's book on six tokens with real PyTorch output: three shared learned matrices, eighteen private q/k/v vectors, why d_in is forced, why scores are divided by the square root of d_k, and why 'weights' now means two different things.
Part 8
Section 3.5 of Raschka's book on seven real GPT-2 tokens: why masking after softmax breaks the row sums, why -inf on the scores fixes it in one pass, and how dropout is a second, random mask that doubles its survivors.
Part 9
Part 9 of LLMs, the whole thing: two heads, one token, and the difference between computing attention and organizing its tensors. A concrete PyTorch walkthrough of view, transpose, concatenation and the output projection.
Part 10
Chapter 4 is the moment attention becomes a complete GPT. I build the model one inspectable component at a time, count its parameters, and trace the gloriously untrained first token.
Part 11
Chapter 5 connects next-token loss to real parameter updates, then separates training from decoding. My notes on batches, perplexity, validation, sampling, checkpoints, and pretrained GPT-2 weights.