Series
Working through large language models from the ground up — turning text into tokens, embeddings and the geometry of meaning, the transformer architecture, attention, how these models are trained, and what separates one generation from the next.
Part 1
The opening part of a series building LLMs from first principles. We start where every model starts: turning language into vectors and meaning into geometry, with a runnable script and its real numbers.
Part 2
Every modern LLM is a transformer, yet BERT, GPT and T5 behave very differently. The first article in a ground-up series on LLMs: the single distinction — the attention mask — that separates an encoder from a decoder.
Part 3
An LLM never sees your text — it sees integers. Building a tokenizer from a regex and a dictionary, hitting the out-of-vocabulary wall, and why GPT’s byte-pair encoding is the fix that needs no unknown token.
Part 4
I printed my first training batch, saw shape [8, 4], and could not say what the two numbers counted. Untangling it - one short story sliced into 1,286 windows, grabbed 8 at a time - made the whole GPT input pipeline finally go transparent.
Part 5
The finale of chapter 2: how meaningless token IDs become the tensor attention actually consumes. It comes down to two lookup tables and one addition - a learnable embedding table for meaning, and a second, position-indexed table for order.
Part 6
The first attention mechanism in Raschka's LLM book, explained the way it finally clicked for me, a Java dev who bounces off dense maths: self-attention is just three operations, a dot product, a softmax, and a weighted average.
Next in this series: Trainable self-attention: query, key and value — coming soon.