How One Weight Learns: Derivatives, Gradients, and the First Update
Start with one input, one target, and one adjustable weight. Calculate a derivative by hand, apply a gradient-descent update, and verify every number in PyTorch.
Topic
Maths for LLMs
Understand the mathematics under large language models, built from zero — words as coordinates, the angle that measures meaning, the dot-product shortcut that computes it, and the contrastive objective that shapes the whole space.
Cross-Entropy: Understanding the Minus, the Log, and the Loss
Cross-entropy looked difficult to me. Let’s unpack −ln(p) from one correct token, understand every symbol, and verify the maths in PyTorch.
Softmax: How LLM Scores Become Probabilities
Softmax is a small equation with a central job: it turns arbitrary model scores into positive weights that sum to one. Here is the arithmetic, the numerical-stability trick, and where the same operation appears inside an LLM.
GELU vs ReLU: The Smooth Gate Inside GPT
ReLU cuts every negative activation to zero. GELU computes a smooth tanh-shaped gate; I unpack the formula one real number at a time and place it inside GPT's feed-forward network.
LayerNorm from Four Numbers: Mean, Variance and a Token Row
I refused to let LayerNorm remain a formula. Starting with four integers, I work through mean, variance, standard deviation, normalization and the exact PyTorch operation used for each GPT token.
Broadcasting: how a [4,256] table adds to a whole batch
The positional-embedding sum quietly relies on broadcasting. Here is the whole rule — align from the right, stretch the 1s — and why PyTorch never copies a byte.
Positional embeddings and vector addition
The maths under positional embeddings, from zero: a token becomes what it is plus where it sits, and that 'plus' is a literal element-wise sum of two vectors. Why the differently-sized tables never actually clash, worked by hand, with slides.
Cosine similarity and the dot product
A from-zero tour of the maths under large language models: how words become coordinates, why similarity is an angle, and the single multiply-and-add a computer uses to compare two meanings.