Broadcasting: how a [4,256] table adds to a whole batch
The positional-embedding sum quietly relies on broadcasting. Here is the whole rule — align from the right, stretch the 1s — and why PyTorch never copies a byte.
Topic
Maths for LLMs
Understand the mathematics under large language models, built from zero — words as coordinates, the angle that measures meaning, the dot-product shortcut that computes it, and the contrastive objective that shapes the whole space.
Positional embeddings and vector addition
The maths under positional embeddings, from zero: a token becomes what it is plus where it sits, and that 'plus' is a literal element-wise sum of two vectors. Why the differently-sized tables never actually clash, worked by hand, with slides.
Cosine similarity and the dot product
A from-zero tour of the maths under large language models: how words become coordinates, why similarity is an angle, and the single multiply-and-add a computer uses to compare two meanings.