Positional embeddings and vector addition

The whole thing is one addition

I lost the better part of a day to positional embeddings. Not because the maths is hard — because everything is called “embedding,” and the shapes look like they should clash. When it finally clicked, the operation underneath turned out to be almost insultingly simple: you add two vectors. A token becomes what it is (its meaning vector) plus where it sits (a position vector). That’s the entire trick — a sum.

This is the maths under that sum, built from zero, with the slides I drew to teach it. It’s the intuition companion to the code-first walkthrough in The Lookup Table That Learns; here I only care about one question — why does adding a vector give a model a sense of order?

The problem: a lookup table is order-blind

An embedding is a pure lookup. The row you get back depends on the token id and nothing else — so the same id always returns the same vector, wherever it appears.

To a model with only token embeddings, “the cat sat on the mat” and “mat the on sat cat the” are the same bag of vectors. Word order carries meaning — “dog bites man” is not “man bites dog” — so this has to be fixed before attention ever runs.

The fix: a second vector, added on

Ask two independent questions about every token. What word are you? — answered by the token embedding, a lookup in the big vocabulary table. Where do you sit? — answered by a second, separate embedding. Then add the two vectors together. Everything else is the mechanics of that add.

The mental model that unstuck me: chairs and stickers. A window has a fixed number of seats; each seat carries a sticker — a vector — and the sticker belongs to the seat, not to whoever sits in it.

Why the sizes never actually clash

Here’s where I got stuck for hours. The two tables are wildly different sizes — the word table is [50257, 256], the position table is [4, 256] — and I couldn’t see how you add them. The answer is that you never add the tables. They’re just dictionaries on a shelf. You add the rows you look up.

The two numbers come from opposite places. The 4 is the window length (context_length = max_length): four tokens means four seats. The 256 is output_dim — and it isn’t chosen for the position’s sake at all. It’s forced to equal the token vector’s width, because you can only add two vectors element by element if they’re the same length. One number is your window; the other is dictated by the word vector.

The worked example: same word, different seat

This is the payoff, done by hand. Take the window "the cat the mat", with “the” in slot 0 and slot 2. Shrink the vectors to size 3 so every number is visible (the real width is 256, same logic).

Before the addition, the two “the”s are byte-for-byte identical. After adding each seat’s vector, they differ — purely because slot 0 contributed slot 0’s vector and slot 2 contributed slot 2’s. Nothing about the word changed; the seat broke the tie.

One sum, across the whole batch

Scale it up and one detail makes the shapes click. The token embeddings for a batch are [8, 4, 256] (8 windows), but the position table is only [4, 256] — no batch dimension. The addition still just works, by broadcasting.

Why a sum and not a concatenation?

A fair question from a datasheet mindset: why add position instead of tacking it on as extra dimensions? Concatenation would grow every token from 256 to 256-plus-however-many, ballooning every downstream matrix. Addition keeps the width at 256 and lets training decide how to share those 256 directions between “meaning” and “position” — the model learns to keep them separable enough to be useful. Cheaper, and it works. Both vectors are learned, so the network is free to carve out the space however it likes.

So the maths under positional embeddings is one line: input = token_vector + position_vector. A token is what it is, plus where it sits. Do that for all 8×4 tokens and you land on the [8, 4, 256] tensor that attention consumes next — the end of the input pipeline, and the start of the interesting part.