#positional-encoding

Maths for LLMs

The maths under positional embeddings, from zero: a token becomes what it is plus where it sits, and that 'plus' is a literal element-wise sum of two vectors. Why the differently-sized tables never actually clash, worked by hand, with slides.

• #embeddings
• #maths-for-llms
• #positional-encoding
• #vectors

LLMs, the whole thing · Part 5

The finale of chapter 2: how meaningless token IDs become the tensor attention actually consumes. It comes down to two lookup tables and one addition - a learnable embedding table for meaning, and a second, position-indexed table for order.

• #embeddings
• #llms
• #positional-encoding
• #pytorch