#pytorch

Maths for LLMs

The positional-embedding sum quietly relies on broadcasting. Here is the whole rule — align from the right, stretch the 1s — and why PyTorch never copies a byte.

• #embeddings
• #pytorch
• #tensors
• #maths-for-llms
• #broadcasting

LLMs, the whole thing · Part 5

The finale of chapter 2: how meaningless token IDs become the tensor attention actually consumes. It comes down to two lookup tables and one addition - a learnable embedding table for meaning, and a second, position-indexed table for order.

• #embeddings
• #llms
• #pytorch
• #positional-encoding

LLMs, the whole thing · Part 4

I printed my first training batch, saw shape [8, 4], and could not say what the two numbers counted. Untangling it - one short story sliced into 1,286 windows, grabbed 8 at a time - made the whole GPT input pipeline finally go transparent.

• #training-data
• #llms
• #pytorch
• #tokenization