#llms

Maths for LLMs

Cross-entropy looked difficult to me. Let’s unpack −ln(p) from one correct token, understand every symbol, and verify the maths in PyTorch.

• #cross-entropy
• #llms
• #maths-for-llms
• #pytorch

LLMs, the whole thing · Part 5

The finale of chapter 2: how meaningless token IDs become the tensor attention actually consumes. It comes down to two lookup tables and one addition - a learnable embedding table for meaning, and a second, position-indexed table for order.

• #embeddings
• #llms
• #positional-encoding
• #pytorch

LLMs, the whole thing · Part 4

I printed my first training batch, saw shape [8, 4], and could not say what the two numbers counted. Untangling it - one short story sliced into 1,286 windows, grabbed 8 at a time - made the whole GPT input pipeline finally go transparent.

• #llms
• #pytorch
• #tokenization
• #training-data