Tag
LLMs, the whole thing · Part 7
Section 3.4 of Raschka's book on six tokens with real PyTorch output: three shared learned matrices, eighteen private q/k/v vectors, why d_in is forced, why scores are divided by the square root of d_k, and why 'weights' now means two different things.
The positional-embedding sum quietly relies on broadcasting. Here is the whole rule — align from the right, stretch the 1s — and why PyTorch never copies a byte.
LLMs, the whole thing · Part 5
The finale of chapter 2: how meaningless token IDs become the tensor attention actually consumes. It comes down to two lookup tables and one addition - a learnable embedding table for meaning, and a second, position-indexed table for order.
LLMs, the whole thing · Part 4
I printed my first training batch, saw shape [8, 4], and could not say what the two numbers counted. Untangling it - one short story sliced into 1,286 windows, grabbed 8 at a time - made the whole GPT input pipeline finally go transparent.