Tag
Softmax is a small equation with a central job: it turns arbitrary model scores into positive weights that sum to one. Here is the arithmetic, the numerical-stability trick, and where the same operation appears inside an LLM.
LLMs, the whole thing · Part 9
Part 9 of LLMs, the whole thing: two heads, one token, and the difference between computing attention and organizing its tensors. A concrete PyTorch walkthrough of view, transpose, concatenation and the output projection.
LLMs, the whole thing · Part 8
Section 3.5 of Raschka's book on seven real GPT-2 tokens: why masking after softmax breaks the row sums, why -inf on the scores fixes it in one pass, and how dropout is a second, random mask that doubles its survivors.
LLMs, the whole thing · Part 7
Section 3.4 of Raschka's book on six tokens with real PyTorch output: three shared learned matrices, eighteen private q/k/v vectors, why d_in is forced, why scores are divided by the square root of d_k, and why 'weights' now means two different things.
LLMs, the whole thing · Part 6
The first attention mechanism in Raschka's LLM book, explained the way it finally clicked for me, a Java dev who bounces off dense maths: self-attention is just three operations, a dot product, a softmax, and a weighted average.