#self-attention

LLMs, the whole thing · Part 7

Section 3.4 of Raschka's book on six tokens with real PyTorch output: three shared learned matrices, eighteen private q/k/v vectors, why d_in is forced, why scores are divided by the square root of d_k, and why 'weights' now means two different things.

• #attention
• #deep-learning
• #llm
• #pytorch
• #self-attention
• #transformers

LLMs, the whole thing · Part 6

The first attention mechanism in Raschka's LLM book, explained the way it finally clicked for me, a Java dev who bounces off dense maths: self-attention is just three operations, a dot product, a softmax, and a weighted average.

• #attention
• #deep-learning
• #llm
• #self-attention
• #transformers