#pytorch

Maths for LLMs

Start with one input, one target, and one adjustable weight. Calculate a derivative by hand, apply a gradient-descent update, and verify every number in PyTorch.

• #deep-learning
• #gradient-descent
• #gradients
• #maths-for-llms
• #pytorch

LLMs, the whole thing · Part 11

Chapter 5 connects next-token loss to real parameter updates, then separates training from decoding. My notes on batches, perplexity, validation, sampling, checkpoints, and pretrained GPT-2 weights.

• #gpt
• #llm
• #llms-from-scratch
• #pretraining
• #pytorch

Maths for LLMs

Cross-entropy looked difficult to me. Let’s unpack −ln(p) from one correct token, understand every symbol, and verify the maths in PyTorch.

• #cross-entropy
• #llms
• #maths-for-llms
• #pytorch

LLMs, the whole thing · Part 10

Chapter 4 is the moment attention becomes a complete GPT. I build the model one inspectable component at a time, count its parameters, and trace the gloriously untrained first token.

• #deep-learning
• #gpt
• #llm
• #pytorch
• #transformers

Maths for LLMs

Softmax is a small equation with a central job: it turns arbitrary model scores into positive weights that sum to one. Here is the arithmetic, the numerical-stability trick, and where the same operation appears inside an LLM.

• #attention
• #maths-for-llms
• #probability
• #pytorch
• #softmax

Maths for LLMs

ReLU cuts every negative activation to zero. GELU computes a smooth tanh-shaped gate; I unpack the formula one real number at a time and place it inside GPT's feed-forward network.

• #deep-learning
• #gelu
• #gpt
• #maths-for-llms
• #pytorch
• #relu

Maths for LLMs

I refused to let LayerNorm remain a formula. Starting with four integers, I work through mean, variance, standard deviation, normalization and the exact PyTorch operation used for each GPT token.

• #deep-learning
• #layer-normalization
• #maths-for-llms
• #pytorch

From Java to Python

Before I could make sense of attention code, I needed PyTorch's tiny tensor vocabulary to stop feeling mysterious. This first session covers shapes, axes, batching, and the important difference between stack, cat, multiplication, and dot products.

• #java-to-python
• #python
• #pytorch
• #tensors

LLMs, the whole thing · Part 9

Part 9 of LLMs, the whole thing: two heads, one token, and the difference between computing attention and organizing its tensors. A concrete PyTorch walkthrough of view, transpose, concatenation and the output projection.

• #attention
• #deep-learning
• #llm
• #multi-head-attention
• #pytorch
• #transformers

LLMs, the whole thing · Part 8

Section 3.5 of Raschka's book on seven real GPT-2 tokens: why masking after softmax breaks the row sums, why -inf on the scores fixes it in one pass, and how dropout is a second, random mask that doubles its survivors.

• #attention
• #causal-attention
• #deep-learning
• #dropout
• #llm
• #pytorch
• #transformers