LayerNorm from Four Numbers: Mean, Variance and a Token Row

The activation row that arrived too early

While working through Chapter 4 of Build a Large Language Model (From Scratch), I hit a useful teaching failure. An activation row full of decimals appeared before I understood why it existed. My reaction was immediate: what is this? I had not learned mean, variance or LayerNorm yet.

The right move was not another formula. It was to throw that row away and restart with four numbers I could calculate by hand:

x = [2, 2, 6, 6]

This is the route from those four numbers to LayerNorm, with every operation left visible.

Mean finds the center

The mean is the ordinary average. Add the values and divide by how many values there are:

mean = (2 + 2 + 6 + 6) / 4
# 4

The mean tells me where this row is centered. I can now subtract that center from every value:

centered = [2 - 4, 2 - 4, 6 - 4, 6 - 4]
# [-2, -2, 2, 2]

The new row has mean zero. Nothing has been learned and no parameter has changed. These values are computed from the current input during the forward pass.

Variance measures the spread

Centering is only half the job. Two rows can both have mean zero while having very different spreads. Variance gives me a number for that spread.

I square each centered value so negative and positive distances do not cancel, then average the squares:

squared = [4, 4, 4, 4]
variance = (4 + 4 + 4 + 4) / 4
# 4

For the LayerNorm calculation used here, this is the population variance. In PyTorch that choice is explicit with unbiased=False.

Standard deviation restores the original scale

Variance squared the distances, so its units are squared too. Taking the square root brings the spread back onto the same scale as the original values:

standard_deviation = sqrt(variance)
# sqrt(4) = 2

Now I can perform the complete normalization: subtract the mean and divide by the standard deviation.

normalized = [-2 / 2, -2 / 2, 2 / 2, 2 / 2]
# [-1, -1, 1, 1]

The output is centered around zero and has a controlled spread. The order and relative relationships remain; the row has simply been put onto a predictable scale.

The complete formula

After working through the arithmetic one line at a time, the complete normalization formula became much less intimidating:

normalized value = (value - mean) / √variance

Written mathematically:

x̂ᵢ = (xᵢ - μ) / √σ²

Here, xᵢ is the value being normalized, μ is the mean of the row, and σ² is its variance. The expression √σ² is the standard deviation.

For the first value in [10, 10, 14, 14], the mean is 12 and the variance is 4:

x̂₁ = (10 - 12) / √4
   = -2 / 2
   = -1

For either occurrence of 14:

x̂ = (14 - 12) / √4
  = 2 / 2
  = 1

Applying the same formula to every value gives:

[10, 10, 14, 14] → [-1, -1, 1, 1]

The numerator, xᵢ - μ, centers each value around zero. The denominator, √σ², controls the spread.

The small epsilon is a safety rail

What if every value is identical? The variance and standard deviation would both be zero, and dividing by zero is not a valid operation. Implementations add a tiny value such as 1e-5 inside the square root:

norm_x = (x - mean) / torch.sqrt(variance + 1e-5)

For my hand-worked example I omitted epsilon to keep the arithmetic exact. With epsilon included, the outputs are approximately [-0.999999, -0.999999, 0.999999, 0.999999].

LayerNorm adds two trainable controls

Normalization itself uses statistics from the current input. LayerNorm then adds a trainable scale and shift for every feature:

class LayerNorm(nn.Module):
    def __init__(self, emb_dim):
        super().__init__()
        self.eps = 1e-5
        self.scale = nn.Parameter(torch.ones(emb_dim))
        self.shift = nn.Parameter(torch.zeros(emb_dim))

    def forward(self, x):
        mean = x.mean(dim=-1, keepdim=True)
        var = x.var(dim=-1, keepdim=True, unbiased=False)
        norm_x = (x - mean) / torch.sqrt(var + self.eps)
        return self.scale * norm_x + self.shift

scale starts as ones and shift starts as zeros, so they initially leave the normalized values alone. Unlike mean and variance, they are parameters. Training can update them when a loss is calculated, gradients flow backward and an optimizer takes a step.

Which values does GPT normalize together?

In the Chapter 4 GPT, an activation tensor has shape (batch, tokens, embedding). With an embedding size of 768, each token owns one row of 768 features. The dim=-1 calls calculate mean and variance across those 768 features independently for every token in every batch item.

input:     (2, 4, 768)
mean:      (2, 4,   1)
variance:  (2, 4,   1)
output:    (2, 4, 768)

The singleton final dimension is retained so PyTorch can broadcast each token's mean and variance back across its 768 features. LayerNorm does not mix the two sentences, and it does not mix the four token positions.

The mental model I am keeping

Mean finds the center. Variance measures squared spread. Standard deviation returns that spread to the original scale. Normalization subtracts the center and divides by the spread. Epsilon prevents division by zero. Finally, LayerNorm gives the model trainable scale and shift controls.

That is the whole mechanism. The implementation in Raschka's official Chapter 4 code and PyTorch's LayerNorm documentation can now be read as operations rather than incantations.