GELU vs ReLU: The Smooth Gate Inside GPT

The first useful difference is visible at −1

While implementing Chapter 4 of Build a Large Language Model (From Scratch), I reached a function called GELU. Calling it an activation function was accurate but insufficient. I wanted to know what happened to an actual number.

I evaluated GELU and ReLU over the same five inputs:

x = torch.tensor([-2.0, -1.0, 0.0, 1.0, 2.0])

ReLU(x) = [0.0000, 0.0000, 0.0000, 1.0000, 2.0000]
GELU(x) = [-0.0454, -0.1588, 0.0000, 0.8412, 1.9546]

The decisive value is -1. ReLU returns exactly 0. GELU returns approximately -0.1588. GELU reduces the negative input strongly, but it does not necessarily erase it.

In the screenshot, torch.linspace(-3, 3, 100) creates 100 evenly spaced inputs for both functions. The shared range makes the contrast honest: ReLU stays exactly on zero throughout the negative side, while GELU dips slightly below zero and curves smoothly back through the origin. On the positive side, both rise with the input, but GELU approaches that straight line gradually.

ReLU is a hard rule

ReLU is short for rectified linear unit. Its complete rule is:

ReLU(x) = max(0, x)

If x is negative, return zero. If x is positive, return x. Applied to one value at a time:

ReLU(-1) = max(0, -1) = 0
ReLU( 2) = max(0,  2) = 2

This produces the sharp corner in the plot. Everything to the left of zero lies flat on zero; everything to the right follows a straight line.

GELU is a smooth, input-dependent gate

GELU also processes every value independently, but it does not make a binary keep-or-delete decision. It computes a multiplier from the input and then multiplies the input by it:

GELU(x) = x × gate(x)

For x = 1, the gate is approximately 0.841192:

1 × 0.841192 = 0.841192

For x = -1, the gate is approximately 0.1588:

-1 × 0.1588 = -0.1588

The gate is computed from x. It is not a stored weight and GELU owns no trainable parameters.

The tanh approximation, one operation at a time

The Chapter 4 implementation uses this approximation:

GELU(x) ≈ 0.5 × x × (
    1 + tanh(
        √(2 / π) × (x + 0.044715 × x³)
    )
)

It looks dense because several small operations are written on one line. For x = 1, I can unfold it:

x³                              = 1.000000
x + 0.044715x³                  = 1.044715
√(2/π)                          = 0.797885
0.797885 × 1.044715             = 0.833562
tanh(0.833562)                  = 0.682384
0.5 × (1 + 0.682384)            = 0.841192
x × 0.841192                    = 0.841192

The last-but-one result is the gate. The final line multiplies the original input by that gate.

What tanh contributes

The hyperbolic tangent function maps any real input to a value between -1 and 1. GELU adds one and multiplies by one half:

gate = 0.5 × (1 + tanh(...))

That converts the tanh output into a smooth gate between zero and one. Strongly negative inputs receive gates close to zero. Positive inputs receive increasingly larger gates. The transition is gradual rather than a hard switch at zero.

Where GELU sits inside GPT

In the Chapter 4 feed-forward network, GELU sits between two trainable linear layers:

nn.Linear(768, 3072)
GELU()
nn.Linear(3072, 768)

The first linear layer expands every token from 768 to 3072 features. GELU changes each of those 3072 values independently and preserves the shape. The second linear layer projects the token back to 768 features:

(batch, tokens, 768)
→ Linear
(batch, tokens, 3072)
→ GELU
(batch, tokens, 3072)
→ Linear
(batch, tokens, 768)

The two linear layers own trainable weights and biases. GELU does not. It still matters during training because its transformation affects the values flowing forward and the gradients flowing backward.

The mental model I am keeping

ReLU applies a hard rule: negative becomes zero, positive passes unchanged. GELU computes a smooth gate from each input. Negative values can survive in reduced form, and positive values pass more strongly as they grow.

The useful shorthand is not “GELU is complicated ReLU.” It is:

ReLU: hard cutoff
GELU: input × smooth gate(input)

The implementation is visible in Raschka's official Chapter 4 code. PyTorch documents the same tanh GELU approximation and the exact ReLU rule.