Training GPT from Scratch: Loss, Learning, and the Next Token

Chapter 4 gave me a GPT architecture that could produce tokens. Chapter 5 of Sebastian Raschka's Build a Large Language Model (From Scratch) explains how its parameters acquire useful values: measure prediction error, compute gradients, update weights, and repeat.

I read the chapter alongside local Python examples. The distinction I wanted to keep clear was between learning better probabilities and choosing tokens from those probabilities. Training changes the model. Decoding changes how I use its output.

The text supplies its own targets

“Unlabeled” does not mean there is nothing to compare against. For next-token prediction, the expected answer is already in the text. Shift the sequence one position:

Text:     Every   effort   moves   you
Inputs:   Every   effort   moves
Targets:  effort  moves    you

Causal attention gives each position its prefix. The first predicts effort; the second predicts moves after seeing Every effort. Every input position contributes a prediction, not only the last one.

My local copy of The Verdict contains 5,145 GPT-2 tokens. Following the chapter, I split the raw text at 90% of its character count, then tokenize each part separately. With 256-token windows, stride 256, and batch size two, that produces nine training batches and one validation batch. These are educational proportions, not a recipe for production-quality pretraining.

Loss turns predictions into a measurable objective

A batch with shape (2, 256) produces logits shaped (2, 256, 50257). There are 512 prediction positions, each with 50,257 candidate-token scores. Vocabulary size counts possible answers; it does not count prediction tasks.

logits = model(inputs)
loss = torch.nn.functional.cross_entropy(
    logits.flatten(0, 1),  # (512, 50257)
    targets.flatten(),    # (512,)
)

For each position, cross-entropy takes the negative log-probability assigned to the expected token, then averages the losses. I pass raw logits: PyTorch's cross-entropy already incorporates log-softmax. Applying softmax first would feed the wrong values into that calculation.

Perplexity is exp(loss) when loss uses natural logarithms. If every target had probability 0.1, perplexity would be 10. For varying probabilities, it is the reciprocal of their geometric mean: an effective number of equally likely choices, not literal uniform guessing.

My earlier six-position example returned loss 10.950879 and perplexity 57004.14. Perplexity can exceed vocabulary size when the model assigns sufficiently poor target probabilities. Comparisons also depend on tokenization and the evaluation context, as the Hugging Face evaluation guide explains.

The four operations that make training happen

model.train()
for inputs, targets in train_loader:
    optimizer.zero_grad()
    loss = calc_loss_batch(inputs, targets, model, device)
    loss.backward()
    optimizer.step()

zero_grad() clears accumulated gradients. The forward pass calculates predictions and loss. backward() computes derivatives of that loss with respect to trainable parameters. The parameters change at optimizer.step().

The chapter uses AdamW. Its learning rate scales updates, its running statistics adapt them, and weight decay separately shrinks parameters. I create the optimizer once, outside the epoch loop, so its state survives between batches.

One step handles one batch. One epoch completes a pass through the loader. Shuffling changes window order, preserving token order inside each window. The same parameters continue evolving across epochs; nothing resets automatically.

Measure generalization, then inspect the text

Training loss measures performance on material used for updates. Validation loss measures held-out material. Falling training loss alongside rising validation loss suggests overfitting: updates are becoming less useful outside the training set. It does not, by itself, prove verbatim memorization.

During evaluation, model.eval() disables dropout; torch.no_grad() disables gradient recording. These are different controls. Neither performs an update. The chapter's helper averages batch losses equally, which differs from a token-weighted mean when batch sizes differ.

My separate CPU practice model uses two layers, embedding width 64, four attention heads, and context length 64. After two epochs, its last sampled validation loss was 8.7834, down from 10.9867 after the first update. Yet greedy generation after Every effort moves you produced fifty periods. Lower loss did not make this short run a useful language model. These results belong to the tiny configuration, not the chapter's 124M configuration.

Temperature and top-k control generation

Greedy decoding always selects the largest next-token logit. Temperature sampling instead uses softmax(logits / temperature) for a positive temperature. Lower values concentrate probability; higher values flatten it. The generator then samples a token and appends it to the context.

For illustrative logits [2, 1, 0], the leading candidate receives approximately 86.7% probability at temperature 0.5, 66.5% at 1, and 50.6% at 2. Its rank stays first; its chance of being sampled changes.

Top-k keeps the highest-scoring candidates and excludes the rest before sampling. With these logits, top-two at temperature one gives approximately [73.1%, 26.9%, 0%]. The updated generator combines these controls with a maximum token count and an optional end-of-sequence stop. In the book's interface, temperature zero selects the greedy branch; it never divides by zero.

These controls change output selection. They do not update weights, add knowledge, or repair overfitting.

Keep learned weights, or start from pretrained ones

Section 5.4 saves learned parameter values through model.state_dict(). Reloading requires a compatible architecture. To resume AdamW training, I also need its optimizer state; keeping the model configuration and training position makes the checkpoint usable later. PyTorch's saving and loading guide covers this distinction.

Section 5.5 loads pretrained GPT-2 weights into the architecture built earlier. The configuration must match, including the 1,024-token context and query/key/value biases. The loader maps embeddings, normalization parameters, attention projections, and feed-forward weights into their corresponding tensors; some matrices require transposition and packed Q/K/V weights require splitting.

My local checks here cover loss calculation and short training runs. Loading pretrained weights is the chapter's next practical comparison, not a result I am claiming from those runs. The full path is available in Raschka's Chapter 5 code.

The architecture defines the computation. Training changes its parameters. Decoding chooses its outputs. Saving and loading preserve or replace those parameters. Keeping those operations separate made Chapter 5 much easier for me to reason about.