The Sliding Window: How One Story Becomes Thousands of Training Examples

The line that stopped me for ten minutes

I'm building a GPT-style model from scratch, one chapter at a time, and last session a line that looked utterly trivial brought me to a full stop. I printed the shape of my very first training batch:

inputs, targets = next(iter(loader))
print(inputs.shape)   # torch.Size([8, 4])

Eight by four. And I genuinely could not explain what those two numbers counted. I had just read an entire short story into memory — thousands of words — so where did 8 come from? Why 4? This post is me pulling that knot apart, because the confusion sat exactly where the data-loading abstraction leaks. Once it clicked, the whole input pipeline went transparent, and I suspect I'm not the only one who trips here.

One story becomes one long list of numbers

The raw material is the-verdict.txt, an Edith Wharton short story — 20,479 characters of ordinary English. A neural network does arithmetic, not letters, so the first move (the subject of the previous post) is to tokenize it: translate the text into integer IDs with the same byte-pair tokenizer GPT-2 uses.

import tiktoken
tokenizer = tiktoken.get_encoding("gpt2")
token_ids = tokenizer.encode(raw_text)
len(token_ids)   # 5145

So the whole story collapses into one flat list of 5,145 numbers, laid end to end in reading order. Not 8. Not 4. One list, 5,145 long. Hold onto that number.

The sliding window: 5,145 numbers become 1,286 examples

A model doesn't train on the whole story at once — it trains on small, fixed-width bites. So we slide a window across the list and snapshot it at regular jumps. Two knobs control the slide: max_length, how wide each window is, and stride, how far it jumps each time.

max_length = 4
stride = 4
for i in range(0, len(token_ids) - max_length, stride):   # i = 0, 4, 8, 12, ...
    input_chunk  = token_ids[i:i + max_length]             # 4 numbers = one window
    target_chunk = token_ids[i + 1:i + max_length + 1]     # same window, shifted by 1
    inputs.append(torch.tensor(input_chunk))
    targets.append(torch.tensor(target_chunk))

The one line worth staring at is the range. It's a counter: start at 0, jump by stride, stop before len(token_ids) - max_length. Why subtract max_length from the stop? Because every window needs four tokens to its right to be complete. If the window started too close to the end, the slice would run off the list and hand back a stunted two-token window. The subtraction is a "don't fall off the cliff" guard — it halts the walk while a full window still fits.

Run that loop over 5,145 tokens jumping 4 at a time and you get 1,286 windows. Every single one is stored. Nothing is discarded. That is what "I read the whole file" actually produced: not 8 anything, but 1,286 training examples.

Inputs and targets: the same list, nudged by one

Look again at the loop — each window is stored twice, as an input and as a target. The target is the identical window shifted right by exactly one token. That + 1 in the slice is the entire trick. Decoded back to English, the first window reads:

input   = ["I", " H", "AD", " always"]
target  = [" H", "AD", " always", " thought"]   # slid left by one

Line them up position by position and the point falls out: at every slot, the target is the next token. A language model has exactly one job — given some tokens, predict the one that follows — and the target column is literally the answer key for that job.

A single four-token window isn't one training example. It's four: "given I, predict  H"; "given I H, predict AD"; and so on. The shift-by-one unpacks four supervised lessons out of one row, for free.

This is why people call it self-supervised. Nobody hand-labelled anything. The answer key is just the same text offset by one position — the story quietly grades itself. That property is the whole reason raw, unlabelled internet text can train these models at all.

The number that isn't in your data

So where does the 8 come from? Not the story. It's the batch size — a knob I chose, not a fact about the text. Picture the 1,286 windows as a deck of cards and the DataLoader as a dealer. batch_size=8 means "deal them eight at a time," which makes 160 batches out of the deck.

inputs, targets = next(iter(loader))
#   8 windows        their 8 answer keys

And here's the resolution to my ten-minute stall: next(iter(loader)) deals one batch. Not the whole deck — the top eight cards. The other 1,278 windows are still sitting in the dataset; the code just never asked for them. My instinct ("but I read the whole file!") was correct: reading the file is what built all 1,286 windows. The 8 is a completely separate decision made one line later about how many to grab at once.

Two numbers, two jobs. 1,286 is what the file gave me — story length divided by stride. 8 is what I asked for — batch size, a free knob I can set to 1, 2, or 32 without the story changing at all. The 4, meanwhile, is dictated by the model's context length. Only one of the three numbers in [8, 4] comes from the data.

A token is not a word

One last thing the real output taught me. That first decoded window — ["I", " H", "AD", " always"] — is four tokens but only three words. The tokenizer split "HAD" into " H" and "AD". It's a small thing, but it's why the counts never line up with your intuition about words, and why "4 tokens" is not "4 words." When you're reasoning about context windows and costs, tokens are the unit that matters, and they rarely map cleanly onto anything you'd call a word.

The mental model that fixed it

The pipeline is four moves: a file becomes one long list of token IDs; a sliding window chops that list into many fixed-width examples; each example carries a shifted-by-one target as its answer key; and a batch grabs a handful of those examples to process together. The confusion evaporates the moment you stop conflating "how many examples exist" with "how many I grabbed."

I wrote the whole journey into a single runnable script that prints what each step produces, so the 1,286 and the 8 never blur together again. If you're stuck on the same shape, tracing it top to bottom is worth more than any diagram — watch the numbers appear and the abstraction stops being magic.