From Tokens to Token IDs

This is the third part of LLMs, the whole thing — the series where I build large language models up from the parts rather than the API. Part I turned words into vectors so that meaning became geometry. Part II walked through the transformer that consumes those vectors. Both quietly assumed something I never justified: that the text had already become numbers.

It hasn't. A language model never sees your text — it sees a list of integers. Before the first embedding lookup, before a single attention score, something has to turn "Hello, world." into [15496, 11, 995, 13]. That something is the tokenizer, and this article is about the two steps it runs — splitting text into tokens, then mapping tokens to integer IDs — plus the wall you hit trying to do it with a plain dictionary.

No maths here, and no model. Just a regular expression, a Python dict, and one honest problem we can't fully solve until the next part.

An LLM never sees your text

The whole input path is four arrows: raw text → tokens → token IDs → embeddings, and then the model. Parts I and II lived on the right-hand side. Everything below is the left-hand side — the plumbing that produces the integers an embedding layer indexes into.

To keep it concrete I'll use the same corpus as Sebastian Raschka's Build a Large Language Model (From Scratch): The Verdict, a public-domain Edith Wharton short story. It's 20,479 characters of English — small enough to read, big enough to break a naive tokenizer.

Step one: split the text with a regex

Tokenizing is just deciding where the boundaries are. The tempting version — text.split() — splits on whitespace and drags punctuation along with the word, so "world." becomes a single token distinct from "world". We want punctuation to stand on its own, so we reach for re:

import re

text = "Hello, world. Is this-- a test?"
result = re.split(r'([,.:;?_!"()\']|--|\s)', text)
result = [t.strip() for t in result if t.strip()]
# => ['Hello', ',', 'world', '.', 'Is', 'this', '--', 'a', 'test', '?']

The trick is the parentheses. Wrapping the pattern in a capture group tells re.split to keep the delimiters it splits on, so commas, periods and the em-dash come back as tokens instead of vanishing. The strip() pass then drops the empty strings the split leaves between adjacent delimiters.

Run that over the full story and you get 4,690 tokens. That's the raw stream. Now each one needs to become an integer.

Step two: turn the tokens into a vocabulary

A vocabulary is nothing fancier than a sorted dictionary from token to integer:

all_words = sorted(set(preprocessed))
vocab = {token: i for i, token in enumerate(all_words)}
# ('!', 0) ('"', 1) ...  ('younger', 1127) ('your', 1128)

Deduplicate with set, sort so the mapping is deterministic (same text, same IDs, every run), then number them. The 4,690 tokens collapse to 1,130 unique ones — that's the vocabulary, and it is the model's entire universe of words. If a token isn't in here, as far as this tokenizer is concerned it does not exist.

A tokenizer is just encode and decode

Wrap the two directions in a class and you have a working tokenizer. encode is a dictionary lookup; decode is the reverse, plus one regex to glue punctuation back onto the word it belongs to:

class SimpleTokenizerV1:
    def encode(self, text):            # text -> token IDs
        toks = self._split(text)
        return [self.str_to_int[t] for t in toks]

    def decode(self, ids):             # token IDs -> text
        text = " ".join(self.int_to_str[i] for i in ids)
        return re.sub(r'\s+([,.?!"()\'])', r'\1', text)

# encode('... the last he painted, you know ...')
# [1, 56, 2, 850, 988, 602, 533, 746, 5, 1126, 596, 5, ...]

Encode, then decode, and you get your sentence back. On text drawn from The Verdict it round-trips perfectly. The problem shows up the moment you feed it anything else.

The crack: a closed vocabulary

Try to encode a greeting the story never used:

tokenizer.encode("Hello, do you like tea?")
# KeyError: 'Hello'

'Hello' never appears in The Verdict, so it isn't in the vocabulary, so the lookup throws. This isn't a bug — it's the design. A vocabulary built by enumerating a fixed corpus is a closed world: it can represent exactly the words it was built from and nothing else. Real language is open — names, typos, slang, a word coined yesterday — so a closed vocabulary crashes on contact with it.

Patching the holes with special tokens

The standard first patch is to reserve a few special tokens: entries in the vocabulary that correspond to no real word but carry structural meaning.

• <|unk|> — a catch-all ID for any token not in the vocabulary, so the lookup can never crash.
• <|endoftext|> — a boundary marker between two unrelated documents concatenated into one training stream.
• [BOS] / [EOS] — begin and end of a sequence, where one sample starts and stops.
• [PAD] — filler so a batch of uneven-length sentences becomes one rectangular tensor.

Adding the first two is a two-line change to the vocabulary, and encode grows by one line — map anything unknown to <|unk|> before the lookup:

preprocessed = [t if t in self.str_to_int else "<|unk|>"
                for t in preprocessed]

The vocabulary goes from 1,130 to 1,132, and the crash is gone. GPT-2, for what it's worth, uses only <|endoftext|> of these — it needs no <|unk|> at all, for a reason we're about to reach.

What <|unk|> quietly throws away

Encode two sentences with the patched tokenizer, then decode them, and look at what comes back:

text = "Hello, do you like tea? <|endoftext|> In the sunlit terraces of the palace."
tokenizer.decode(tokenizer.encode(text))
# '<|unk|>, do you like tea? <|endoftext|> In the sunlit terraces of the <|unk|>.'

Both "Hello" and "palace" came back as <|unk|>. The tokenizer no longer crashes — but it has collapsed two completely different words into the same ID. The model can't tell a greeting from a building; every rare or unseen word becomes the same grey blank. And you can't fix this by growing the vocabulary, because no finite list contains every word in a living language. <|unk|> trades a crash for silent information loss. We don't want a bigger dictionary — we want to stop needing one.

The real fix: tiktoken, re, and byte-pair encoding

Here is the move that dissolves every problem above at once, and it's what real GPT models use. Instead of a word-level vocabulary, GPT tokenizes with byte-pair encoding (BPE) via OpenAI's tiktoken library. A BPE tokenizer still has a fixed vocabulary — 50,257 entries for GPT-2 — but the entries are subwords, not whole words, right down to single characters. Anything it hasn't seen, it breaks into pieces it has:

import tiktoken
enc = tiktoken.get_encoding("gpt2")

enc.encode("someunknownPlace")
# [11246, 34680, 27271]  ->  ['some', 'unknown', 'Place']

No <|unk|>, ever. "palace" becomes ['pal', 'ace']; a keyboard-mash like "Akwirw" splits into four sub-word pieces; a common word like "Hello" stays a single token. Every string is representable, the vocabulary stays a fixed size, and nothing collapses into a shared blank — the three problems from this article, solved together.

And notice what BPE still leans on: a re pattern. Before it merges anything, tiktoken runs a regular expression to pre-split the text into rough chunks — the same idea as step one, just a more careful pattern — and then applies its learned merges on top. The regex never went away; BPE is built on it. (tiktoken does the merging in Rust, which is why it clocks in around 5× faster than a pure-Python BPE on the same text.)

Where this leaves us

Turning tokens into token IDs is the doorway every LLM walks through, and you can build a working version with a regex and a dictionary in an afternoon. But that naive tokenizer has a closed vocabulary, and the standard patch — <|unk|> — buys you a tokenizer that doesn't crash at the price of throwing meaning away. Byte-pair encoding is how GPT gets both: an open vocabulary at a fixed size.

How BPE actually learns those subword merges — the training loop that decides "pal" + "ace" is worth its own token — is the next part of LLMs, the whole thing.