Tag
LLMs, the whole thing · Part 3
An LLM never sees your text — it sees integers. Building a tokenizer from a regex and a dictionary, hitting the out-of-vocabulary wall, and why GPT’s byte-pair encoding is the fix that needs no unknown token.
I built a local instrument to look at embedding space, and most of what I assumed turned out to be wrong: nothing is orthogonal, no individual dimension means anything, and king minus man plus woman does not give you queen.
LLMs, the whole thing · Part 1
The opening part of a series building LLMs from first principles. We start where every model starts: turning language into vectors and meaning into geometry, with a runnable script and its real numbers.