#tokenization

LLMs, the whole thing · Part 4

I printed my first training batch, saw shape [8, 4], and could not say what the two numbers counted. Untangling it - one short story sliced into 1,286 windows, grabbed 8 at a time - made the whole GPT input pipeline finally go transparent.

• #training-data
• #llms
• #pytorch
• #tokenization

LLMs, the whole thing · Part 3

An LLM never sees your text — it sees integers. Building a tokenizer from a regex and a dictionary, hitting the out-of-vocabulary wall, and why GPT’s byte-pair encoding is the fix that needs no unknown token.

• #fundamentals
• #nlp
• #gpt
• #llm
• #tokenization