How LLMs Work: Tokens and Next-Word Prediction
The one big idea: a Large Language Model (an "LLM" — a computer program like ChatGPT that reads and writes text) does one surprisingly simple thing over and over. It looks at the text so far and guesses the next small piece of text. Then it adds that piece and guesses again. That is almost the whole secret.
Everything that feels magical — writing an essay, answering a question, translating Kyrgyz into English — is built out of that single repeated guess. In this lesson we open the machine and look at every gear: how text is chopped into tokens, how the model turns them into numbers, how it produces a probability for every possible next token, how temperature makes answers vary, what the context window can and cannot remember, and why making these models bigger unlocks surprising new skills.
What you will be able to do
- Explain what a token is and split a real sentence into tokens by hand.
- Describe a vocabulary and how each token becomes a numeric token ID.
- Walk through the next-token prediction loop step by step, including the probabilities.
- Explain why temperature and sampling make the same prompt give different answers.
- Say what a context window is, what fits in it, and what gets forgotten.
- Explain in plain words why scale (parameters + data + compute) leads to emergent abilities.
- Explain why fluent text is not the same as understanding or having goals.
1. Tokens: chopping text into pieces
A model cannot read letters the way you do. Before anything happens, the text is cut into small chunks called tokens. A token is often a whole word, but very often it is a word-piece — a fragment of a word. Roughly, in English, 1 token is about 4 characters, and 100 tokens is about 75 words. These are just rules of thumb, not exact laws.