Attention and the Transformer
In the last lesson you met the RNN — a network that reads a sentence one word at a time and tries to remember everything in a single running memory. It works, but it has a painful weakness: it forgets. This lesson is about the idea that fixed that weakness and, in doing so, launched the entire era of modern AI. That idea is called attention, and the network built around it is called the Transformer.
The big idea in one sentence: instead of forcing a model to cram a whole sentence into one small memory, let it look back at every word and pay more attention to the words that actually matter right now. Every large language model you have heard of — ChatGPT, Claude, Gemini — is a Transformer. By the end of this page you will understand how they are built.
- Why the RNN "single-memory bottleneck" hurts, and how attention removes it.
- The open-book exam way to picture attention.
- How to read attention weights as percentages over the words of a sentence.
- What Query, Key and Value mean in plain words.
- The three-step attention recipe: similarity → softmax → weighted mix — worked with real numbers.
- What self-attention and multi-head attention do.
- How the Transformer (2017) stacks these pieces, why it needs positional encoding, and why it trains so fast on GPUs.
- The difference between encoder-style (BERT) and decoder-style (GPT) models.
Recall how an RNN reads text. It goes word by word, left to right. After each word it updates one hidden state — a fixed-size list of numbers that is supposed to hold a summary of everything read so far. When the sentence ends, that single state is the network's entire memory of the sentence.