Training in Practice: Epochs, Batches and Learning Rate
You have already met the idea that a neural network learns by looking at examples, guessing, checking how wrong it was, and nudging its numbers a little. That is the story in one sentence. This lesson is the story in full detail: how a real training run is actually organised, what every word on the progress screen means, how to read the graphs that tell you if things are going well, and how to stop the network from cheating by memorising. By the end you will be able to look at a real training log and say, like a doctor reading a heartbeat, "healthy", "sick", or "here is exactly what is wrong".
What you will be able to do
- Define batch, iteration (step) and epoch, and count how many of each a run contains.
- Explain why we train in mini-batches instead of one example at a time or all data at once.
- Read the training loop as simple pseudocode.
- Diagnose a run from its training curves: healthy, underfitting, overfitting, or learning-rate-too-high.
- Split data correctly into train / validation / test and know why the test set is touched only once.
- Use early stopping, dropout, weight decay and data augmentation to fight overfitting.
- Pick sensible starting hyperparameters and tune them one change at a time.