Evaluating Models: Accuracy, Precision, Recall
The big idea: Training a model is only half the job. The other half is asking an honest question: is this model actually any good? To answer that we need measurements — numbers that tell us how often the model is right, how often it is wrong, and what kind of wrong it is. This lesson gives you the full toolbox that professional machine-learning engineers use to judge a model.
You will learn that a single "score" can lie to you badly, and how to look deeper with the confusion matrix and the four numbers that come out of it.
- Explain why one accuracy number is not enough, especially when the data is imbalanced.
- Build a confusion matrix from scratch and define TP, FP, FN, TN.
- Compute accuracy, precision, recall and F1 by hand from that matrix.
- Decide which metric matters most for a given problem (spam vs cancer vs fraud).
- Understand the precision-recall trade-off and the threshold dial.
- Measure regression models with MAE and RMSE, and know when to prefer each.
- Always compare a model against a simple baseline.
- Read sklearn's
classification_reportandconfusion_matrix.
Imagine a friend tells you: "My model is 99% accurate!" That sounds amazing. But it might be one of the most useless models ever built. How can that be?
Suppose the model's job is to detect a rare disease that only 1 person in 100 actually has. A lazy model could simply answer "healthy" to every single person — never even looking at them. It would be wrong only on the 1 sick person out of 100, so it would be right 99 times out of 100. That is 99% accuracy for a model that never catches a single sick person. It is completely worthless, yet the accuracy number looks fantastic.