Measure predictive quality and diagnose failure with metrics, error analysis, class-imbalance reasoning, generalization diagnostics, thresholds, calibration, and interpretation.
Tutorials
Follow the learning path in order or jump directly to the concept or workflow you need.
Your model just aced its training data. Ninety-eight percent accuracy. Beautiful curves. Then you hand it new examples, and it stumbles like it never saw…
Read tutorial
A model that scores 90% accurate can still fail at the exact job it was built for. The headline number looks impressive, but it may be hiding a model that…
Read tutorial
A model that predicts "no fraud" for every transaction can score 98% accuracy while catching zero fraud. The number feels trustworthy. It is not. Accuracy…
Read tutorial
Two models can post nearly identical test scores and still fail for opposite reasons. One misses because it never learned the pattern. The other misses…
Read tutorial
Your spam filter just flagged an email with a score of 0.55. Is it spam? The model isn't telling you—it's asking you to decide. Somewhere between the…
Read tutorial
A model can rank every case perfectly and still hand you probabilities you cannot spend. Calibration is the difference between a score that orders risk and…
Read tutorial
"The model says education matters" sounds like a complete explanation. It is not. It is the beginning of a much narrower claim: the model leaned on…
Read tutorial
If you are like most beginners, you start guessing. Maybe more data will fix it. Maybe a bigger model. Maybe fancier features. You try one, see little…
Read tutorial
You train a fraud detector on a dataset where only 0.5% of transactions are fraudulent. The ROC AUC comes back at 0.95. You feel great. Then you look at…
Read tutorial
You've added features. You've tuned hyperparameters. You've tried a more flexible model. And still, validation performance sits at the same stubborn…
Read tutorial
A regression model reports a solid R-squared, so you call it done. Then you plot the predictions against the actual values and notice something…
Read tutorial
Your model aced the test set. Clean evaluation, strong metrics, confident you. Then you deploy it, and the real world quietly moves on without telling you.
Read tutorial
You improved the model twice. The validation score went up, then up again. Now every change you try makes it worse or does nothing. You are not out of…
Read tutorial
A regression model hands you one number. The person acting on that number needs to know how much weight it can bear. A prediction interval is how you show…
Read tutorial