beginner
12 min read
Train, Validation, and Test Sets: Who Gets to See the Data?
You've trained a model. The accuracy looks great. You're ready to ship it. Then someone asks the question that stops every beginner cold: How do you know…
Read tutorialbeginner to intermediate
Validation and Model Selection
Choose models without fooling yourself: build baselines, split data honestly, cross-validate, tune carefully, and keep the final test truly final.
Tutorials
Articles in Validation and Model Selection
Follow the learning path in order or jump directly to the concept or workflow you need.
intermediate
10 min read
Cross-Validation Explained: Get a More Stable Estimate From Limited Data
You split your data once. Your model scores 0.91. You re-run the same split with a different random seed, and suddenly it scores 0.84. Same model. Same…
Read tutorial
intermediate
9 min read
Hyperparameter Tuning Explained: Search Without Fooling Yourself
Your best tuning score is not your model's true performance. It is the score of the luckiest experiment you ran.
Read tutorial
beginner
9 min read
Machine Learning Baselines: The Simple Prediction Your Model Must Beat
You train your first classifier. It hits 85% accuracy on the test set. You feel great—until you realize that a rule which ignores your data entirely would…
Read tutorial
intermediate
10 min read
Grouped and Time-Based Cross-Validation: Split Data Without Leaking the Future
You train a model. Cross-validation gives you a confident accuracy score. You deploy. The model underperforms in ways your validation never hinted at.
Read tutorial
beginner
12 min read
A Reliable Model Selection Workflow: From Baseline to Final Test
Here is the trap I see beginners fall into constantly: they try five algorithms, compare the scores, keep the winner, then tweak it, compare again, and…
Read tutorial
intermediate
9 min read
Reproducible Machine Learning Experiments: Make Results Comparable
You tune a model, retrain, and the score moves. The question is whether that movement came from your edit or from luck. Most beginners cannot tell the…
Read tutorial
beginner
10 min read
Which Machine Learning Algorithm Should You Use? A Problem-First Decision Guide
You have a tabular dataset, a prediction to make, and a list of algorithm names that reads like a menu in a language you have not learned yet. Random…
Read tutorial
intermediate
10 min read
The IID Assumption in Machine Learning: When Random Splits Make Sense
You train a model, split your data randomly, and the test score looks great. Then the model meets real-world data and quietly falls apart. The usual…
Read tutorial
intermediate
10 min read
Nested Cross-Validation Explained: Evaluate the Whole Selection Process
You run a grid search, watch the cross-validation score climb, and feel good about the model you are about to ship. Then the model lands on real data and…
Read tutorial
intermediate
12 min read
Bootstrap Resampling in Machine Learning: Estimate Uncertainty, Not Certainty
Your model scores 0.82 accuracy on the test set. That feels like a fact—something solid you can report and defend. Run the experiment again on a different…
Read tutorial
intermediate
9 min read
Compare Random, Grouped, and Time-Based Validation in Scikit-Learn
Same model. Same features. Two validation schemes. One score says 0.94, the other says 0.61.
Read tutorial
intermediate
9 min read
Estimate a Statistic’s Uncertainty With a Reproducible Bootstrap Experiment
You have one sample, one number, and no honest way to say how much that number would move if you collected the data again. The bootstrap does not create…
Read tutorial
beginner
8 min read
Run a Bounded Hyperparameter Search With Scikit-Learn
A search is not a machine that finds the best number. It is a bounded experiment whose output is a comparison.
Read tutorial
intermediate
9 min read
How Bootstrap Resampling Builds an Interval for a Statistic
You have a thousand bootstrap replicates sitting in a NumPy array. You have a histogram. What you do not have is the two numbers you actually need to…
Read tutorial
advanced
9 min read
How Certain Is a Cross-Validation Score? Derive the Limits of Fold-Based Uncertainty
Two models. Mean cross-validation scores of 0.842 and 0.847. The instinct is immediate: the second one wins.
Read tutorial
intermediate
10 min read
Run Nested Cross-Validation for Model Selection in Scikit-Learn
You tuned a model, watched the cross-validation score climb, and reported the best number. Then new data arrived and the model performed worse than…
Read tutorial
intermediate
9 min read
Why the Best Validation Score Is Often Too Optimistic
You run a grid search. The leaderboard prints a winner at 0.91. You ship it, and the first honest measurement lands near 0.86. Nothing broke. The model did…
Read tutorial
intermediate
10 min read
Why the Mean, Median, or Class Prevalence Is the Best Constant Baseline
You build a baseline, get a number, and have no idea whether that number is the best a constant can do or just a habit copied from a tutorial. Here is the…
Read tutorial