beginner
8 min read
Run a Bounded Hyperparameter Search With Scikit-Learn
A search is not a machine that finds the best number. It is a bounded experiment whose output is a comparison.
Read tutorialworkflow
Model Validation
Methods for estimating generalization and comparing candidate models while preserving an honest information boundary.
Tagged articles
Model Validation articles
40 articles in this tag.
intermediate
9 min read
Build and Evaluate a Stacking Classifier With Scikit-Learn
A stacking classifier can score beautifully on your test set for a reason that has nothing to do with skill: the meta-model was trained on predictions the…
Read tutorial
intermediate
9 min read
Choose a Classification Threshold With a Cost-Based Scikit-Learn Experiment
A model with a respectable AUC can still make the wrong decision every single day. The scores are fine. The cutoff is the problem.
Read tutorial
beginner
10 min read
Compare Scikit-Learn Pipeline Models Without Touching the Test Set
You have three pipelines. Their scores are close enough that the ranking feels like a coin flip. So you run them against the test set to break the tie —…
Read tutorial
intermediate
11 min read
Compare Feature-Selection Strategies in a Leakage-Safe Workflow
You run SelectKBest, get a tidy list of "important" features, then rerun with a different random seed and the list changes. That surprise is not a bug in…
Read tutorial
intermediate
9 min read
Compare Random, Grouped, and Time-Based Validation in Scikit-Learn
Same model. Same features. Two validation schemes. One score says 0.94, the other says 0.61.
Read tutorial
beginner
9 min read
Compare Ridge and Lasso Coefficients With a Controlled Scikit-Learn Experiment
You fit ridge, you fit lasso, lasso wins by a hair, and you quietly file away "lasso is better." That conclusion is a coin flip wearing a lab coat. One…
Read tutorial
intermediate
10 min read
Cross-Validation Explained: Get a More Stable Estimate From Limited Data
You split your data once. Your model scores 0.91. You re-run the same split with a different random seed, and suddenly it scores 0.84. Same model. Same…
Read tutorial
advanced
9 min read
How Certain Is a Cross-Validation Score? Derive the Limits of Fold-Based Uncertainty
Two models. Mean cross-validation scores of 0.842 and 0.847. The instinct is immediate: the second one wins.
Read tutorial
intermediate
9 min read
Data Leakage in Machine Learning: The Shortcut That Corrupts Evaluation
You train a model. The validation score comes back at 0.98. You feel like a genius. Then the model goes into the real world and performs like a coin flip.
Read tutorial
beginner
10 min read
A Decision Tree Experiment: Inspect Splits and Observe Overfitting
A decision tree can score 100% on the data it was trained on and still be wrong about almost everything else. That gap is not a bug. It is the lesson.
Read tutorial
intermediate
9 min read
Test a Feature Engineering Idea With a Leakage-Safe Experiment
A feature idea is a hypothesis. Change one thing, measure the same way twice, and let the output decide.
Read tutorial
intermediate
9 min read
Feature Selection vs Feature Engineering: Remove Signal or Build It?
You're staring at a wide table of columns, and the model is underperforming. The fix could mean deleting half the columns, reshaping the ones that remain,…
Read tutorial
intermediate
10 min read
Experiment With Gradient Boosting: Learning Rate, Trees, and Validation
A single accuracy number tells you almost nothing. A controlled experiment tells you whether that number means anything at all.
Read tutorial
intermediate
10 min read
Grouped and Time-Based Cross-Validation: Split Data Without Leaking the Future
You train a model. Cross-validation gives you a confident accuracy score. You deploy. The model underperforms in ways your validation never hinted at.
Read tutorial
intermediate
9 min read
Hyperparameter Tuning Explained: Search Without Fooling Yourself
Your best tuning score is not your model's true performance. It is the score of the luckiest experiment you ran.
Read tutorial
intermediate
10 min read
The IID Assumption in Machine Learning: When Random Splits Make Sense
You train a model, split your data randomly, and the test score looks great. Then the model meets real-world data and quietly falls apart. The usual…
Read tutorial
intermediate
11 min read
Imbalanced Classification: Why Accuracy Can Hide the Failure
A model that predicts "no fraud" for every transaction can score 98% accuracy while catching zero fraud. The number feels trustworthy. It is not. Accuracy…
Read tutorial
beginner
8 min read
Plot and Interpret Learning Curves With Scikit-Learn
You have seen the classic two-line plot. One line starts high and drifts down. The other starts low and climbs. Somewhere on the right, they meet, and the…
Read tutorial
beginner
11 min read
Fit and Evaluate Linear Regression With Scikit-Learn
A model that fits is not yet a model you can trust. Fitting takes one line. Trust takes a split, a reading of the numbers, and a look at where the errors…
Read tutorial
absolute beginner
9 min read
A Practical Machine Learning Learning Roadmap Before Deep Learning
Most beginners treat machine learning as a collection of algorithms to memorize. They collect names like trading cards—logistic regression, random forest,…
Read tutorial
intermediate
9 min read
Why the Best Validation Score Is Often Too Optimistic
You run a grid search. The leaderboard prints a winner at 0.91. You ship it, and the first honest measurement lands near 0.86. Nothing broke. The model did…
Read tutorial
beginner
12 min read
A Reliable Model Selection Workflow: From Baseline to Final Test
Here is the trap I see beginners fall into constantly: they try five algorithms, compare the scores, keep the winner, then tweak it, compare again, and…
Read tutorial
beginner
9 min read
Build a Sparse-Text Naive Bayes Classifier With Scikit-Learn
The most dangerous bug in a beginner text classifier is not in the model. It is in the order of operations. Fit your vectorizer on the whole dataset before…
Read tutorial
intermediate
10 min read
Nested Cross-Validation Explained: Evaluate the Whole Selection Process
You run a grid search, watch the cross-validation score climb, and feel good about the model you are about to ship. Then the model lands on real data and…
Read tutorial
intermediate
10 min read
Run Nested Cross-Validation for Model Selection in Scikit-Learn
You tuned a model, watched the cross-validation score climb, and reported the best number. Then new data arrived and the model performed worse than…
Read tutorial
intermediate
9 min read
PCA vs Feature Selection: Compress the Data or Keep the Original Variables?
You have a wide table of features, a model to build, and a quiet suspicion that half those columns are noise. The instinct is to shrink the dataset. But…
Read tutorial
beginner
9 min read
Compare Polynomial Degrees With Leakage-Safe Validation
Fit degree 1 through 9, watch training error fall to almost zero, and pick degree 9. That is the trap. Training error measures how well a model remembers…
Read tutorial
beginner
10 min read
Polynomial Regression Explained: Make Curves With a Linear Model
You fit a straight line to data that clearly curves, and the line misses everything. Not by a little—systematically. It sits above the points in one…
Read tutorial
beginner
9 min read
Fit a Leakage-Safe Scikit-Learn Pipeline on Tabular Data
You scale the whole table, split it, train a model, and get 0.94. You ship it, and real data hands you 0.71. Nothing broke in production. The break…
Read tutorial
intermediate
8 min read
Inspect Random-Forest Out-of-Bag Scores as the Forest Grows
oob_score=True looks like a free validation set. It is not free, and it is not quite a validation set. It is a running estimate computed on the same rows…
Read tutorial
intermediate
9 min read
Reproducible Machine Learning Experiments: Make Results Comparable
You tune a model, retrain, and the score moves. The question is whether that movement came from your edit or from luck. Most beginners cannot tell the…
Read tutorial
intermediate
10 min read
Run a Scikit-Learn Error Analysis: Find Slices and Choose a Next Experiment
Your model reports 0.91 accuracy. That number is probably correct, and it is probably useless for deciding what to fix next.
Read tutorial
beginner
10 min read
Scikit-Learn Pipelines Explained: Keep Preparation and Prediction Together
You fit a scaler on your full dataset, split into training and test sets, train a model, and watch it score beautifully. Then the model meets real data and…
Read tutorial
intermediate
9 min read
How Stacking Uses Out-of-Fold Predictions to Train a Meta-Model
Your stacking classifier reports 0.97 accuracy during training. On held-out data, it drops to 0.81 — barely better than the single best base model. Nothing…
Read tutorial
intermediate
8 min read
Test SVM Scaling, Kernels, and Regularization With Scikit-Learn
Same data. Same SVC call. One run scores 0.85, the next 0.55, and the only thing that changed was whether you scaled the features first.
Read tutorial
intermediate
8 min read
Test Whether an Interaction Feature Helps a Model
Add a product term, watch the validation score tick up, declare victory. I have watched this exact sequence produce a confident conclusion on a dataset…
Read tutorial
beginner
12 min read
Train, Validation, and Test Sets: Who Gets to See the Data?
You've trained a model. The accuracy looks great. You're ready to ship it. Then someone asks the question that stops every beginner cold: How do you know…
Read tutorial
intermediate
10 min read
Voting and Stacking Ensembles: Combine Different Models Without Losing the Plot
More models do not automatically mean a better model. An ensemble only wins when its members make different mistakes—and the way you combine them…
Read tutorial
beginner
10 min read
Which Machine Learning Algorithm Should You Use? A Problem-First Decision Guide
You have a tabular dataset, a prediction to make, and a list of algorithm names that reads like a menu in a language you have not learned yet. Random…
Read tutorial