Reproducible Machine Learning Experiments: Make Results Comparable

You tune a model, retrain, and the score moves. The question is whether that movement came from your edit or from luck. Most beginners cannot tell the difference, because they treat a single score as a stable fact about the model. It is not. A score is one draw from a distribution, and until you understand what makes that distribution spread out, every comparison you run is a guess wearing a number's clothing.

The fix is not to eliminate all variation. It is to separate the variation you should lock down from the variation you should measure. That distinction is the core skill in reproducible machine learning.

Why Your Scores Keep Moving

Imagine training the same model twice on the same dataset and getting two different accuracy values. Nothing about your code changed. The data did not change. Yet the numbers disagree. This is not a bug. It is the normal behavior of a stochastic process.

Two distinct sources of variation are at work. The first is which rows land in the training set versus the validation set. A model trained on one subset of your data will not perform identically to a model trained on another. The second source is randomness inside the algorithm itself: initialization, shuffling, subsampling. Many estimators in scikit-learn and other libraries use pseudo-random number generators internally, and those generators produce different sequences on every run unless you control them.

The beginner mistake is treating both sources as noise to eliminate. The real skill is deciding which to lock down and which to keep visible. Some variation is an artifact of your setup, and you should fix it so comparisons stay clean. Other variation is genuine information about how stable your model is, and hiding it behind a single number will fool you.

This builds directly on the train/validation/test boundary and cross-validation ideas you have already seen. The new question is not how to split data. It is how to make the results of those splits trustworthy and comparable.

Lock Down the Deterministic Setup

The deterministic setup is everything that should produce the same output when you run the same code twice. Three pieces matter most: random seeds, data version, and environment.

Random seeds. A random seed is the starting point for a pseudo-random number generator. Set the seed, and the generator produces the same sequence every time. In a scikit-learn workflow, seeds matter in three places: the train/test split, cross-validation folds, and any estimator that uses randomness internally. Setting a seed in one place but not the others leaves your experiment only partially reproducible.

A common pattern is to set the seed once near the top of your script and pass it to the splitter and the model. If you use train_test_split, pass random_state. If you use KFold, pass shuffle=True and random_state. If your model accepts a random_state parameter, set it there too. The goal is that the same code and data produce the same result, run after run.

Data version. A dataset that changes between runs silently invalidates every comparison you make. If you add rows, remove outliers, or fix a labeling error, the new data is a different input. Record which version of the data each experiment used. A simple convention—a filename with a date or a hash, a note in your log, a folder per dataset version—is enough at this stage. The point is that you can always answer the question: what exactly did this model see?

Environment. Python version and library versions can change results. A scikit-learn upgrade can alter default behavior. A NumPy version change can shift numerical results. The lightweight fix is a requirements file or lock file that records the exact versions you used. Full containerization with Docker is useful later, but most beginners do not need it yet. A lock file that lets you recreate the environment is the practical starting point.

One honest caveat: full determinism is a spectrum, not a switch. Parallel processing and some libraries can introduce nondeterminism even with seeds set. Do not chase perfect reproducibility. Chase a record good enough that you can trust your comparisons and explain your results.

Keep Genuine Uncertainty Visible

Here is where beginners overcorrect. They set one seed, get one number, and assume reproducibility means the number is now trustworthy. It is reproducible, yes. But it can still be unrepresentative.

Setting one seed gives you one reproducible draw from the distribution. If the model is sensitive to which data it sees, that single draw can be unusually good or unusually bad. You have removed the variation from your setup, but you have also hidden the variation that tells you how much to trust the estimate.

Cross-validation already exposes part of this spread. The fold-to-fold differences in your scores are real signal about how stable the model is across data subsets. Repeated runs with different seeds expose the additional spread from algorithm randomness. Both are worth measuring.

The decision rule is simple: fix the seed for a clean comparison of two model versions, but report the spread when you want to know how much to trust the estimate. Report the mean and the range across folds or runs. A model whose score ranges from 0.71 to 0.79 across folds is a different proposition from one that ranges from 0.74 to 0.76, even if both average 0.75.

The concrete failure mode is comparing one lucky run against one unlucky run. You change a hyperparameter, retrain, and see a 0.03 improvement. That improvement might be real. It might also be the difference between a fortunate draw and an unfortunate one. Without measuring the spread, you cannot tell.

Build a Small Experiment Record

The bridge between "I ran something" and "I can defend this result" is a record. Three weeks after an experiment, memory is useless. You will not remember which edit produced which number, which seed you used, or whether the data had been cleaned yet. A timestamped log solves this.

The minimal record has six fields:

• What changed: the code version or the specific edit you made
• Data version: which dataset this experiment used
• Seed policy: which seeds you used, or whether you ran multiple seeds
• Evaluation design: the split or cross-validation setup
• Metric and spread: the score, plus the range across folds or runs
• Environment: Python and library versions

A plain-text file or spreadsheet is enough. Full experiment-tracking tools exist and are valuable, but they are optional at this stage. The habit matters more than the tool.

A useful log entry looks like this:

2025-06-14 14:32
Change: added interaction feature between age and income
Data: churn_v3.csv (after removing duplicate rows)
Seeds: 42 (split), 42 (model), 5 runs with seeds 1-5
Evaluation: 5-fold CV, stratified
Metric: ROC-AUC mean 0.812, range 0.794-0.831
Environment: Python 3.11, scikit-learn 1.4.2, pandas 2.2.1

That entry contains everything needed to reconstruct the experiment or explain it to someone else. It also makes the comparison in the next section trustworthy, because you can see exactly what differed between two runs.

Compare Model Changes Without Fooling Yourself

Now the discipline pays off. A reliable comparison follows one rule: change one thing at a time, keep the evaluation design identical, and compare distributions of scores rather than single numbers.

Start with a baseline. Run it with your seed policy and record the spread. Then run your candidate change with the same seed policy and the same evaluation design. If the candidate's improvement is larger than the spread you measured, the change is probably real. If the improvement is smaller than the spread, you cannot distinguish the change from noise.

The common beginner trap is comparing a tuned model against a baseline that used a different split or a different seed. That comparison confounds your change with the setup. You are no longer measuring whether your edit helped. You are measuring whether your edit plus a different data split helped, and those two effects are tangled together.

When should you average multiple runs instead of relying on one seeded run? Use a single seeded run when you are doing quick exploration and just need a rough signal. Use multiple runs with different seeds when you are about to make a decision based on the result: choosing a final model, reporting a number, or concluding that one approach beats another. The cost of a few extra training runs is small compared with the cost of choosing the wrong model.

What to Record and What to Let Vary

The whole mental model compresses into a clean decision boundary.

Fix the things that make comparisons unfair. Measure the things that tell you how much to trust the estimate. Reproducible machine learning is not about bureaucracy. It is the difference between knowing whether your edit helped and guessing.

Make It a Habit

Pick one model you have already trained. Write a three-line experiment record for it: data version, seed, evaluation design, and the score spread across folds. Then rerun it twice with different seeds and watch how much the score moves.

That single exercise will show you the spread you have been ignoring. It will also give you a tangible artifact to carry into your next model comparison. The next time a score moves, you will know whether the movement came from your idea or from the luck of the draw.