advanced
11 min read
Derive AdaBoost’s Weight Updates From Exponential Loss
AdaBoost's algorithm listing looks like three separate rules stitched together: fit a weak learner, compute a coefficient, multiply the sample weights by…
Read tutorialvariant
Mathematical Theory
Articles whose authoritative article variant is theory and whose primary outcome is formal derivation, assumptions, or mathematical reasoning.
Tagged articles
Mathematical Theory articles
47 articles in this tag.
intermediate
11 min read
Derive the Bias–Variance–Noise Decomposition for Squared Error
Retrain the same model on a different random split and the predictions move. Nobody changed the code. Nobody changed the hyperparameters. The wobble is not…
Read tutorial
intermediate
9 min read
How Bootstrap Resampling Builds an Interval for a Statistic
You have a thousand bootstrap replicates sitting in a NumPy array. You have a histogram. What you do not have is the two numbers you actually need to…
Read tutorial
intermediate
8 min read
Why a Bootstrap Sample Leaves Observations Out of Bag
A random forest hands you an out-of-bag score, and someone nearby says, "Each tree leaves out about a third of the data." That number feels like a setting.…
Read tutorial
intermediate
11 min read
Decompose the Brier Score Into Reliability, Resolution, and Uncertainty
Two forecasters can post the same Brier score and deserve opposite verdicts. One is honest but says almost nothing. The other is sharp but lies about its…
Read tutorial
intermediate
9 min read
Derive the Cost-Sensitive Classification Threshold
A model returns 0.4. The default cutoff turns that into "negative," and nobody asked whether that was the right call.
Read tutorial
advanced
9 min read
How Certain Is a Cross-Validation Score? Derive the Limits of Fold-Based Uncertainty
Two models. Mean cross-validation scores of 0.842 and 0.847. The instinct is immediate: the second one wins.
Read tutorial
intermediate
11 min read
How Cumulative-Link Models Represent Ordered Outcomes
A satisfaction rating of 4 when the truth was 5 is not the same kind of mistake as predicting 1. Most models cannot tell the difference. This one can — but…
Read tutorial
intermediate
11 min read
How DBSCAN Forms Clusters: Density Reachability and Connectivity
Two points sit the same distance from a dense blob. DBSCAN labels one a cluster member and the other noise. That looks arbitrary until you stop asking how…
Read tutorial
intermediate
9 min read
Decision Tree Cost-Complexity Pruning: Derive the Fit–Size Tradeoff
A fully grown decision tree can score near-perfect on the data it was trained on and still lose to a single split on data it has never seen. The usual…
Read tutorial
intermediate
10 min read
How Decision Trees Score Classification Splits: Gini, Entropy, and Gain
You fit a tree, print it, and there it is: the root splits on worst radius, not mean texture. The library made a choice. You can't say why.
Read tutorial
beginner
10 min read
Deriving Gradient Updates From a Differentiable Model Objective
You can write theta = theta - lr * grad from memory and still freeze when someone asks where grad came from. That gap is the whole problem. The update rule…
Read tutorial
intermediate
8 min read
Deriving the Logistic Regression Classification Objective
You can fit a logistic regression in one line and read its coefficients in the next. What almost nobody shows you is where the loss function comes from.…
Read tutorial
intermediate
10 min read
Covariate, Label, and Concept Shift: Formal Definitions and Limits
A model degrades in production. Three engineers offer three diagnoses: "the inputs drifted," "the class balance changed," "the meaning changed." All three…
Read tutorial
intermediate
11 min read
Feature-Hashing Collisions: Probability, Buckets, and Memory Tradeoffs
You picked 2^18 buckets. You trained the model. The metrics came back slightly worse than your explicit-vocabulary baseline, and now you are staring at the…
Read tutorial
intermediate
10 min read
How Feature Scaling Changes Distance Geometry: A Mathematical Derivation
Two houses sit side by side in your table. One is 1,800 square feet with 3 bedrooms. The other is 2,000 square feet with 5 bedrooms. To a k-NN model, the…
Read tutorial
advanced
13 min read
Derive the EM Algorithm for Gaussian Mixture Models
You can write the Gaussian mixture density in a single line. The trouble starts the moment you take the log.
Read tutorial
advanced
11 min read
Gradient Boosting as Additive Function Fitting: Derive the Pseudo-Residuals
Fit the next tree to the error. That instruction works beautifully for squared loss and then quietly falls apart the moment you switch to log loss. The…
Read tutorial
intermediate
9 min read
When Gradient Descent Converges: Smoothness, Convexity, and Step Size
"Gradient descent works" and "gradient descent is guaranteed to converge" sound like the same sentence. They are not. One is a hope backed by experience.…
Read tutorial
intermediate
10 min read
Hierarchical Clustering Linkages: Define Distances and Work Through Merges
Change one parameter, get a different tree. Same points, same distance metric, same greedy loop — yet the dendrogram reorganizes itself. That parameter is…
Read tutorial
intermediate
9 min read
How Isolation Forest Turns Path Length Into an Anomaly Score
You fit an Isolation Forest, call decision_function, and get a column of numbers. You sort them, pick a cutoff, and ship the alerts. But if someone asks…
Read tutorial
intermediate
10 min read
Derive the K-Means Objective and Its Assignment–Update Steps
Fit K-means twice on the same data with the same k, and you can get two different clusterings. The second run often reports a lower cost than the first.…
Read tutorial
intermediate
12 min read
KNN Regression as Local Averaging: Neighborhood Size, Bias, and Variance
Fit KNN regression with k=1 and the prediction curve threads through every training point. Set k=50 and the same curve flattens toward a horizontal line.…
Read tutorial
intermediate
10 min read
Why Lasso Can Set Coefficients to Zero: An L1 Derivation
Fit a lasso on a dataset with a dozen features and something strange happens. A few coefficients come back as exactly 0.0. Not 0.0031. Not 0.0007. Zero.…
Read tutorial
intermediate
11 min read
Least-Squares Residuals as Projections: Why Orthogonality Appears
You fit a line, plot the residuals, and they look like a clean cloud. No funnel, no curve, no obvious outlier. It is tempting to read that plot as a…
Read tutorial
intermediate
10 min read
Linear Discriminant Analysis: Derive the Gaussian Classifier
You fit LinearDiscriminantAnalysis, plot the boundary, and get a straight line. Then someone asks why it is straight, and the honest answer is "because the…
Read tutorial
beginner
10 min read
Least-Squares Linear Regression: Derive the Objective and Solution
Call LinearRegression().fit(X, y) and coefficients appear in milliseconds. The library never shows you the equation it solved, why that equation has a…
Read tutorial
intermediate
11 min read
Derive Local Outlier Factor From Local Reachability Density
A point can sit closer to its neighbors than anything else in the dataset and still be the anomaly. That is the contradiction Local Outlier Factor was…
Read tutorial
intermediate
12 min read
MCAR, MAR, and MNAR: Formal Assumptions About Missing Data
Two analysts open the same CSV. Both see the same 12% of rows with a blank in the income column. One drops those rows and reports a mean of 54,000. The…
Read tutorial
intermediate
9 min read
Why the Best Validation Score Is Often Too Optimistic
You run a grid search. The leaderboard prints a winner at 0.91. You ship it, and the first honest measurement lands near 0.86. Nothing broke. The model did…
Read tutorial
intermediate
9 min read
Derive Softmax Probabilities and the Multiclass Cross-Entropy Objective
A multiclass model hands you K raw scores. The obvious move — divide each by their sum — collapses the first time a score goes negative. Here is the fix,…
Read tutorial
intermediate
11 min read
Naive Bayes: Derive the Classification Score From Bayes' Rule
The intuition article told you Naive Bayes "combines prior and evidence." Then you tried to write $P(x_1, x_2, \dots, x_n \mid y)$ for real data, and the…
Read tutorial
advanced
11 min read
When Is Least Squares Unbiased? Assumptions Behind OLS Guarantees
A coefficient can be unbiased on average and still be wrong in your one dataset. Worse, it can be unbiased and still predict badly, and unbiased and still…
Read tutorial
intermediate
10 min read
Why the Mean, Median, or Class Prevalence Is the Best Constant Baseline
You build a baseline, get a number, and have no idea whether that number is the best a constant can do or just a habit copied from a tutorial. Here is the…
Read tutorial
intermediate
11 min read
Derive PCA: Variance-Maximizing Directions and Covariance Eigenvectors
You can call PCA(), read explained_variance_ratio_, and get a working result. Then someone asks why the principal directions are eigenvectors of the…
Read tutorial
intermediate
10 min read
Derive the Poisson Regression Objective With a Log Link
A straight line on count data will eventually promise you a negative number of events. The Poisson likelihood with a log link is what stops it.
Read tutorial
intermediate
12 min read
How Class Prevalence Changes Precision and the PR Baseline
A model can keep the same weights, the same features, and the same threshold, and still report precision 0.9 on your test set and precision 0.3 in…
Read tutorial
intermediate
9 min read
Quantile Regression: Derive the Pinball-Loss Objective
Your model keeps under-predicting the values you actually care about, and averaging the errors away does not fix it.
Read tutorial
advanced
8 min read
Why Averaging Trees Helps: Derive the Random-Forest Variance Effect
Everyone repeats that a forest reduces variance. Almost nobody writes down the formula that governs the reduction — and that formula contains a term that…
Read tutorial
intermediate
11 min read
How Regression Trees Choose Splits: Derive Squared-Error Reduction
A classification tree asks which split makes the labels purest. A regression tree has no labels to purify — only numbers that scatter. So what is it…
Read tutorial
intermediate
11 min read
Ridge Regression: Derive the Penalized Least-Squares Solution
Two features that measure almost the same thing can make ordinary least squares produce coefficients that swing wildly while the predictions barely move.…
Read tutorial
advanced
11 min read
Split Conformal Prediction: Derive the Rank-Based Coverage Guarantee
A prediction interval built from a model's own residuals has no guarantee at all. Split conformal prediction replaces that hope with a finite-sample…
Read tutorial
intermediate
10 min read
Squared Error vs Huber Loss: How Large Residuals Change the Objective
One point sits far from the rest of the scatter, and the fitted line visibly tilts toward it. Move that point twice as far away. Squared error answers with…
Read tutorial
intermediate
9 min read
How Stacking Uses Out-of-Fold Predictions to Train a Meta-Model
Your stacking classifier reports 0.97 accuracy during training. On held-out data, it drops to 0.81 — barely better than the single best base model. Nothing…
Read tutorial
intermediate
12 min read
Support Vector Machines: Derive the Maximum-Margin Classifier
Most explanations of support vector machines stop at the picture: two clouds of points, a line between them, a gap on either side. That picture is correct,…
Read tutorial
advanced
10 min read
How the SVM Dual Makes Kernel Substitution Possible
The primal SVM asks for a weight vector in feature space. The trained model never hands you one. It hands you a sum over training points instead — and that…
Read tutorial
intermediate
11 min read
XGBoost’s Regularized Objective: Derive Leaf Weights and Split Gain
Two candidate splits sit in front of you. One carves the data into two clean, balanced groups. The other produces a lopsided partition that looks worse by…
Read tutorial