advanced
11 min read
Derive AdaBoost’s Weight Updates From Exponential Loss
AdaBoost's algorithm listing looks like three separate rules stitched together: fit a weak learner, compute a coefficient, multiply the sample weights by…
Read tutorialconcept
Optimization
Objectives and procedures for fitting model parameters, including losses, gradients, iterative updates, and convergence behavior.
Tagged articles
Optimization articles
27 articles in this tag.
intermediate
11 min read
K-Means Clustering Explained: Group Data by Proximity, Then Question the Groups
K-means does not discover the real categories hiding in your data. It draws geometric boundaries around points that happen to sit close together. Those…
Read tutorial
intermediate
9 min read
Decision Tree Cost-Complexity Pruning: Derive the Fit–Size Tradeoff
A fully grown decision tree can score near-perfect on the data it was trained on and still lose to a single split on data it has never seen. The usual…
Read tutorial
beginner
10 min read
Deriving Gradient Updates From a Differentiable Model Objective
You can write theta = theta - lr * grad from memory and still freeze when someone asks where grad came from. That gap is the whole problem. The update rule…
Read tutorial
intermediate
8 min read
Deriving the Logistic Regression Classification Objective
You can fit a logistic regression in one line and read its coefficients in the next. What almost nobody shows you is where the loss function comes from.…
Read tutorial
advanced
13 min read
Derive the EM Algorithm for Gaussian Mixture Models
You can write the Gaussian mixture density in a single line. The trouble starts the moment you take the log.
Read tutorial
intermediate
11 min read
Gradient Boosting Explained: How Small Corrections Become a Strong Model
Random forests grow many trees independently and average their votes. Gradient boosting grows trees one at a time, each one aimed at the mistakes the…
Read tutorial
advanced
11 min read
Gradient Boosting as Additive Function Fitting: Derive the Pseudo-Residuals
Fit the next tree to the error. That instruction works beautifully for squared loss and then quietly falls apart the moment you switch to log loss. The…
Read tutorial
beginner
12 min read
Gradient Descent Explained: How Models Improve One Step at a Time
A model starts with random parameters and ends up making sharp predictions. Nobody tells it the answer. It just keeps nudging itself in the right direction…
Read tutorial
intermediate
9 min read
When Gradient Descent Converges: Smoothness, Convexity, and Step Size
"Gradient descent works" and "gradient descent is guaranteed to converge" sound like the same sentence. They are not. One is a hope backed by experience.…
Read tutorial
beginner
9 min read
Visualize Gradient-Descent Step Sizes on a Simple Objective
You set a learning rate, hit run, and the loss curve does something strange. Maybe it barely moves. Maybe it bounces. Maybe it turns into nan. The number…
Read tutorial
intermediate
10 min read
Derive the K-Means Objective and Its Assignment–Update Steps
Fit K-means twice on the same data with the same k, and you can get two different clusterings. The second run often reports a lower cost than the first.…
Read tutorial
intermediate
10 min read
Why Lasso Can Set Coefficients to Zero: An L1 Derivation
Fit a lasso on a dataset with a dozen features and something strange happens. A few coefficients come back as exactly 0.0. Not 0.0031. Not 0.0007. Zero.…
Read tutorial
intermediate
11 min read
Least-Squares Residuals as Projections: Why Orthogonality Appears
You fit a line, plot the residuals, and they look like a clean cloud. No funnel, no curve, no obvious outlier. It is tempting to read that plot as a…
Read tutorial
beginner
11 min read
Linear Regression From First Principles: Fit a Line, Read the Error
Most introductions to linear regression make it sound like drawing a line through some dots. That description is true and almost useless. Any line can be…
Read tutorial
beginner
10 min read
Least-Squares Linear Regression: Derive the Objective and Solution
Call LinearRegression().fit(X, y) and coefficients appear in milliseconds. The library never shows you the equation it solved, why that equation has a…
Read tutorial
intermediate
10 min read
Why the Mean, Median, or Class Prevalence Is the Best Constant Baseline
You build a baseline, get a number, and have no idea whether that number is the best a constant can do or just a habit copied from a tutorial. Here is the…
Read tutorial
intermediate
11 min read
Derive PCA: Variance-Maximizing Directions and Covariance Eigenvectors
You can call PCA(), read explained_variance_ratio_, and get a working result. Then someone asks why the principal directions are eigenvectors of the…
Read tutorial
intermediate
10 min read
Derive the Poisson Regression Objective With a Log Link
A straight line on count data will eventually promise you a negative number of events. The Poisson likelihood with a log link is what stops it.
Read tutorial
intermediate
9 min read
Quantile Regression: Derive the Pinball-Loss Objective
Your model keeps under-predicting the values you actually care about, and averaging the errors away does not fix it.
Read tutorial
intermediate
11 min read
How Regression Trees Choose Splits: Derive Squared-Error Reduction
A classification tree asks which split makes the labels purest. A regression tree has no labels to purify — only numbers that scatter. So what is it…
Read tutorial
intermediate
11 min read
Ridge Regression: Derive the Penalized Least-Squares Solution
Two features that measure almost the same thing can make ordinary least squares produce coefficients that swing wildly while the predictions barely move.…
Read tutorial
intermediate
10 min read
Squared Error vs Huber Loss: How Large Residuals Change the Objective
One point sits far from the rest of the scatter, and the fitted line visibly tilts toward it. Move that point twice as far away. Squared error answers with…
Read tutorial
intermediate
12 min read
Support Vector Machines: Derive the Maximum-Margin Classifier
Most explanations of support vector machines stop at the picture: two clouds of points, a line between them, a gap on either side. That picture is correct,…
Read tutorial
advanced
10 min read
How the SVM Dual Makes Kernel Substitution Possible
The primal SVM asks for a weight vector in feature space. The trained model never hands you one. It hands you a sum over training points instead — and that…
Read tutorial
intermediate
9 min read
XGBoost Explained: Why Gradient Boosting Became a Practical Workhorse
XGBoost has a reputation problem. It gets treated like a secret weapon—a mysterious algorithm that wins competitions and powers production systems while…
Read tutorial
intermediate
11 min read
XGBoost’s Regularized Objective: Derive Leaf Weights and Split Gain
Two candidate splits sit in front of you. One carves the data into two clean, balanced groups. The other produces a lopsided partition that looks worse by…
Read tutorial