advanced
11 min read
Derive AdaBoost’s Weight Updates From Exponential Loss
AdaBoost's algorithm listing looks like three separate rules stitched together: fit a weak learner, compute a coefficient, multiply the sample weights by…
Read tutorialconcept
Loss Functions
Objective functions that quantify prediction penalties and shape fitted parameters, model behavior, or the tradeoff among error types.
Tagged articles
Loss Functions articles
18 articles in this tag.
intermediate
9 min read
Derive the Cost-Sensitive Classification Threshold
A model returns 0.4. The default cutoff turns that into "negative," and nobody asked whether that was the right call.
Read tutorial
intermediate
8 min read
Deriving the Logistic Regression Classification Objective
You can fit a logistic regression in one line and read its coefficients in the next. What almost nobody shows you is where the loss function comes from.…
Read tutorial
advanced
11 min read
Gradient Boosting as Additive Function Fitting: Derive the Pseudo-Residuals
Fit the next tree to the error. That instruction works beautifully for squared loss and then quietly falls apart the moment you switch to log loss. The…
Read tutorial
intermediate
10 min read
Derive the K-Means Objective and Its Assignment–Update Steps
Fit K-means twice on the same data with the same k, and you can get two different clusterings. The second run often reports a lower cost than the first.…
Read tutorial
beginner
11 min read
Linear Regression From First Principles: Fit a Line, Read the Error
Most introductions to linear regression make it sound like drawing a line through some dots. That description is true and almost useless. Any line can be…
Read tutorial
beginner
10 min read
Least-Squares Linear Regression: Derive the Objective and Solution
Call LinearRegression().fit(X, y) and coefficients appear in milliseconds. The library never shows you the equation it solved, why that equation has a…
Read tutorial
intermediate
12 min read
Logistic Regression for Classification: From Scores to Probabilities
A logistic regression prediction is not a label. It is a probability—and the label is a decision you make on top of that probability.
Read tutorial
beginner
9 min read
Loss Functions in Machine Learning: Tell the Model Which Errors Matter
Most people assume a machine learning model learns to be right. It doesn't. A model learns to minimize a number you hand it—and that number is the loss…
Read tutorial
intermediate
9 min read
Derive Softmax Probabilities and the Multiclass Cross-Entropy Objective
A multiclass model hands you K raw scores. The obvious move — divide each by their sum — collapses the first time a score goes negative. Here is the fix,…
Read tutorial
intermediate
10 min read
Why the Mean, Median, or Class Prevalence Is the Best Constant Baseline
You build a baseline, get a number, and have no idea whether that number is the best a constant can do or just a habit copied from a tutorial. Here is the…
Read tutorial
intermediate
10 min read
Derive the Poisson Regression Objective With a Log Link
A straight line on count data will eventually promise you a negative number of events. The Poisson likelihood with a log link is what stops it.
Read tutorial
intermediate
9 min read
Quantile Regression: Derive the Pinball-Loss Objective
Your model keeps under-predicting the values you actually care about, and averaging the errors away does not fix it.
Read tutorial
intermediate
11 min read
How Regression Trees Choose Splits: Derive Squared-Error Reduction
A classification tree asks which split makes the labels purest. A regression tree has no labels to purify — only numbers that scatter. So what is it…
Read tutorial
beginner
11 min read
Regression vs Classification: Identify the Prediction Problem First
You have data. You have a prediction goal. And you're stuck on a question that feels like it should be simple: should I use regression or classification?
Read tutorial
intermediate
11 min read
Ridge Regression: Derive the Penalized Least-Squares Solution
Two features that measure almost the same thing can make ordinary least squares produce coefficients that swing wildly while the predictions barely move.…
Read tutorial
beginner
12 min read
Test How an Extreme Observation Changes a Regression Fit
One row can steer a line. Add a single extreme observation to an otherwise clean dataset, refit, and watch the slope tilt — even though every other point…
Read tutorial
intermediate
10 min read
Squared Error vs Huber Loss: How Large Residuals Change the Objective
One point sits far from the rest of the scatter, and the fitted line visibly tilts toward it. Move that point twice as far away. Squared error answers with…
Read tutorial