Why Lasso Can Set Coefficients to Zero: An L1 Derivation

Fit a lasso on a dataset with a dozen features and something strange happens. A few coefficients come back as exactly 0.0. Not 0.0031. Not 0.0007. Zero. Fit ridge on the same data and every coefficient survives, small but alive.

The usual explanation — "the penalty pushes small numbers down until they round to zero" — is wrong, and it hides the actual mechanism. Nothing is rounding. The optimizer is choosing a point where a coefficient is exactly zero because that point is genuinely optimal under the L1 penalty. To see why, we need the objective, the constraint geometry, and one small case where you can watch a coefficient cross the line.

This is the derivation. I'll assume you already have OLS normal equations and ridge shrinkage in hand — if not, the prerequisite articles cover both. Here we start from the lasso objective and follow it to the zero.

Set Up the Objective and the Notation

Fix the notation first, because every later symbol needs a concrete referent.

Let $X$ be the $n \times p$ design matrix — $n$ observations, $p$ features. Let $y$ be the $n$-vector of targets. Let $\beta$ be the $p$-vector of coefficients we are solving for. The residual vector is

The intercept is handled separately and is not penalized. Penalizing it would make the fit depend on where you happened to center the target, which is not a modeling choice you want the penalty making for you.

The lasso objective is

Read it term by term:

• $\frac{1}{2n}\|y - X\beta\|_2^2$ is the squared error — the fit term. It wants $\beta$ to explain the data.
• $\|\beta\|_1 = \sum_{j=1}^{p}|\beta_j|$ is the L1 norm — the penalty term. It wants $\beta$ to be small, and it charges for every nonzero entry.
• $\lambda \ge 0$ is the single knob trading one against the other.

Ridge uses the same framework with one structural change: the penalty becomes $\lambda\|\beta\|_2^2$. Same fit term, same $\lambda$, different norm. That single substitution is the entire source of the behavioral difference, and the rest of this article is about why.

Three assumptions carry the derivation:

1. Columns of $X$ are standardized (centered and scaled to unit variance). The L1 penalty charges absolute magnitude, so if one feature is measured in millimeters and another in kilometers, you are penalizing units, not signal.
2. $\lambda \ge 0$. A negative penalty would reward large coefficients, which is not regularization.
3. No perfect collinearity in $X$, so the solution is unique.

Two Equivalent Views: Penalty and Constraint

The penalized form above is convenient for computation. The constrained form is convenient for geometry, and they describe the same solution set.

For every $\lambda$ there exists some $t$ that yields the same solution, and vice versa. This is Lagrangian duality — I am stating it, not proving it here. The practical content is the direction of the knob: larger $\lambda$ corresponds to smaller $t$, which means more shrinkage and more zeros.

Now picture the two pieces.

The squared-error term draws ellipsoids around the OLS solution $\hat{\beta}$. Each ellipsoid is a contour of equal error: points on the same ellipse fit the data equally well. The center is $\hat{\beta}$, the unpenalized optimum.

The constraint draws a region around the origin. For L1 that region is a diamond in two dimensions; for L2 it is a circle.

The lasso solution is the first point where a growing error ellipsoid touches the constraint region. Start with a tiny ellipsoid near $\hat{\beta}$ — it misses the constraint entirely. Inflate it. The moment it makes contact, you have found the constrained optimum. That contact point is the answer.

Everything that follows is about the shape of the thing the ellipsoid touches.

Why the Diamond Has Corners and the Circle Does Not

In two dimensions, the L1 ball $\{|\beta_1| + |\beta_2| \le t\}$ is a diamond with vertices sitting on the axes at $(t, 0)$, $(0, t)$, $(-t, 0)$, and $(0, -t)$. In $p$ dimensions it becomes a cross-polytope, and its vertices and low-dimensional faces lie on coordinate subspaces — places where one or more coefficients are exactly zero.

The L2 ball $\{\beta_1^2 + \beta_2^2 \le t^2\}$ is a smooth sphere. No vertices. No flat faces. Its boundary curves everywhere, so an expanding ellipsoid generically meets it at a single tangent point in the interior of a face — a spot where every coordinate is nonzero. Coordinate-zero points do exist on the sphere (the axis points), but the smooth boundary gives the ellipsoid no reason to prefer them.

Here is the touch argument. An expanding ellipsoid generically first contacts a smooth surface at a tangent point in the interior of a face — a spot where every coordinate is nonzero. It can contact a corner exactly, and corners sit where some $\beta_j = 0$.

Be honest about the word "generically." The corner is not guaranteed. But it is a positive-probability event for L1 and a measure-zero event for L2. That asymmetry is the whole story: the diamond offers the ellipsoid a set of zero-coordinate landing pads, and the circle offers none.

Note: The corner is not the only way to get a zero. In higher dimensions the ellipsoid can land on an edge or a face of the cross-polytope, and any such contact pins at least one coefficient to zero. The vertex is just the easiest picture to hold in your head.

The Coordinate-Wise View: Soft Thresholding

The geometry is convincing, but I do not want you to take sparsity on faith from a picture. There is a second route to the same conclusion, and it is algebraic.

Take the special case where the design is orthonormal: $X^\top X = I$. This is a teaching device, not the general case, but it strips the problem down to its mechanism. Under orthonormality the lasso solution reduces to

Read that as a mechanism, not a formula. Take the OLS coefficient. Subtract $\lambda$ from its magnitude. If nothing is left, the coefficient is exactly zero — not small, zero. This is soft thresholding, and it is the cleanest statement of lasso coefficient thresholding:

Compare ridge under the same orthonormal design:

Ridge divides. Lasso subtracts. Division by a finite number never reaches zero, no matter how large $\lambda$ grows. Subtraction crosses zero and stops there. That is the difference between shrinkage and selection, and it is the same fact the diamond was showing you.

Common mistake: Treating the orthonormal result as the general rule. With correlated predictors, the threshold is no longer a clean per-coefficient comparison against $\lambda$. The next section shows what actually happens.

A Small Worked Case: Shrinkage, Then a Zero

Let's make it concrete. Two standardized predictors, and suppose the OLS solution is

I am choosing these numbers so the arithmetic stays hand-checkable. In the orthonormal case, soft thresholding gives the lasso solution directly:

Walk $\lambda$ upward and watch what happens.

At $\lambda = 0.5$ the second coefficient hits zero, because $|\hat{\beta}_2^{\text{OLS}}| = 0.5 \le \lambda$. The threshold condition is satisfied exactly, and the coefficient does not pass through zero into negative territory — it stops. For any $\lambda > 0.5$, feature 2 is gone from the model. At $\lambda = 2.0$, feature 1 follows.

That moment at $\lambda = 0.5$ is what "selection" means operationally: the active feature set changes. Before it, two features carry weight. After it, one does. The model did not merely get smaller; its structure changed.

One more piece of context worth knowing: the full solution path $\hat{\beta}(\lambda)$ is piecewise linear in $\lambda$. Between threshold crossings, each coefficient moves along a straight line. That is why path algorithms can compute the entire sequence of solutions efficiently rather than refitting from scratch at every $\lambda$ — but that is an implementation detail, not part of the derivation.

What the Derivation Does Not Promise

A zero coefficient is a statement about the objective at a particular $\lambda$. It is not a statement about the world. Five limits matter.

Correlated predictors. With near-duplicate columns, lasso tends to keep one and zero the other, and which one it picks can flip across resamples. The predictions may be stable while the selection is not. If your features come in correlated groups, this instability is the norm, not an edge case.

Sparse is not important. A zero means "this coefficient was not worth its L1 cost at this $\lambda$." It does not mean the variable is irrelevant. A feature can be genuinely predictive and still be zeroed because a correlated neighbor already carries the signal.

Scaling matters. Penalizing raw coefficients on unstandardized features penalizes measurement units. Standardize first, or the penalty is doing something you did not intend.

Recovery has conditions. Exact support recovery — getting the true nonzero set back — requires enough samples, low noise, and a design that is not too correlated. When those conditions fail, L1 selection can behave close to random. The number of samples needs to be sufficiently large relative to the number of true nonzero coefficients, the noise level, and the correlation structure of $X$.

$\lambda$ is a modeling decision. Cross-validation tends to under-penalize, keeping a few irrelevant features because they do not hurt prediction. BIC-style criteria tend to over-penalize. Neither is a truth oracle; they optimize different things.

When to Reach for Lasso, and When Not To

Convert the mechanism into a decision rule.

Reach for lasso when $p$ is large relative to $n$, you suspect only a subset of features matter, and you want a sparse, inspectable model. The corners of the L1 ball are doing useful work for you.

Prefer ridge when predictors are many and correlated and you mainly care about prediction stability rather than selection. Ridge will not hand you a feature list, but it will not hand you an unstable one either.

Consider elastic net when you want sparsity but need correlated groups to enter or leave together. It blends the L1 and L2 penalties, which is the natural extension when the diamond's corners are too sharp for your data.

And do not treat a lasso fit as a causal or scientific conclusion. Treat it as a hypothesis generator. The zero coefficients tell you which features might deserve attention; held-out validation tells you whether that attention pays off.

Tip: Put scaling and the penalty inside a pipeline so the same transformation is applied at fit and predict time. A scaler fitted on training data and forgotten at prediction time is a quiet, recurring bug.

Where to Go Next

The zero coefficients come from the corners of the L1 ball, and the soft-thresholding formula is the same fact written in algebra. Two views, one mechanism.

Here is the next move I would make. Take a small standardized dataset, fit a lasso, and sweep $\lambda$ across a wide range. Plot the coefficient path — $\lambda$ on a log scale against each coefficient. Watch a line dive to zero and stay there. That picture will do more for your intuition than any derivation, because you will have produced it yourself.

Then apply the decision rule: use the sparsity as a hypothesis about which features deserve attention, and verify it out of sample before you believe it.