The Shape of Occam's Razor: Demystifying L1 and L2 Regularization

The Disease: Exploding Weights and Memorization

We've all been there. You train a deep model on a small dataset, your training loss plummets to exactly zero, and your test loss explodes into the stratosphere. Your model didn't learn the underlying physics of the data; it just memorized the noise.

If you peek under the hood of a heavily overfitted model, you'll almost always observe a specific symptom: the weights are massive. To perfectly fit every single outlier and noisy data point, the model creates a wildly oscillating decision boundary. Mathematically, wild oscillations require huge coefficients.

We need a mathematical formulation of Occam's Razor: given two models that explain the data equally well, prefer the simpler one. But how do we define "simple"? In machine learning, we define simplicity by restricting the size or the number of the weights.

This is where regularization comes in. Instead of just minimizing the loss, we minimize the loss plus a penalty on the weights.

The two most famous penalties are L1 (Lasso) and L2 (Ridge). But where do they come from, and why do they behave so differently?

The Geometric View: Lagrangian Constraints

To understand how L1 and L2 work, we need to view them through the lens of constrained optimization. Imagine you are trying to minimize your mean squared error (MSE), but I put a strict budget on your weights. You can only use a total "weight budget" of $C$.

By the method of Lagrange multipliers, minimizing $\text{Loss} + \lambda \cdot \text{Penalty}$ is mathematically equivalent to minimizing the Loss subject to a hard constraint on the Penalty. The $\lambda$ hyperparameter directly controls the radius $C$ of our constraint region.

• L2 Regularization (Ridge): Constrains the squared sum of weights: $w_1^2 + w_2^2 \leq C$. Geometrically, this is a circle (or a hypersphere in higher dimensions).
• L1 Regularization (Lasso): Constrains the absolute sum of weights: $|w_1| + |w_2| \leq C$. Geometrically, this is a diamond (or a cross-polytope).

Let's visualize how the loss landscape interacts with these constraint regions.

The Probabilistic View: MAP Estimation

Geometry is beautiful, but probability tells a deeper story. What if I told you that regularizing your weights is mathematically identical to applying a Bayesian prior to your model?

In standard Maximum Likelihood Estimation (MLE), we try to find the weights $\theta$ that maximize the probability of our data given the weights: $P(D | \theta)$. But Bayes' Theorem tells us we should actually maximize the posterior—the probability of the weights given the data:

This is called Maximum A Posteriori (MAP) estimation. $P(\theta)$ is our prior belief about the weights before seeing any data. If we assume our weights should be small and centered around zero, we can assign a probability distribution to them. To maximize this, we minimize the negative log of the numerator:

Notice the structure? It perfectly matches our regularization equation! The negative log-likelihood $-\log P(D | \theta)$ is our Loss function (like MSE or Cross-Entropy). The negative log-prior $-\log P(\theta)$ is our Penalty term.

When L1 and L2 Fail: The Elastic Net

So, we use L2 to prevent explosion, and L1 when we want sparsity (feature selection). But the real world is messy, and both methods have fatal flaws.

If you have a dataset where "Feature A" and "Feature B" are practically identical (collinear), Lasso (L1) will arbitrarily push the weight of one to zero and keep the other. This makes your model highly sensitive to the specific training sample. Ridge (L2) will smoothly divide the weight equally between both.

To get the best of both worlds, Zou and Hastie introduced the Elastic Net in 2005. It's elegantly simple—just add both penalties together:

Geometrically, the constraint region looks like a diamond with rounded edges. It retains the corners needed to drive weights to zero, but the slight curve stabilizes the selection of correlated features.

Writing Custom Regularizers From Scratch

In modern deep learning libraries like PyTorch, L2 regularization is often hidden behind the weight_decay parameter in optimizers like Adam or SGD. But what if you want to implement L1, or an Elastic Net, or a custom penalty for your PDE solver?

To really understand it, you should implement the loss penalty directly during your forward pass. Here is how you write a custom Elastic Net regularization hook in pure PyTorch.

import torch
import torch.nn as nn

def elastic_net_penalty(model: nn.Module, l1_ratio: float, alpha: float):
    """
    Computes Elastic Net penalty for all weights in the model.
    alpha: Overall strength of the regularization
    l1_ratio: Mix between L1 (1.0) and L2 (0.0)
    """
    l1_loss = torch.tensor(0.0, requires_grad=True).to(next(model.parameters()).device)
    l2_loss = torch.tensor(0.0, requires_grad=True).to(next(model.parameters()).device)
    
    for name, param in model.named_parameters():
        # We typically only regularize weights, not biases
        if 'bias' not in name:
            l1_loss = l1_loss + torch.sum(torch.abs(param))
            l2_loss = l2_loss + torch.sum(torch.square(param))
            
    # Elastic Net formula
    penalty = alpha * (l1_ratio * l1_loss + (1 - l1_ratio) * l2_loss)
    return penalty

# Training Loop Example
for data, target in dataloader:
    optimizer.zero_grad()
    
    output = model(data)
    base_loss = criterion(output, target)
    
    # Compute our custom penalty
    reg_loss = elastic_net_penalty(model, l1_ratio=0.5, alpha=1e-4)
    
    # Total loss is Data Loss + Prior (Penalty)
    total_loss = base_loss + reg_loss
    
    total_loss.backward()
    optimizer.step()
      

By decoupling the penalty from the optimizer, you gain full mathematical control. You can track exactly how much of your loss is coming from the data vs. the prior, and you can easily experiment with custom penalties (like smoothly regularizing spatial derivatives in Physics-Informed Neural Networks).