The Mathematical Pivot: Conquering the 50% Sparsity Wall in Spiking Neural Networks

Why Spiking Neural Networks, and Why Do They Need Pruning?

Modern artificial intelligence runs on enormous amounts of power. A single training run for a large language model can consume the electricity of hundreds of households for months. At the other end of the spectrum, implantable medical sensors, wildlife trackers, and edge robotics controllers must run for years on a watch battery. This tension between capability and efficiency is perhaps the defining engineering challenge of our time.

Spiking Neural Networks (SNNs) offer a biologically grounded answer. Rather than computing dense, floating-point activations on every forward pass — as conventional ANNs do — SNNs communicate through discrete binary spikes, just like the neurons in your brain. Most neurons are silent at any given instant, so computation is inherently sparse. On dedicated neuromorphic processors like Intel's Loihi or IBM's TrueNorth, this translates directly to energy savings of several orders of magnitude compared to conventional GPUs.

But deploying even a modest SNN onto the tightest neuromorphic hardware requires pushing far beyond the network's natural sparsity. You need to prune it — systematically remove synaptic connections until the model is lean enough to fit the resource envelope of the target chip, all without catastrophic loss of accuracy. This is easy to state, surprisingly hard to do correctly, and the subject of everything that follows.

Sparsity in SNNs is not just about compression — it is about matching the topology of biological neural circuits, where the neocortex operates at roughly 1–10% synaptic activity at any given moment. — Motivation for this research direction

The Architecture: A Three-Layer Fully-Connected SNN

The network we built for this research is intentionally simple — a three-layer fully-connected architecture, clean enough to reason about mathematically but complex enough to reveal real phenomena. It uses Leaky Integrate-and-Fire (LIF) neurons throughout, implemented in PyTorch via the snntorch library.

784 → 256 → 128 → 10 · LIF neurons · β = 0.9 decay · dropout = 0.2

The Leaky Integrate-and-Fire Neuron: How It Actually Works

The LIF model is the workhorse of computational neuroscience. Unlike an ANN neuron that applies a nonlinearity and immediately passes a floating-point value forward, an LIF neuron maintains an internal membrane potential $U(t)$ that accumulates charge over time. The membrane evolves according to:

Discrete-time membrane dynamics. β ∈ (0,1) is the decay factor; W is the synaptic weight; s_in is the incoming spike (0 or 1).

Whenever the membrane potential exceeds a threshold $U_{\text{thr}}$, the neuron fires — emitting a binary spike $s[t] = 1$ — and immediately resets: $U[t] \leftarrow U[t] - U_{\text{thr}}$. Otherwise it remains silent ($s[t] = 0$). In our network, we set $\beta = 0.9$, meaning the membrane retains 90% of its charge between timesteps — a fairly slow decay, appropriate for temporal integration over $T = 10$ simulation steps.

Classification happens by accumulating spike counts over all ten timesteps. The output class is simply the neuron that spikes the most: $\hat{y} = \arg\max_i \sum_{t=1}^{T} s_i^{\text{out}}[t]$.

Why Direct Current Injection, Not Poisson Rate Encoding?

The canonical way to convert a continuous-valued image into SNN-compatible spikes is Poisson rate encoding: treat each pixel's intensity as a firing probability, and for each timestep sample a Bernoulli random variable. This is biologically elegant. It is also, in our case, completely useless.

We tried Poisson encoding first. After 12 training epochs the SNN achieved 12.6% test accuracy — indistinguishable from random chance for a 10-class problem. The diagnosis is straightforward: Poisson encoding requires inputs normalized to $[0,1]$ and interprets them as independent probabilities. Our structured data has correlations across features that the Bernoulli sampling destroys entirely. The temporal information is annihilated before it reaches the first LIF layer.

Baseline Results: SNN vs. ANN

The SNN outperforming its ReLU counterpart is a meaningful sanity check — it confirms that the LIF dynamics are genuinely contributing signal, not just noisily approximating the ANN's computation.

Quantifying Importance: Criticality Scores and Temporal Redundancy

Before we can prune anything sensibly, we need a principled answer to: which connections actually matter? Raw weight magnitude — the default in most off-the-shelf pruning libraries — is a rough proxy at best. A weight can be small because it carries little information, or it can be small because the network has learned to use precise, low-amplitude signals. Discarding these naively is a recipe for accuracy collapse at high sparsity.

Surrogate Gradient Criticality

SNNs introduce a fundamental complication: the spike function $s[t] = \mathbf{1}[U[t] \geq U_{\text{thr}}]$ is a step function, whose true derivative is zero almost everywhere and infinite at the threshold. Standard backpropagation fails completely.

The field's solution is the surrogate gradient: during the backward pass, replace the step function's derivative with a smooth surrogate, typically the arctan function: $\sigma'_{\text{surr}}(U) = \frac{1}{\pi} \cdot \frac{1}{1 + (\pi U)^2}$. This is mathematically dishonest but empirically robust — it preserves gradient flow while maintaining the spike's binary character during the forward pass.

We exploit these surrogate gradients directly to score neuron importance. For each input feature $i$, we define its criticality score as:

Criticality of input neuron i: mean absolute surrogate gradient of the loss with respect to the incoming weight column. Accumulated over 6–8 training batches.

The scores are then min-max normalized to $[0, 1]$ and saved. What we found was striking: the minimum criticality score across all 784 input features was 0.649. Every single input neuron — including the noisy redundant ones — was being meaningfully used. The SNN had learned a highly distributed representation, which has important implications for how aggressively we can prune.

Temporal Redundancy: Finding the Decision Timestep via KL-Divergence

The second axis of analysis is time. With $T = 10$ simulation steps, the natural question is: does the network actually need all ten? Perhaps it makes its decision early and the remaining steps are just inertia in the membrane potentials.

We quantify this using Kullback-Leibler divergence between consecutive timestep output distributions. For each timestep transition $t \to t+1$, we compute how much the output spike distribution changes:

KL divergence between output spike distributions at consecutive timesteps, averaged over a test batch. High KL = the network is still "learning" from that step.

By accumulating these divergences and finding where 95% of the total is reached, we identify the decision timestep. Our network hit this threshold at step 9 — meaning the final timestep contributes barely 5% of cumulative information. Truncating it gives a 10% theoretical energy saving on neuromorphic hardware with zero architectural change and zero weight modification.

The Crisis: The 50% Sparsity Wall and the CVXPY Collapse

With our criticality scores in hand, we had what seemed like a principled approach. Formulate network pruning as a constrained optimization problem: minimize the magnitude of pruned weights while enforcing hard constraints that protect critical neurons. This is the Criticality-Constrained Quadratic Pruning (CQP) formulation, solved via CVXPY using the OSQP solver.

The CQP Formulation

Let $m_i \in [0,1]$ be a continuous relaxation of the pruning mask for input neuron $i$ (where $m_i = 0$ means pruned, $m_i = 1$ means kept). Let $\|\mathbf{w}^{(1)}_i\|_1$ be the L1-norm of the weight column connecting input $i$ to the first hidden layer. The problem is:

CQP formulation. The objective minimizes squared weight magnitude of kept weights; constraints enforce sparsity target and protect high-criticality neurons with threshold τ = 0.9.

The objective is convex (quadratic in $\mathbf{m}$), the constraints are all linear, so CVXPY can hand this to OSQP and get a globally optimal solution. Or so we thought.

The Results That Broke Our Understanding

Autopsy: The Continuous Relaxation Trap

Network pruning is, in its pure form, a Mixed-Integer Programming (MIP) problem. Each weight either exists or it does not — $m_i \in \{0, 1\}$. MIP is NP-hard in the general case; you cannot solve it to optimality for 200,704 variables (the weight count of fc1 alone) in any practical timeframe.

CVXPY's strategy is continuous relaxation: allow $m_i \in [0,1]$ instead of demanding binary values. The resulting quadratic program is solvable in seconds. The implicit assumption is that the solver will return a nearly-binary solution — values close to 0 or 1 that you can safely round to integers.

This assumption fails catastrophically here, and the lambda sweep proved it beyond doubt.

The Smoking Gun: λ = 0.0 Produces 62% Sparsity

We tried adding a soft penalty term to the objective instead of hard constraints, sweeping a regularization coefficient λ:

Soft-constraint CQP: λ controls how strongly high-criticality neurons are penalized for being pruned.

At $\lambda = 0.0$, the penalty term vanishes entirely. The solver sees only the sparsity constraint and minimizes weight magnitude. This should have been mathematically equivalent to pure magnitude pruning at exactly 50% sparsity — accuracy ≈ 91.7%, actual sparsity ≈ 50.0%.

This is the Continuous Relaxation Trap in its purest form. The continuous problem has a different optimal solution than the integer problem it is meant to approximate. When you discretize the fractional solution, you are not solving the original problem — you are solving a garbled version of it. The bigger the gap between the LP relaxation and the true MIP, the worse the damage from rounding.

Why the Gap Is So Large Here

The LIF neuron's surrogate gradient creates a landscape where small differences in mask values lead to disproportionate differences in spike patterns. A mask value of 0.49 vs 0.51 looks like a rounding error in the continuous domain; in the binary domain it means the synapse is either completely severed or completely intact. With 200,704 such decisions being made simultaneously, the accumulated rounding error is catastrophic.

The Pivot: Combined Importance Scoring and Iterative Healing

We scrapped CVXPY entirely. The new approach stays completely within PyTorch and solves the binary assignment problem exactly — not approximately — by construction.

Combined Importance Scoring

The insight is simple: if you sort $n$ scalars and take the bottom $k$, you have exactly $k/n$ sparsity. No rounding, no relaxation, no approximation. The only question is: what scalar should you sort by?

We combine weight magnitude and criticality into a single metric, parameterized by a hyperparameter $\alpha$ that controls how strongly criticality modulates the base magnitude signal:

Combined importance score for input neuron i. α = 1.0 gives equal weight to magnitude and criticality; α → 0 recovers pure magnitude pruning; α → ∞ approaches pure criticality pruning.

To prune to sparsity $s$, we compute importance scores for all 784 input neurons, sort ascending, take the bottom $\lfloor 784 \times s \rfloor$ indices, and set their weight columns to zero. The sparsity target is hit exactly — mathematically guaranteed by construction, with zero rounding error.

def cqp_prune(model, w_norms, crit_scores, sparsity, alpha=1.0):
    # Combined importance: weight magnitude × criticality^alpha
    importance = w_norms * (crit_scores ** alpha)

    # Sort ascending — least important neurons get pruned
    n_prune = int(len(importance) * sparsity)
    prune_idx = np.argsort(importance)[:n_prune]

    # Build binary mask — exact sparsity, no rounding
    mask = np.ones(len(importance), dtype=np.float32)
    mask[prune_idx] = 0.0

    # Apply: zero the weight columns of pruned neurons
    pruned_model = copy.deepcopy(model)
    with torch.no_grad():
        pruned_model.fc1.weight.data *= torch.tensor(mask).unsqueeze(0)

    actual_sparsity = 1.0 - mask.mean()
    return pruned_model, actual_sparsity, mask

The Problem with One-Shot Pruning: Gradient Staleness

Even with exact sparsity, a single one-shot cut at 50% still only yielded 57.5% accuracy. The problem is what we called gradient staleness: the criticality scores we used to make the pruning decision were computed on the unmodified, unpruned network. The moment you remove 50% of the connections in fc1, the information flow through the network changes radically. The gradients you measured before the pruning are no longer valid descriptions of which remaining neurons matter most.

It is like drawing a map of a road network, then demolishing half the roads, and using the old map to decide which additional roads to remove. The map is stale the moment the demolition begins.

The Fix: Iterative Pruning with Fresh Gradient Recalculation

The solution is iterative: prune a small fraction, fine-tune for a few epochs so the network adapts, recalculate fresh criticality scores on the current network state, and repeat. Each pruning step operates on up-to-date gradient information, so the importance estimates are always valid for the network as it actually exists at that moment.

def iterative_cqp_prune(baseline_model, target_sparsity, trainloader, steps=3):
    # Gradually prune to target, fine-tuning and re-scoring between steps
    current_model = copy.deepcopy(baseline_model)

    # Sparsity schedule: 0 → target in `steps` equal increments
    sparsities = np.linspace(0, target_sparsity, steps + 1)[1:]

    for s in sparsities:
        # Step 1: recalculate criticality on the CURRENT network state
        fresh_scores = get_fresh_criticality_scores(current_model, trainloader)
        current_w_norms = (current_model.fc1.weight.data
                           .abs().mean(dim=0).cpu().numpy())

        # Step 2: prune to sparsity s using fresh scores
        current_model, actual_s, mask = cqp_prune(
            current_model, current_w_norms, fresh_scores,
            sparsity=s, alpha=1.0
        )

        # Step 3: fine-tune with masked gradients to heal the network
        fine_tune_with_masked_gradients(current_model, mask, trainloader)

    return current_model, actual_s

The sparsity schedule uses numpy.linspace to carve equal-sized increments from 0% to the target. For a 60% target with 3 steps: 20% → 40% → 60%. Each intermediate network is a valid, fine-tuned network rather than a brutally truncated one, so the importance estimates at each step are computed on a well-conditioned model.

The Zombie Weights Bug: When Adam Raises the Dead

With the iterative pruning loop implemented, we ran the first 60% sparsity experiment and watched in confusion as accuracy did not converge where expected. Pruned weights were spontaneously recovering — climbing back from exactly 0.0 to small non-zero values between evaluation steps. Gradient staleness was fixed, but something new was wrong.

The culprit was Adam's momentum. The Adam optimizer maintains two moment estimates for each parameter: the first moment (exponentially weighted mean of gradients) and the second moment (exponentially weighted mean of squared gradients). These accumulators are initialized from the pre-pruning training run and persist across fine-tuning calls.

When a weight is set to exactly zero by our pruning mask and fine-tuning begins, Adam's first moment estimate — built from all previous gradient history — can be non-zero. On the first update step, Adam computes:

Adam update rule. If the first moment m̂₁ ≠ 0 and the weight w_i has been zeroed, this update nudges the weight away from zero — resurrecting it.

This is the zombie weight: a connection we deliberately severed, brought back to life by the optimizer's memory of the past. Even a tiny resurrection — $w_i$ going from 0.0 to 0.003 — breaks the binary sparsity guarantee and pollutes the gradient calculations for subsequent pruning steps.

The Fix: Gradient Masking Before Each Update

The solution is to prevent dead weights from receiving gradient updates at all, by multiplying gradients by the pruning mask immediately before calling optimizer.step():

def fine_tune_with_masked_gradients(model, mask, trainloader, epochs=3):
    mask_tensor = torch.tensor(mask, dtype=torch.float32)
    optimizer = torch.optim.Adam(model.parameters(), lr=1e-4)
    criterion = nn.CrossEntropyLoss()

    for epoch in range(epochs):
        for x, y in trainloader:
            spk_in = encode_input(x)
            out = model(spk_in)
            loss = criterion(out, y)

            optimizer.zero_grad()
            loss.backward()

            # Kill gradients on pruned weights BEFORE the optimizer sees them
            with torch.no_grad():
                model.fc1.weight.grad *= mask_tensor.unsqueeze(0)

            optimizer.step()

            # Belt-and-suspenders: enforce zero again after update
            with torch.no_grad():
                model.fc1.weight.data *= mask_tensor.unsqueeze(0)

The dual enforcement — zero the gradient before the update, zero the weights after — is belt-and-suspenders engineering. The gradient masking is the primary fix (it prevents the momentum from accumulating for dead weights). The post-update weight zeroing is a cheap safety net in case any floating-point arithmetic creates a non-zero from a nominally zero update.

Final Results: Victory at 70% Sparsity, and the Transition to MNIST

With the zombie weights bug squashed, the iterative CQP pipeline converged cleanly. The results were striking enough to confirm that the mathematical approach was fundamentally correct — not just a marginal improvement over the baselines.

The 76.1 percentage point advantage over Gradient-only pruning at 70% sparsity is the number that matters most — it is the sharpest contrast and the most memorable result for reviewers. Gradient-only pruning without the magnitude component is a perfect illustration of what goes wrong without combined scoring: it creates a degenerate solution where a small set of high-gradient neurons are over-protected, leaving the rest of the network gutted.

Layer Sensitivity: Not All Layers Are Equal

A complementary experiment pruned each layer individually at 30%, 50%, and 70%, leaving the others intact. The results reveal the network's structural hierarchy.

fc2 being the most prunable makes intuitive sense: it is the narrowest bottleneck in the network (256 → 128), so it has been forced by training to learn maximally compressed, non-redundant representations. There is no slack to spare. But it turns out that all four of its dimensions at 50% sparsity are replaceable — the information can flow through the surviving half without meaningful loss. fc3 has fewer than 1,300 parameters total, but they map directly to class predictions; losing half of them is like removing half the output neurons.

The Transition to Real MNIST

Everything above was computed on a synthetic dataset generated by sklearn.make_classification — a pragmatic workaround for a restricted environment where MNIST download URLs returned HTTP 403 errors. The synthetic data served its purpose: it validated the mathematical correctness of the pipeline across all failure modes and edge cases.

The MNIST transition required no code changes beyond replacing the data loading step. The same models, same training loop, same criticality computation, same iterative pruning pipeline — all transferred directly. What changed was the nature of the input distribution.

What This Means for a Paper Submission