PaperlayerView exact revision on arXiv

Source-linked AI summary

When Does Sparsity Mitigate the Curse of Depth in LLMs

Dilxat Muhtar, Xinyuan Song, Sebastian Pokutta, Max Zimmer, Nico Pelleriti, Thomas Hofmann, Shiwei Liu

arXiv:2603.15389v2cs.CL

TL;DR

The paper addresses why later layers in deep Pre-LN LLMs become under-utilized as variance accumulates and asks whether sparsity-like mechanisms can mitigate this effect. It combines controlled depth experiments with layer-effectiveness measures to study implicit and explicit sparsity. Across evaluated settings, reduced interaction density is associated with lower variance and improved layer utilization, motivating a practical training recipe that reports a 4.6 accuracy gain on downstream tasks.

  • Problem

    Later LLM layers can be under-utilized because accumulated Pre-LN variance pushes deep blocks toward near-identity behavior, while sparsity’s effect on variance propagation is poorly understood.

  • Method

    The paper combines controlled depth-scaling experiments with Causal, Permutation, and Usefulness Scores to compare implicit and explicit sparsity across LLM settings.

  • Results

    Across evaluated settings, implicit and explicit sparsity are consistently associated with reduced variance accumulation and improved layer effectiveness.

  • Takeaways & Limitations

    Sparsity can be treated not only as a compute-saving tool but also as an optimization mechanism for controlling variance growth and improving compute utilization in deep LLMs.

  • Takeaways & Limitations

    The theoretical analysis omits normalization methods, attention-head correlations, and feedback between learned parameters and activation statistics, so it is qualitative rather than an end-to-end predictive model.

Abstract

from arXiv · show

Recent work has demonstrated the curse of depth in large language models (LLMs), where later layers contribute less to learning and representation than earlier layers. Such under-utilization is linked to the accumulated growth of variance in Pre-Layer Normalization, which can push deep blocks toward near-identity behavior. In this paper, we provide evidence that sparsity-like mechanisms can dampen variance propagation and are associated with improved depth utilization Our investigation covers two sources of sparsity: (i) implicit sparsity, which emerges from training and data conditions, including weight sparsity induced by weight decay and attention sparsity induced by long-context inputs; and (ii) explicit sparsity, which is enforced by architectural design, including key/value-sharing in Grouped-Query Attention and expert-activation sparsity in Mixtureof-Experts. Our claim is thoroughly supported by controlled depth-scaling experiments and targeted layer effectiveness interventions. Across settings, we observe a consistent relationship: mechanisms with reduced effective interaction density tend to exhibit lower output variance and better layer differentiation. We eventually distill our findings into a practical rule-of-thumb recipe for training depth-effective LLMs, yielding a notable 4.6 accuracy improvement on downstream tasks. Our results suggest that sparsity-like design choices are an important and previously underemphasized factor in effective depth scaling for LLMs. Code is available at https://github. com/pUmpKin-Co/SparsityAndCoD.

1. Introduction

The introduction frames the curse of depth as under-utilization of later Transformer layers linked to variance accumulation, then proposes sparsity as a possible variance-regulating mechanism. Controlled experiments examine implicit and explicit sparsity and report improved layer effectiveness and downstream accuracy when complementary mechanisms are combined.

  • Motivation: Later Transformer layers can be under-utilized, with layer skipping causing negligible performance degradation and revealing redundancy.The redundancy also indicates inefficient use of training resources, despite enabling layer-pruning-based compression.
  • Motivation: In Pre-LN architectures, accumulated residual-stream variance can make deep layers functionally ineffective as their Jacobians approach identity.The introduction contrasts this variance-control perspective with the paper’s focus on sparsity-like mechanisms.
  • Research question: Sparsity is defined broadly as reduced effective interaction density, parameter participation, activation mass, or independent computational paths.The paper considers MoE, GQA, and long-context attention as examples of sparse computation whose variance effects remain poorly understood.
  • Approach: The paper uses controlled depth-scaling experiments and three metrics—Causal, Permutation, and Usefulness Scores—to quantify layer effectiveness.Models are trained from scratch across 12 to 32 layers while other hyperparameters are held constant.
  • Findings: Both implicit sparsity from weight decay and long-context inputs and explicit sparsity from MoE and GQA are reported to reduce variance accumulation and improve layer effectiveness.The paper presents these mechanisms as complementary sources of sparsity-like behavior.
  • Findings: 4.6 accuracy gain on downstream tasks is reported for the practical rule-of-thumb combining complementary sparsity mechanisms.The contribution describes this recipe as a way to train depth-effective LLMs.

2. Variance Propagation and Curse of Depth

This section explains the curse of depth through variance propagation in Pre-LN residual blocks and operationalizes layer effectiveness with causal, permutation, and usefulness measures. Controlled depth experiments associate increasing depth and variance with identity-like Jacobians, redundancy, and declining layer effectiveness.

  • Definitions: Pre-LN blocks normalize inputs before transformation, while variance is measured across hidden-state dimensions and averaged over tokens.High last-layer variance indicates signal accumulation across depth and can make layer gradients negligible.
  • Experimental setup: Controlled experiments vary only depth from 12 to 32 layers while tracking output variance, layer effectiveness, and Jacobian convergence toward identity.Learning-rate sweeps select the best validation-performing setting for each configuration.
  • Layer effectiveness metrics: Causal Score measures how much removing a layer affects subsequent representations, with lower scores indicating minimal impact and potential redundancy.The global score aggregates effects across layer pairs and normalizes for model depth.
  • Layer effectiveness metrics: Permutation Score measures performance degradation when layer positions are swapped, with scores near zero indicating redundancy.Higher scores indicate that layers are less interchangeable.
  • Layer effectiveness metrics: Usefulness Score measures each layer’s nonlinear contribution through linear approximation, and its global form reports the fraction of layers with significant performance impact.Higher usefulness indicates efficient depth utilization, whereas lower usefulness reveals redundancy.
  • Main observation: Increasing depth is accompanied by larger variance, Jacobians closer to identity, increasingly diagonal-dominant Jacobian patterns, and deteriorating layer effectiveness.Figure 2 and Figure 3 jointly support the association between variance accumulation, identity-like transformations, and the curse of depth.
  • Main observation: Very deep models can use 2.56× more parameters while having 14 low-effectiveness layers, illustrating substantial inefficiency despite increased depth.The comparison reports 18 versus 12 effective layers for the indicated models.

3. Sparsity as Variance Regularizer

The paper presents sparsity as a variance regulator: reducing effective interaction density lowers variance propagation across residual depth. It examines implicit sparsity from training and inputs alongside explicit architectural sparsity.

  • Theoretical motivation: Sparser residual updates yield slower variance growth with depth because the per-layer variance factor decreases as mask density decreases.The theoretical bound depends on sparsity through the density parameter ρℓ, with smaller ρℓ producing a smaller factor (1 + √αℓρℓ)^2.
  • Theoretical motivation: The theoretical variance result assumes layer weights are independent of the sparsity masks, an assumption that training-induced sparsity violates.The paper notes that weights and sparsity patterns become coupled when sparsity emerges during training.
  • Implicit sparsity: Implicit sparsity includes weight decay and sequence-length scaling, which induce sparsity in parameters or attention patterns during training or from inputs.Weight decay drives small parameters toward zero, while longer sequences produce near-zero attention weights through positional bias and softmax normalization.
  • Implicit sparsity: Longer sequences reduce attention-output variance under a uniform-attention approximation by averaging over T independent value coordinates.The resulting variance decreases inversely with sequence length T.
  • Explicit sparsity: Explicit sparsity fixes absent connections or computational paths architecturally, including shared key-value heads in GQA and top-k expert activation in MoE.GQA reduces independent key-value computations from H to H/G, while MoE activates only k of E experts per token.
  • Implicit sparsity: Weight decay reduces output variance while increasing the fraction of effectively zero parameters across thresholds.The paper quantifies weight sparsity as the fraction of parameters below threshold ϵ.
  • Explicit sparsity: GQA and MoE provide variance-reduction mechanisms through shared key-value computations or averaging across selected experts.The idealized GQA analysis gives approximately σ²_V/n and an additional 1/G factor, while top-k MoE yields approximately 1/k variance reduction under stated assumptions.

4. Verification of Variance Dampening

End-to-end experiments test whether implicit and explicit sparsity dampen variance propagation. Across weight decay, sequence length, GQA, and MoE, greater sparsity generally corresponds to lower variance and better layer effectiveness, subject to capacity trade-offs.

  • Verification of Variance Dampening: Controlled end-to-end training evaluates whether sparsity mitigates variance accumulation and the curse of depth.The study examines weight decay, sequence length, GQA, and MoE while tracking variance, performance, and layer effectiveness.
  • Weight Decay: Within the optimal range λ ∈[0, 0.1], perplexity improves from 15.63 to 14.83 and usefulness score rises from 0.75 to 0.81.Stronger weight decay reduces last-layer variance and increases weight sparsity, but λ ≥1.0 degrades performance; λ = 3.0 causes perplexity 773.42.
  • Sequence Length: Longer training sequences reduce last-layer variance and increase attention sparsity across thresholds ϵ ∈{10−3, 10−4, 10−6}.Scaling from T = 256 to T = 2048 improves perplexity by 4+ points and raises usefulness from 0.69 to 0.81; T = 8192 yields diminishing returns.
  • Grouped Query Attention: MQA (G = 16) has 2× lower variance than MHA (G = 1), while perplexity improves from 14.52 to 14.47 and usefulness rises from 0.81 to 0.87.The results associate key-value sharing with fewer independent attention interactions, lower variance, and 13 versus 14 effective layers for MHA and MQA.
  • Mixture of Experts: MoE dampens variance by approximately 6× and 3× in the 400M-active and 1B-active configurations, respectively, while improving perplexity by over 2.For the 1B-active configuration, usefulness score increases from 0.81 to 0.94 relative to the dense counterpart.

5. Sparsity as an Enabler for Depth

Combining sparsity mechanisms enables deeper models to retain stronger layer utilization under a fixed 1.2B-parameter budget. The best configuration combines GQA and MoE, improving accuracy while avoiding the utilization loss of a naive deeper model.

  • Sparsity as an Enabler for Depth: Depth-scaling experiments compare L = 16 and L = 32 models at constant 1.2B parameters using accuracy and Usefulness Score.The experiments integrate different sparsity strategies while keeping training data and parameter budgets identical.
  • Implicit Sparsity: Weight decay with λ = 0.3 raises accuracy to 41.4 and Usefulness Score to 0.63, a 20% improvement over the naive baseline.Excessive regularization at λ = 1.0 degrades both performance and utilization.
  • Combined Sparsity: The combined GQA (G = 2) and MoE configuration achieves average accuracy 44.1 and Usefulness Score 0.75.This represents a 4+ point accuracy gain over the L = 16 baseline and higher utilization than the naive L = 32 model, 0.75 versus 0.53.

6. Related Work

Prior work identifies the curse of depth as layer redundancy linked to variance growth in Pre-LN Transformers. This paper contrasts established explicit variance-control methods with sparsity mechanisms that reduce interaction density or parameter participation.

  • Curse of Depth: The curse of depth describes increasing layer redundancy in modern Pre-LN LLMs, where deeper layers are pushed toward identity mappings by variance growth.Earlier studies report that skipping layers can cause negligible performance degradation.
  • Variance Control: Prior approaches mitigate the curse of depth through scaled initialization, normalization scaling, alternative normalization, and advanced residual connections.These methods explicitly control variance rather than relying on sparsity from architectural or training choices.
  • Sparsity Taxonomy: Sparsity is categorized as implicit or explicit: weight decay and concentrated attention emerge naturally, whereas MoE and GQA are architecturally enforced.MoE activates only k of E experts per token, while GQA shares key-value projections across query heads.

7. Limitations

The theoretical analysis abstracts away several mechanisms present in contemporary Transformer training, so its theorems provide qualitative rather than calibrated predictions. Empirical results are used to substantiate the broader sparsity–variance claims.

  • Limitations: The theoretical analysis omits normalization methods, attention-head correlations, and feedback between learned parameters and activation statistics.These omissions limit direct application of the theorems to fully trained LLMs.
  • Limitations: The theorems are qualitative statements about interaction sparsity and variance growth, not calibrated end-to-end residual-stream models.The paper relies on empirical regularities to bridge the gap between idealized analysis and deployed models.

8. Conclusion

Variance growth in deep networks produces a curse of depth, while implicit and explicit sparsity damp variance propagation, improve layer utilization, and support effective depth scaling.

  • Sparsity mitigates the curse of depth by reducing variance accumulation and improving layer effectiveness.

Impact Statement

The paper reports that output variance accumulates with depth, with stronger growth in MLP blocks, while deeper-layer Jacobians increasingly approach identity mappings.

  • Output variance accumulates with depth across models of different depths.
  • MLP block variances grow more rapidly and more strongly across depth than attention block variances.
  • Residual components grow only moderately, suggesting shortcut paths also contribute to variance accumulation.
  • Off-diagonal Jacobian components gradually decrease with depth in a 32-layer model.

A.4. Kurtosis Analysis

The appendix examines kurtosis, attention sparsity, sequence length, and theoretical sparsity mechanisms as explanations for variance control across depth.

  • Kurtosis Analysis: Higher kurtosis appears in early rather than deep layers, indicating variance explosion is not driven by outlier features.
  • Attention Sparsity: Longer training sequences consistently induce stronger implicit attention sparsity across evaluation lengths.
  • Sensitivity Analysis: Sparsity improves layer usefulness across threshold choices, while deeper models retain more low-effectiveness layers.
  • Initialization: Scaled initialization dampens variance propagation, while larger sequence lengths provide an additional depth-effectiveness benefit.
  • Theoretical Analysis: The theoretical variance gain depends on sparsity through an energy-retention factor, with smaller retention yielding lower global variance growth.

B.4. Weight Decay Variance Control: Theory and Proof

The appendix analyzes weight decay, sequence length, and Top-k MoE as variance-control mechanisms, while stating assumptions that limit the scope of their theoretical guarantees.

  • Weight Decay: The weight-decay analysis is a linearized calculation and is not a complete model of coupled Transformer training dynamics.
  • Weight Decay: Increasing weight decay tightens both initialization-contraction and gradient-noise terms, yielding a smaller upper bound on layer-output variance.
  • Sequence Length: Under uniform independent averaging, sequence length reduces attention-output variance, but this theorem does not establish sparse attention empirically.
  • Mixture of Experts: In trained MoE layers, input-dependent unequal routing and expert coupling make the 1/k reduction a limiting intuition rather than a predictive law.
  • Mixture of Experts: Under idealized uniform-gating and independence assumptions, Top-k MoE reduces output and Jacobian variance by a factor 1/k.

C.1. Depth Control Experiments

The depth-control experiments vary model depth and sparsity-related settings under controlled training configurations. They report layer-effectiveness visualizations across weight decay, sequence length, grouped-query attention, and dense-versus-MoE models.

  • Depth control: Models are trained across depths from 12 to 32 layers while holding other hyperparameters constant to verify the curse of depth.These runs use 10B training tokens and fixed optimization settings, with learning-rate sweeps across depth configurations.
  • Implicit sparsity settings: Weight-decay and sequence-length settings are visualized separately for λ ∈ {0.1, 1, 3} and T ∈ {1024, 4096, 8192}.Figures 20–22 correspond to weight decay, while Figures 23–25 correspond to sequence length.
  • Reported scores: The experiments report score visualizations for the tested settings, with one listed result sequence containing values including 0.365, 0.371, 0.370, and 0.393.The supplied passage does not identify the metric or configuration corresponding to each value.
  • Explicit sparsity settings: Grouped-query attention is visualized at G ∈ {1, 16}, while separate figures compare Dense-1B with MoE-7BA1B.The GQA configurations correspond to key-value head sizes of 16 and 1, respectively.
Loading 2603.15389v2…