PaperlayerView exact revision on arXiv

Source-linked AI summary

The Curse of Depth in Large Language Models

Wenfang Sun, Xinyuan Song, Pengxiang Li, Lu Yin, Yefeng Zheng, Shiwei Liu

arXiv:2502.05795v6cs.LGcs.AI

TL;DR

Modern LLMs often contain deep layers that contribute less effectively, creating resource inefficiency. The paper attributes this to Pre-LN and introduces LayerNorm Scaling, which controls depth-related variance; experiments report consistent pre-training gains across 130M–7B models and downstream improvements after fine-tuning.

  • Problem

    Modern LLMs contain deeper layers that contribute significantly less to learning and representation, creating resource inefficiency despite costly training.

  • Method

    The paper proposes LayerNorm Scaling, which scales LayerNorm outputs inversely with the square root of layer depth to control Pre-LN variance growth.

  • Results

    LayerNorm Scaling consistently outperforms previous normalization and scaling techniques across model sizes from 130M to 7B and improves downstream fine-tuning performance.

  • Takeaways & Limitations

    LayerNorm Scaling is presented as a simple, hyperparameter-free modification that enables deeper layers to contribute more meaningfully during training.

  • Takeaways & Limitations

    The paper leaves a more detailed theoretical understanding of LayerNorm Scaling’s behavior in vision transformers to future work.

Abstract

from arXiv · show

In this paper, we introduce the Curse of Depth, a concept that highlights, explains, and addresses the recent observation in modern Large Language Models (LLMs) where nearly half of the layers are less effective than expected. We first confirm the wide existence of this phenomenon across the most popular families of LLMs such as Llama, Mistral, DeepSeek, and Qwen. Our analysis, theoretically and empirically, identifies that the underlying reason for the ineffectiveness of deep layers in LLMs is the widespread usage of Pre-Layer Normalization (Pre-LN). While Pre-LN stabilizes the training of Transformer LLMs, its output variance exponentially grows with the model depth, which undesirably causes the derivative of the deep Transformer blocks to be an identity matrix, and therefore barely contributes to the training. To resolve this training pitfall, we propose LayerNorm Scaling (LNS), which scales the variance of output of the layer normalization inversely by the square root of its depth. This simple modification mitigates the output variance explosion of deeper Transformer layers, improving their contribution. Across a wide range of model sizes (130M to 7B), our experiments show that LNS consistently outperforms previous normalization and scaling techniques in enhancing LLM pre-training performance. Moreover, this improvement seamlessly carries over to supervised fine-tuning. All these gains can be attributed to the fact that LayerNorm Scaling enables deeper layers to contribute more effectively during training. Our code is available at \href{https://github.com/lmsdss/LayerNorm-Scaling}{LayerNorm-Scaling}.

1 Introduction

The paper identifies the Curse of Depth: deep layers in modern LLMs often contribute little, wasting substantial training resources. It attributes this phenomenon to Pre-LN and proposes LayerNorm Scaling to make deeper layers more effective.

  • Ineffective layers are especially costly because modern LLMs require thousands of GPUs and months of training, motivating better utilization of every layer.
  • The Curse of Depth describes deeper LLM layers contributing significantly less than earlier layers while remaining robust to pruning and perturbations.This behavior limits meaningful transformations and creates resource inefficiency.
  • Experiments across Qwen3, LLaMA2, and DeepSeek find resilience to deep-layer removal, more removable layers at larger model sizes, and greater similarity among deep representations.
  • The paper identifies Pre-LN as the root cause of ineffective deep layers: its output variance accumulates with depth, making deep-layer derivatives approach the identity matrix.
  • LayerNorm Scaling scales normalized outputs according to layer depth to control variance growth and improve deeper layers’ contribution during training.

2 Empirical Evidence of the Curse of Depth

The empirical analysis compares layer contributions across normalization schemes using pruning impact and representation similarity. Pre-LN models show increasingly ineffective deep layers, whereas Post-LN models make deeper layers more critical.

  • The evaluation compares Pre-LN and Post-LN models by measuring layer-pruning effects at different depths.
  • The study evaluates Qwen3, LLaMA2, DeepSeek, and BERT-Large to compare normalization effects across architectures and model scales.BERT-Large is Post-LN; the other listed models are Pre-LN-based.
  • Performance Drop measures the performance change after removing a layer, with smaller values indicating lower contribution; MMLU is used for most models and SQuAD v1.1 for BERT-Large.
  • Angular Distance measures directional change between representations at layers ℓ and ℓ+n; smaller values indicate greater similarity and potentially redundant transformations.The reported distance averages 256K tokens sampled from C4.
  • In Pre-LN models, many deep layers—particularly beyond the 18th—can be pruned with minimal or sometimes positive performance change, unlike deep layers in Post-LN BERT-Large.
  • Controlled LLaMA-130M experiments show similar patterns: Post-LN deep layers become more distinct and critical, while Pre-LN deep layers remain similar and largely removable.

3 Analysis of the Curse of Depth

The analysis explains the Curse of Depth as a consequence of Pre-LN variance growth: deep-layer derivatives approach identity mappings, limiting meaningful transformations and expressivity.

  • Pre-LN normalizes layer inputs before attention or feed-forward computations in each Transformer block.
  • Pre-LN output variance grows with depth, while the relevant layer and intermediate variances follow the same overall trend.
  • When variance grows exponentially with depth, the gradient norm converges to a constant, limiting deep-layer learning dynamics.
  • Deep Pre-LN Transformer blocks increasingly behave like identity mappings, limiting model expressivity and meaningful transformations.
  • Figure 5 shows increasing diagonal dominance and vanishing off-diagonal Jacobian entries across deeper LLaMA2-7B layers.

4 LayerNorm Scaling (LNS)

LayerNorm Scaling controls Pre-LN variance by scaling normalized outputs according to layer depth. The resulting variance growth is slower, allowing more effective deep-layer training while preserving stability.

  • LayerNorm Scaling multiplies each layer’s normalized output by a factor inversely proportional to the square root of its layer index.
  • The scaling prevents excessive variance growth with depth and preserves Pre-LN stability while enhancing deeper-layer contributions.
  • LayerNorm Scaling reduces the variance upper bound from exponential growth Θ(exp(L)) to at most quadratic growth Θ(L^2).
  • Fewer scaled layers act as identity mappings, so the network utilizes more layers as depth increases.
  • LayerNorm Scaling provides moderate variance growth intended to improve deep-layer effectiveness and prevent early-layer gradient amplification.

5 Experiments

Across language-model scales, architectures, fine-tuning tasks, and a vision-transformer setting, LayerNorm Scaling generally improves training or downstream performance while controlling deep-layer variance.

  • 5.1 LLM Pre-training: LayerNorm Scaling achieves the lowest perplexity across tested model sizes and reduces perplexity by 0.97 on LLaMA-130M and 1.31 on LLaMA-1B versus Pre-LN.
  • 5.2 Supervised Fine-tuning: LayerNorm Scaling improves average LLaMA-250M fine-tuning performance by 1.80% versus Mix-LN and gains 1.86% over the best baseline for LLaMA-1B.
  • 5.3.1 OLMo: LNS outperforms Pre-LN on OLMo-7B as training progresses, ultimately producing a substantial training-loss gap.
  • 5.3.2 Qwen2.5: LNS reduces Qwen2.5-0.5B perplexity from 20.62 to 19.57 after 6B training tokens.
  • 5.5 LNS Enhances the Effectiveness of Deep Layers: For LNS, deeper-layer pruning causes larger accuracy drops, while most adjacent-layer angular distances exceed 0.6, indicating more diverse representations than Pre-LN.
  • 5.6 LayerNorm Scaling in Vision Transformer: In ViT-S, applying LNS after attention and MLP blocks controls deeper-layer variance below ∼150 instead of exceeding ∼3,000 by 30K update steps.

6 Ablation Study

The ablations compare LayerNorm Scaling with initialization, normalization, and insertion-position alternatives. LNS performs best overall, while some learnable or conflicting scaling choices harm performance.

  • Scaling methods: LNS achieves the best overall performance among the compared scaling methods.Scaled Initialization outperforms Depth-Scaled Initialization, while LayerScale and SkipInit degrade performance.
  • Scaling methods: Combining LNS with Scaled Initialization performs worse than using LNS alone.The authors recommend removing Scaled Initialization when applying LNS.
  • Normalization methods: Mix-LN and LayerNorm Scaling improve over Pre-LN, whereas Admin and Group-LN degrade performance.Sandwich-LN slightly outperforms Pre-LN in the LLaMA-130M comparison.
  • Insertion position: 25.76 perplexity is achieved when LNS is inserted after LayerNorm, compared with 26.73 for the baseline Pre-LN setting.Placing the scaling factor after the residual connection yields perplexity 1358.11, indicating training divergence.

7 Related Work

Prior work documents weaker and more perturbation-resistant deep layers, while Pre-LN improves deep Transformer training stability. Related approaches address scaling, normalization, signal propagation, and compression, but leave a distinct gap addressed by this paper.

  • Ineffective deeper layers: Earlier studies report that deep LLM layers tolerate more pruning and perturbation than shallow layers.Removing early layers causes larger performance declines, while middle and deep layers show robustness to layer swapping and dropping.
  • Layer normalization: Pre-LN became dominant because of its performance and stability advantages in modern LLMs.Prior analyses associate Pre-LN with improved gradient behavior for deep Transformers and easier layer stacking.
  • Architectural and scaling methods: Related methods modify residual dependencies, normalization, or residual scaling to stabilize or deepen Transformers.Examples include Admin, DeepNorm, Sandwich LayerNorm, and depth-dependent residual scaling.
  • Depth scaling and transfer: Existing depth-focused work mainly develops parameterization or optimizer-scaling rules for transferable training dynamics and hyperparameters.The cited approaches include µP, Depth-µP, residual scaling, covariance dynamics, and signal-propagation analyses.

8 Conclusion

The paper identifies the Curse of Depth as a consequence of ineffective deep layers and links it to variance accumulation in Pre-LN Transformers. It analyzes attention and feed-forward variance, then proposes scaling and derives bounds for the resulting dynamics.

  • Conclusion: The Curse of Depth describes deeper layers contributing less to learning and representation while remaining robust to pruning and perturbations.This behavior is associated with resource inefficiency because the layers fail to perform meaningful transformations.
  • Variance analysis: The attention analysis assumes approximately uniform softmax weights, with each element of A roughly 1/n.The softmax vector concentrates around the uniform distribution as the number of keys or values grows.
  • Variance analysis: The feed-forward analysis uses a two-layer linear network with a ReLU activation, FFN(x) = W2 · ReLU(W1 · x + b1) + b2.The derivation then analyzes how variance behaves across layers under simplifying assumptions.
  • Variance analysis: Variance accumulates layer by layer, and the derived bounds imply faster-than-linear growth with depth.The analysis connects large variance increases to gradient spikes.
  • Gradient analysis: The proof strategy estimates gradient behavior by combining Jacobian spectral norms for feed-forward and self-attention layers.It uses assumptions about dimensions, heads, sequence length, and weight standard deviations.

A.2.1 Proof of Lemma A.2

The proof bounds the gradient norm through the Jacobians of the feed-forward and self-attention components. It combines spectral-norm estimates under simplifying dimensional and variance assumptions.

  • Gradient bound: The proof estimates ∂yℓ/∂x1 by analyzing the spectral norms of the FFN and self-attention Jacobians.This provides an upper bound on gradient propagation through the Transformer layers.
  • Assumptions: The derivation assumes equal embedding and FFN dimensions for simplicity.The stated dimensions include d, dFFN, and dhead, along with the number of heads and sequence length.
  • Self-attention analysis: The attention Jacobian is expressed using the concatenated head output Z(·) and its Jacobian JZ.This supplies the self-attention component of the spectral-norm analysis.
  • Gradient bound: The combined bound assumes that all σ values are the same.The resulting estimate combines inequalities for the attention and feed-forward components.

A.2.2 Analysis of the Upper Bound

The analysis derives upper bounds for variance propagation through Transformer layers, using product forms, logarithms, Taylor expansions, and infinite-series approximations. It concludes that variance accumulates layer by layer and characterizes the resulting growth under the stated assumptions.

  • Product-form bound: The derivation rewrites the relevant bound as a product over layers and treats A and B as constants independent of σxℓ.The product is then analyzed through logarithms and series expansions.
  • Bound behavior: Under the stated assumptions, faster growth of σxℓ lowers the upper bound of the analyzed product.This relationship is obtained from classical infinite-product analysis.
  • Series analysis: The harmonic-series contribution grows logarithmically, while higher-order terms converge to constants.The analysis uses the approximation 1/k ∼ log n + γ for large n.
  • Variance growth: The variance accumulates layer by layer, producing an upper-bound growth characterization for σxℓ.The derivation combines layerwise variance relations with bounds on the relevant products.

B Variance Growth in Pre-LN Training

The experiments track layerwise output variance during training and show that Pre-LN models maintain low shallow-layer variance but develop exponential variance growth in deeper layers. Layer pruning further indicates that language branches can exhibit depth-dependent sensitivity, unlike the more uniform vision branch examined here.

  • Pre-LN variance: Pre-LN output variance remains low in shallow layers but grows exponentially in deeper layers across training stages.This pattern appears at 1000, 3000, and 6000 epochs in LLaMA-130M.
  • Pre-LN variance: The persistent variance increase across training confirms that deep-layer amplification is not merely a temporary training effect.The observation motivates stabilization techniques such as LayerNorm Scaling.
  • Vision–language comparison: The Qwen 2.5-VL-7B language branch clearly exhibits the Curse of Depth, whereas its vision branch remains uniformly important under layer pruning.The comparison uses one-layer-at-a-time pruning and MMMU evaluation.

D Limitations

The paper’s limitations concern architectural scope, task coverage, and fine-grained understanding of representations across depth. Its conclusions may therefore differ for uncovered architectures, specialized tasks, or specific types of stored information.

  • Scope of Architectures: The theoretical analysis focuses on Transformer-based LLMs using Pre-LN and does not cover alternative normalization, mixture-of-experts, or structured-sparsity architectures.Post-LN-only and normalization-free models are explicitly outside the study’s theoretical scope.
  • Task Coverage: Most empirical evaluations use general-purpose benchmarks such as MMLU, leaving domain-specific and long-context reasoning dynamics for future work.These tasks may reveal different patterns in deep-layer contributions.
  • Fine-grained Representation Quality: The study does not determine whether LayerNorm Scaling preserves syntactic, semantic, or factual information across depth.Its effects on fine-grained representation content remain unexamined.
Loading 2502.05795v6…