PaperlayerView exact revision on arXiv

Source-linked AI summary

Delta Attention Residuals

Cheng Luo, Zefan Cai, Junjie Hu

arXiv:2605.18855v1cs.LGcs.CV

TL;DR

Redundant cumulative hidden states make Attention Residuals increasingly low-contrast, raising the underexplored question of which layer-wise sources should be routed. Delta Attention Residuals route over per-sublayer or block-level deltas instead of cumulative states, using additive routing to preserve the residual stream. Across 220M–7.6B parameters, Delta Attention Residuals consistently improve over baseline and AttnRes; at 7.6B, Delta Block improves −8.2% over baseline.

  • Problem

    Redundant cumulative hidden states make Attention Residuals increasingly low-contrast, raising the underexplored question of which layer-wise sources should be routed.

  • Method

    Delta Attention Residuals route over per-sublayer or block-level deltas instead of cumulative states, using additive routing to preserve the residual stream.

  • Results

    Across 220M–7.6B parameters, Delta Attention Residuals consistently improve over baseline and AttnRes; at 7.6B, Delta Block improves −8.2% over baseline.

  • Takeaways & Limitations

    Delta routing provides sharper cross-layer selection, while Delta Block offers a lower-overhead configuration that matches per-sublayer quality.

Abstract

from arXiv · show

Attention Residuals replace standard additive residual connections with learned softmax attention over previous layer outputs, enabling selective cross-layer routing. However, standard Attention Residuals still attend over cumulative hidden states in previous layers, which are highly redundant. We show that this redundancy leads to routing collapse in deeper layers: attention weights become low-contrast and closer to uniform (max weight ${\approx}$0.2), limiting the model's ability to select informative states in previous layers. This raises a key but underexplored design question: what layer-wise representations should be routed in Attention Residuals? To answer this question, we propose Delta Attention Residuals, which attend over deltas -- the change introduced by each sublayer ($\mathbf{v}_i = \mathbf{h}_{i+1} - \mathbf{h}_i$) -- instead of cumulative states. Delta representations are structurally diverse and yield higher-contrast attention distributions (max weight ${\approx}$0.6), enabling more selective and effective routing across layers. This principle applies at both per-sublayer and block granularity. Across all tested scales (220M--7.6B), Delta Attention Residuals consistently outperform both standard residuals and Attention Residuals, with 1.7--8.2\% validation perplexity gains. Delta Attention Residuals also enables converting pretrained checkpoints into Delta Attention Residuals via standard fine-tuning. Code is available at https://github.com/wdlctc/delta-attention-residuals-code.

1 Introduction

Delta Attention Residuals address routing collapse in Attention Residuals by routing over structurally diverse layer deltas rather than redundant cumulative states. This produces sharper, more selective cross-layer routing and consistently improves performance across Qwen-based models from 220M to 7.6B parameters.

  • Problem: Attention Residuals suffer routing collapse because cumulative hidden states become redundant, making softmax attention low-contrast and nearly uniform in deeper layers.Adjacent cumulative states become increasingly similar with depth, reducing the discriminability of routing candidates.
  • Motivation and contribution: Routing over deltas increases deep-layer attention sharpness to approximately 0.6 maximum weight, compared with approximately 0.2 for Attention Residuals.Figure 1 reports 1.8× higher average maximum weight for Delta Block than AttnRes, 0.62 versus 0.35.
  • Method: Delta Attention Residuals route over each sublayer’s change, v_i = h_i+1 − h_i, at either per-sublayer or block granularity instead of cumulative states.Deltas capture individual sublayer contributions and are naturally more diverse because different sublayers operate in different functions and subspaces.
  • Contributions: Delta Attention Residuals consistently improve over standard residuals and Attention Residuals across Qwen-based models ranging from 220M to 7.6B parameters.The method is evaluated across multiple model scales and outperforms both baselines.

2 Method: Delta Attention Residuals

Delta Attention Residuals replace cumulative hidden-state routing in Attention Residuals [Kimi, 2025] with additive softmax routing over per-sublayer changes, preserving the residual stream while enabling more selective cross-layer access. The method supports both fine-grained sublayer sources and block-level deltas, whose structural diversity avoids the redundancy that drives deep routing toward uniformity.

  • Background: In standard Pre-Norm transformers, each sublayer updates h_i to h_i+1 = h_i + v_i, where v_i = h_i+1 − h_i is the introduced change.Standard residuals accumulate sublayer outputs with fixed unit coefficients; Attention Residuals [Kimi, 2025] instead learn weighted combinations of preceding sources.
  • Routing redundancy: Cumulative hidden states become increasingly redundant, causing deep Attention Residual routing to approach uniformity with maximum softmax weight ∼0.2 at Qwen3-0.6B scale.Adjacent running-sum states share a large common prefix, producing similar routing logits and weakening source selection.
  • Delta sources: Delta sources remain structurally diverse across attention and MLP sublayers and depths, supporting sharp routing and coarsening naturally to block deltas Δ_b = h_b+1 − h_b.Fine-grained Delta AttnRes uses 2L sources for L layers, while block grouping aggregates multiple sublayer outputs into one source.
  • Delta Attention Residuals: Delta Attention Residuals route per-sublayer outputs as additive information, preserving the residual stream instead of replacing it with a weighted combination of cumulative states.The routed delta is added to the current residual stream, while cumulative-state routing can collapse intermediate sublayer contributions at block boundaries.

3 Experiments

Experiments across 220M–7.6B models show Delta Block and Delta Attention Residuals consistently outperform standard and cumulative-source replacement routing, with Delta Block offering the strongest quality–efficiency tradeoff. Delta routing also preserves sharp attention distributions, remains effective on pretrained checkpoints, and scales to 8B parameters.

  • Multi-scale comparison: 36.83: Delta AttnRes achieves the best validation PPL at 220M, 533M, and 1044M, while Delta Block follows within 0.7% and both beat baseline.Delta AttnRes scores 36.83, 31.05, and 29.13; Delta Block scores 37.08, 31.16, and 29.19, respectively.
  • Routing mechanism: At 1044M, cumulative-source replacement routing degrades: AttnRes is 6.9% worse than baseline and Full AttnRes is 12.3% worse.Delta routing avoids this degradation by using delta sources with additive routing, preserving the residual stream and eliminating resets.
  • Efficiency and block size: Delta Block is the practical default: at 1044M it nearly matches Delta AttnRes quality while reducing overhead to 20% throughput and 26% memory.Delta AttnRes reaches 29.13 PPL at 34k tok/s and 77.7 GB, whereas Delta Block reaches 29.19 at 86k tok/s and 28.4 GB.
  • Existing architecture: At Qwen3-0.6B, Delta Block improves PPL by 2.4% over baseline, while its maximum routing weight stays near 0.6 as AttnRes collapses near 0.2 in deep layers.Delta Block’s average maximum weight is 0.62 versus 0.35 for AttnRes, or 1.8× higher.
  • 8B scaling: −8.2%: Delta Block reaches 16.00 validation PPL at 8B, beating baseline’s 17.43, while AttnRes worsens to 18.58 (+6.6%).This confirms at 7.57B parameters that additive delta routing scales successfully whereas cumulative-source replacement routing degrades.
  • Checkpoint conversion: 55.6%: Delta Block improves average downstream accuracy over baseline’s 55.0% and AttnRes’s 54.1% when fine-tuning pretrained checkpoints.AttnRes incurs an initialization loss spike from 2.8 to 3.96 and needs about 2000 steps to recover, whereas Delta Block starts smoothly.

4 Analysis

The analysis shows that cumulative-state redundancy causes routing collapse and information loss, while delta sources preserve discriminability and enable sharper, more selective cross-layer routing. Empirically, Delta Block maintains sharp routing and avoids the perplexity degradation observed with AttnRes.

  • Routing sharpness: Max routing weight falls from ∼1.0 early to ∼0.2 in deep AttnRes layers, whereas Delta Block maintains ∼0.6 routing and averages 1.8× higher sharpness (0.62 vs. 0.35).Cumulative states share most components with neighboring states, producing near-uniform attention; delta sources remain diverse because attention and MLP outputs occupy different subspaces.
  • Empirical validation: 31.76 PPL versus 29.70 for the baseline (+6.9%) shows AttnRes degradation at 1044M, while Full AttnRes degrades further to 33.36.AttnRes discards the residual stream and collapses intermediate sublayer contributions within each block; the resulting information loss compounds with depth.
  • Routing patterns: Deep Delta Block layers concentrate >50% weight on specific early outputs, unlike the near-uniform cumulative-state routing of AttnRes.The learned weights confirm that delta sources preserve discriminability throughout depth and create sharp cross-layer shortcuts.
  • Routing patterns: Deep layers give disproportionate attention to the token embedding, allowing additive routing to re-inject diluted embedding signal without disrupting the residual stream.This pattern is consistent with progressive embedding-signal dilution under standard residuals.

5 Conclusion

Delta Attention Residuals route cross-layer connectivity over per-sublayer deltas rather than cumulative hidden states, producing sharper routing and best perplexity across tested configurations. Combined with additive routing, the method preserves the residual stream while improving selectivity.

  • 5 Conclusion: Delta Attention Residuals replace cumulative hidden states with per-sublayer deltas as routing sources for cross-layer connectivity.The routed delta is v_i = h_i+1 − h_i, capturing what changed rather than what accumulated.
  • 5 Conclusion: 3× sharper routing results from deltas, with deep-layer maximum attention weights of approximately 0.6 versus approximately 0.2 for cumulative states.The contrast reflects more selective cross-layer routing.
  • 5 Conclusion: Delta methods combined with additive routing preserve the residual stream and achieve the best perplexity from 220M custom configurations through standard Qwen3-0.6B and Qwen3 configurations.The supplied passage states this scope but ends mid-sentence after “Q”.

A Related Work

Prior work studies cross-layer connections, residual-stream behavior, contrastive differencing, and learned routing. Delta Attention Residuals build on these directions by analyzing source redundancy and routing representations along depth.

  • Cross-Layer Connections: Cross-layer methods range from concatenating previous outputs to static depth weighting, widened residual streams, constrained mixing, and dynamic per-stream weighting.DenseNet [Huang et al., 2017], DenseFormer [Pagliardini et al., 2024], Hyper-Connections [Zhu et al., 2024], mHC [DeepSeek, 2025], and MUDDFormer [Xiao et al., 2025] represent these alternatives.
  • Residual Stream Analysis: Residual-stream research interprets residual networks as ensembles, formalizes the residual stream, stabilizes deep transformers, and identifies duality between depth and sequence-axis attention.These perspectives motivate the paper’s source redundancy analysis.
  • Contrastive and Delta Methods: Differencing redundant components can sharpen signals, as shown by subtracting amateur from expert logits, contrasting early and late layers, and transferring fine-tuning residuals.Contrastive Decoding [Li et al., 2023], DoLa [Chuang et al., 2024], and Proxy-Tuning [Liu et al., 2024] exemplify this direction.
  • Gating and Routing: Learned routing has been applied to skip connections, width-dimension pathways, and token-wise depth decisions, whereas this work routes source representations along depth.Highway Networks [Srivastava et al., 2015], GLU [Shazeer, 2020], ReZero [Bachlechner et al., 2021], MoE [Shazeer et al., 2017, Fedus et al., 2022], and Mixture of Depths [Raposo et al., 2024] establish related routing mechanisms.

B Comparison with Original Attention Residuals

Delta Attention Residuals differ from original AttnRes in their routed sources, additive routing, and absence of block-boundary resets. These changes preserve residual information while maintaining sharper, more selective routing than cumulative-state replacement.

  • Sources: Delta methods route structurally diverse deltas instead of redundant cumulative sources, maintaining sharper attention with max weight ∼0.6 versus AttnRes’s ∼0.2 in deep layers.Per-sublayer deltas use v_i = h_i+1 − h_i, while block-level variants use Δ_b = h_current − h_prev.
  • Routing: Delta methods add routed deltas to the residual stream, whereas AttnRes replaces the hidden state with a weighted source sum and discards the current residual stream.At initialization, zero queries produce uniform weights, and additive routing keeps the perturbation bounded.
  • Reset: Delta methods never reset at block boundaries, while AttnRes resets partial_block to zero, forcing reconstruction and compounding information loss when each layer resets.Delta routing augments the naturally accumulated hidden state through standard residual addition.

C Delta Block: Block-Level Variant

Delta Block aggregates consecutive attention and MLP outputs into block-level deltas, reducing routing sources while preserving additive residual routing. It trades routing granularity for efficiency, retaining diverse sources and matching Delta AttnRes on most evaluations.

  • C Delta Block: Block-Level Variant: Delta Block preserves residual information, avoids information loss at block boundaries through additive routing, and supports safe zero-initialization.These properties are inherited from Delta AttnRes.
  • C Delta Block: Block-Level Variant: Delta Block reduces routing sources from 2L to approximately L/B, lowering per-layer routing cost by a factor of approximately 2B.Each block aggregates its sublayer outputs into one block delta, while the current block contributes a partial delta.
  • C Delta Block: Block-Level Variant: Block deltas remain mutually distinct and routing stays sharp because they combine structurally heterogeneous attention and MLP outputs across B layers.This avoids the source redundancy that affects cumulative block sources in the original AttnRes at depth.
  • C Delta Block: Block-Level Variant: Delta Block trades a small amount of routing granularity for lower overhead and is preferred for very deep models or tight memory budgets.Per-sublayer Delta AttnRes offers maximum routing expressivity and is preferred at small-to-medium scales, while Delta Block matches it on most evaluations using fewer sources.

NeurIPS Paper Checklist

The checklist indicates that the paper’s stated contributions are supported, theoretical assumptions and empirical settings are documented, and code and data access enable reproduction. It notes single-run experiments as a limitation, mitigated by evaluations across five model scales.

  • Claims: The paper’s abstract and introduction accurately state three supported contributions: identifying routing collapse, proposing Delta Attention Residuals, and enabling fine-tuning conversion.These claims are supported by experiments in §§3–3.5.
  • Theory assumptions and proofs: Proposition 1 states its running-sum assumptions explicitly and provides both a formal argument and empirical validation.The checklist therefore treats the theoretical result as having documented assumptions and supporting justification.
  • Reproducibility and open access: Reproduction is supported through disclosed hyperparameters, optimizer settings, hardware, training budget, complete pseudocode, and publicly available code.The paper also uses the publicly available FineWeb-Edu dataset and Qwen3 model family.
  • Experimental setting and resources: The experimental setup specifies AdamW, cosine warmup scheduling, batch size, sequence length, 8×H100 hardware, training steps, fine-tuning details, and 0-shot downstream evaluation.Compute reporting includes BF16 precision, throughput, peak GPU memory, and 4×H100 fine-tuning resources.
Loading 2605.18855v1…