Value Residual Learning

• Deeper attention layers can over-smooth representations, making token representations increasingly similar and diluting token-level features as sequence-level features dominate.DenseFormer’s learned connection coefficients also indicate that deeper layers require more attention to initial embeddings.Source: §1 · p. 1
• ResFormer adds a residual connection from first-layer value vectors to each current layer before attention, complementing standard hidden-state residuals.The current layer’s attention matrix is shared by both value states.Source: §1 · pp. 1–2
• Figure 1 contrasts the skip connections used by vanilla Transformer, NeuTRENO, DenseFormer, ResFormer, and SVFormer in simplified three-layer attention-only structures.Source: Figure 1 · p. 2, §1 · p. 2
• SVFormer shares a single value state across layers by decoupling values from the attention operation, reducing KV cache by nearly half.It performs better with longer training sequences and can reduce KV cache further when integrated with GQA.Source: §1 · p. 2
• 16.11% fewer model parameters and 20.3% less training data achieve equivalent validation loss to Transformer with ResFormer, while memory usage and computational cost remain similar.Source: §1 · p. 2

2.1 Shortcut Connections

• Dense connections and stochastic depth extend ResNet-style shortcuts, while DenseFormer lets layers access preceding hidden states to improve information flow.Source: §2.1 · p. 2
• Self-attention can smooth token representations, and NeuTRENO counters over-smoothing by adding first-to-current value-vector differences to attention outputs.Source: §2.1 · pp. 2–3

2.2 KV cache compressing

• Multi-Query Attention and Grouped-Query Attention share keys and values across query groups, while CLA and LISA reuse representations or attention across layers to reduce KV-cache redundancy.Source: §2.2 · p. 3
• SVFormer differs from these methods by decoupling values from query and key during attention, enabling value sharing across layers.Source: §2.2 · p. 3

3.2 ResFormer

• ResFormer adds a skip connection from the first layer’s value V1 to later attention computations, complementing standard hidden-state residuals.The approach uses the current layer’s attention matrix with value states that combine first-layer and current-layer information.Source: §3.2 · p. 3, §3.1 · p. 3
• ResFormer variants differ by whether value-residual coefficients are fixed, tied, sparse, or trainable, including Constant-, Identity-, Sparse-, Learnable-, and Learnable ResFormer Plus.Learnable ResFormer Plus initializes stronger first-layer value contributions for higher layers.Source: §3.2 · p. 3

3.3 SVFormer

• SVFormer shares the first layer’s value vectors across later layers, computing later outputs as attention matrices multiplied by V1 and thereby saving nearly half the KV cache.Its design stores and computes value vectors only for the first layer during inference.Source: §3.3 · p. 3
• In Figure 2’s validation-loss comparisons, ResFormer achieves approximately 16.1%–20.3% reductions in both model parameters and training data across models ranging from 82M to 1.6B parameters.The 468M-parameter comparison uses 20B tokens overall and evaluates validation loss every 2B tokens.Source: Figure 2 · p. 4
• Sharing values has less negative impact than sharing keys when components are shared every two layers, according to the reported comparison.The comparison is summarized in Table 1.Source: Table 1 · p. 3, §3.3 · p. 4

4.2 ResFormer vs. NeuTRENO, DenseFormer

• 16.11% fewer parameters and 20.3% fewer training tokens let ResFormer match Transformer validation loss, while retaining similar memory and computational cost.ResFormer also outperforms NeuTRENO and DenseFormer across scaling experiments; Learnable ResFormer improves average downstream-task accuracy by 1.7 points over vanilla Transformer.Source: §4.2 · p. 5

4.3 Truly Better or Just Faster?

• Identity ResFormer significantly outperforms vanilla Transformer across all tested learning rates, indicating the gain is not explained by faster training alone.Its first-layer WV gradient norm is approximately twice as large and its WO gradient norm approximately 1.5 times larger, while analogous learning-rate changes in vanilla Transformer do not reproduce the improvement.Source: §4.3 · p. 6

4.4 Ablation Study of Value Residual

• Value skip-connections from V1 provide the main benefit: later-layer-only connections can degrade performance, whereas sparse connections concentrated in layers 6–8 perform best.Connections from V2 provide little benefit when residuals begin from H0 or H1, while later layers benefit most from supplementary V1 information.Source: §4.4 · pp. 6–7
• Sharing the current attention matrix when adding V1 yields the best mapping among tested value-residual alternatives, while only value residuals improve performance among tested residual types.Learned residual weights favor later layers, and dense and learnable variants recover patterns resembling effective sparse V1 connections.Source: §4.4 · pp. 6, 8, Table 6 · p. 7

4.5 Post-Analysis of ResFormer

• Value residuals preserve high similarity between first-layer values and later post-residual values, while increasing hidden-state information density relative to vanilla Transformer.ResFormer’s learned pre-residual values become more similar across tokens with depth, and removing attention has an impact more comparable to removing MLP layers.Source: §4.5 · pp. 8–9

4.6 SVFormer vs. GQA,CLA

• At sequence length 64,000, SVFormer has lower final loss than CLA and GQA, but at length 2,048 it underperforms GQA and is gradually surpassed by vanilla attention during training.SVFormer trains faster early, benefits from longer sequences and smaller learning rates, and can combine with GQA; matching Transformer loss requires 12.2% more parameters while nearly halving KV cache.Source: §4.6 · p. 9, Table 8 · p. 9, Figure 10 · p. 9

• ResFormer adds first-layer-to-current-layer value residuals before attention, while SVFormer extends the design by sharing a single value state across layers and achieves nearly 50% KV-cache reduction.The paper evaluates both variants across diverse language-modeling scenarios.Source: §5 · p. 9

• Learnable ResFormer converges to a relative optimum rather than identifying the optimal λ setting, and larger-scale experiments were not conducted because of computational constraints.Source: Limitations · p. 9

• The data comes from publicly available company-provided datasets that have undergone filtering, and the trained models are used only for experimental analysis rather than public deployment.Source: Ethics Statement · p. 10

• The analysis compares token-importance entropy, value-state norms, and hidden-state norms across layers in 468M models, and token-importance entropy in Llama (8B) and Mistral (7B).Source: Figure 12 · p. 12, Figure 13 · p. 12
• Attention concentration is quantified from the entropy of an importance vector derived from the attention matrix, with higher entropy indicating stronger concentration on specific tokens.The attention matrix is treated as lower triangular when constructing the importance vector.Source: §A.1 · p. 12
• Trained models exhibit attention concentration, whereas randomly initialized models show over-smoothing without concentration; language models often focus on low-semantic tokens, especially the start token (attention sink) (Xiao et al., 2024).The passage also notes analogous concentration on low-informative background areas in trained ViT models (Darcet et al., 2024).Source: §A.1 · p. 12
• Deep models show a “concentration - dispersion - concentration” pattern across layers, suggesting that information may be lost during concentrated phases.Source: §A.1 · p. 12, Figure 12 · p. 12

ResFormer alleviates attention concentration

• ResFormer reduces attention clustering relative to vanilla Transformer and mitigates attention sinks, value-state drains, and residual-state peaks.The proposed value shortcut may disrupt mutual reinforcement between attention sinks and value-state drains.Source: ResFormer alleviates attention… · p. 12
• ResFormer and SVFormer distribute token importance more uniformly than Transformer and NeuTRENO, whose start tokens can attract massive attention despite low semantic information.Source: ResFormer alleviates attention… · pp. 12, 14, Figure 14 · p. 13, Figure 15 · p. 14

Any negative effect?

• Sparse ResFormer performance depends on which layers receive value residuals and on the chosen λ proportion of first-layer values.Source: Table 10 · p. 14, ResFormer alleviates attention… · p. 14
• Higher value-residual strength improves deeper sparse networks but can reduce performance in shallower ones.Learnable-ResFormer applies value residuals mainly to later layers, while post-residual rescaling benefits shallow-sparse variants more than deep-sparse variants.Source: ResFormer alleviates attention… · p. 14

A.2 Pre-train Dataset

• 20B tokens were sampled from the original 627B-token dataset while preserving its domain proportions.The sampling target follows a scaling-law estimate requiring at least 17.5B tokens for a model with 700M non-embedding parameters.Source: §A.2 · p. 14, Table 12 · p. 15

A.3 Training Details

• Training configurations vary model layers, attention heads, hidden dimensions, and FFN dimensions across model sizes.Source: §A.3 · p. 14, Table 14 · p. 16
• Per-GPU batch size changes with training sequence length while the total tokens per batch remains fixed at 2 million.Source: §A.3 · p. 15, Table 13 · p. 15