Source-linked AI summary
A Unified View of Attention and Residual Sinks: Outlier-Driven Rescaling is Essential for Transformer Training
Zihan Qiu, Zeyu Huang, Kaiyue Wen, Peng Jin, Bo Zheng, Yuxin Zhou, Haofeng Huang, Zekun Wang, Xiao Li, Huaqing Zhang, Yang Xu, Haoran Lian, Siqi Zhang, Rui Men, Jianwei Zhang, Ivan Titov, Dayiheng Liu, Jingren Zhou, Junyang Lin
TL;DR
The paper asks what functional role emergent attention and residual outliers play in language models and studies whether they work through normalization-based rescaling. Across architectures and training settings, it finds that outlier-driven rescaling supports training and performance, while parameter absorption and explicit gating reduce outliers while preserving or improving model and quantization behavior.
Problem
The paper investigates whether attention sinks and residual sinks are merely harmful outliers or serve a functional role in transformer computation.
Method
The paper validates an outlier-driven rescaling hypothesis across attention and residual normalizations, architectures, model sizes, and training token counts.
Results
Outliers work with normalization to support stable training and performance, while learnable parameter absorption and gated rescaling reduce explicit outliers without sacrificing model quality.
Takeaways & Limitations
Explicitly providing rescaling can preserve or improve performance, reduce architecture sensitivity, and improve robustness to aggressive low-bit quantization.
Takeaways & Limitations
The paper does not investigate why rescaling is necessary for effective training or representation learning.
Abstract
from arXiv · showhide
We investigate the functional role of emergent outliers in large language models, specifically attention sinks (a few tokens that consistently receive large attention logits) and residual sinks (a few fixed dimensions with persistently large activations across most tokens). We hypothesize that these outliers, in conjunction with the corresponding normalizations (\textit{e.g.}, softmax attention and RMSNorm), effectively rescale other non-outlier components. We term this phenomenon \textit{outlier-driven rescaling} and validate this hypothesis across different model architectures and training token counts. This view unifies the origin and mitigation of both sink types. Our main conclusions and observations include: (1) Outliers function jointly with normalization: removing normalization eliminates the corresponding outliers but degrades training stability and performance; directly clipping outliers while retaining normalization leads to degradation, indicating that outlier-driven rescaling contributes to training stability. (2) Outliers serve more as rescale factors rather than contributors, as the final contributions of attention and residual sinks are significantly smaller than those of non-outliers. (3) Outliers can be absorbed into learnable parameters or mitigated via explicit gated rescaling, leading to improved training performance (average gain of 2 points) and enhanced quantization robustness (1.2 points degradation under W4A4 quantization).
1 Introduction
The paper argues that attention and residual outliers work with softmax and RMSNorm to rescale non-outlier components, rather than acting primarily as direct contributors. Removing or clipping them harms training, while parameter absorption and explicit gating reduce outliers while preserving or improving performance and quantization robustness.
- Outliers interact with softmax attention and RMSNorm to rescale non-outlier components, a mechanism the paper calls outlier-driven rescaling.
- Clipping outliers or removing normalization reduces outliers but degrades model performance and training stability.
- RMSNorm weights for outlier-prone dimensions can be much smaller than average, such as 0.006 versus 1, supporting a rescale-factor role.
- Residual sinks can be absorbed into learnable parameters, eliminating explicit residual-stream outliers while retaining outlier-driven rescaling.
- GatedNorm mitigates residual sinks while preserving or enhancing performance and improving quantization performance through smoother activations.
- Gating-based rescaling reduces reliance on outliers, enabling GLU to match or exceed SwiGLU and weakening sensitivity to architecture choice.
2 Outliers in Large Language Models
The paper examines attention and residual outliers in pre-norm transformers by recording attention maps and residual activations across layers and models. Reordered activation dimensions reveal residual-sink structures resembling attention-sink patterns, with architecture-specific differences in their prominence.
- The analysis focuses on pre-norm transformers whose residual stream evolves through layer updates Hi+1 = Hi + Fi(Hi).
- The study analyzes four models by recording attention maps and residual activations across all layers using the same input sequences.
- Feature dimensions are sorted by average activation so consistently large dimensions appear together in visualizations.
- Qwen3-235B-A22B combines a first-token attention sink with massive activations in dimensions 1806 and 1423, while dimension 1423 is large across most tokens.
- Qwen3-Next has weaker attention sinks and no prominent massive activations, yet dimension 1572 still forms a residual sink.
- GPT-OSS removes attention sinks and massive activations from real tokens through learnable sinks, consistent with an input-independent bias interpretation.
3 Outlier-Driven Rescaling
The section tests whether outliers work with normalization to rescale non-outlier components, finding that removing or constraining them harms stability or performance, while explicit rescaling mitigates sinks and improves robustness to architectural choices.
- Experiments primarily use a 2B-parameter pre-norm transformer trained on 120B tokens, with variants evaluated under consistent settings.
- Replacing normalization with Dynamic Tanh reduces outliers but degrades training stability and final performance because it cannot provide outlier-driven rescaling.
- Direct activation clipping preserves normalization but disrupts outlier-driven rescaling, causing performance degradation and sometimes training divergence.
- Explicit gating after normalization reduces residual sinks while preserving or improving performance, and elementwise sigmoid gating outperforms tensorwise gating.
- Gated rescaling reduces reliance on outliers, allowing sigmoid-based GLU to match or exceed SwiGLU and making the model less sensitive to architecture choices.
- Removing normalization reduces residual sinks but harms performance and stability, while linear attention can eliminate maximum attention outliers without eliminating residual sinks.
4 Scaling Outlier Mitigations and Deployment-Level Quantization
Large-scale experiments compare parameter absorption and explicit gated rescaling for outlier mitigation, including their training-loss and aggressive quantization behavior. GatedNorm maintains smoother activations, achieves lower final loss, and provides the strongest reported quantization robustness.
- Scaling Outlier Mitigations: GatedNorm avoids the early outlier surge seen in the baseline and PreAffine while maintaining consistently low activation magnitudes.PreAffine’s initial surge occurs within the first 10% of training and may reflect temporary reliance on activation outliers before rescaling is absorbed into parameters.
- Scaling Outlier Mitigations: GatedNorm achieves a lower final loss than the compared models, with the performance gap emerging in the mid-to-late training stages.
- Deployment-Level Quantization: GatedNorm achieves +1.0 point over baseline on knowledge tasks and exceeds +2.0 points on STEM and Code tasks under the reported comparisons.The experiments evaluate MoE-7B-A-2B and MoE-24B-A3B settings trained on 1.2T and 500B tokens, respectively.
- Deployment-Level Quantization: GatedNorm has the smallest FP4 W4A4 performance drop at -1.23 points, compared with -1.50 for GA and -2.76 for PreAffine.On MGSM, only GatedNorm keeps degradation within 5 points, while the other methods incur nearly 10-point losses.
- Deployment-Level Quantization: GatedNorm explicitly supplies outlier-suppressing rescaling, whereas PreAffine relocates outliers but still relies on them for outlier-driven rescaling.The reported gating behavior assigns smaller scores to dimensions with large |y|, producing smoother final activations.
5 Related Works
Related work connects transformer outliers to normalization, special-token attention patterns, optimization, and functional scaling roles. Existing mitigation approaches target training dynamics, architectural mechanisms, or the outliers’ effects on quantization and performance.
- Related Works: BERT outliers in fixed dimensions have been attributed primarily to LayerNorm weight and bias parameters and associated with special-token attention patterns.
- Related Works: GPT models exhibit input-independent outliers in fixed dimensions, and prior work links some outliers to normalization while distinguishing them from attention sinks.These outliers are reported to hurt both training and inference quantization, motivating mitigation methods.
- Related Works: Training-time mitigation strategies include weight decay, gradient clipping, constrained weight variance, explicit regularization, and studies of optimizer effects.
- Related Works: Prior functional accounts characterize attention outliers as context-aware scaling factors and identify normalization-driven signal scaling in StyleGAN feature maps.
6 Conclusion
The paper concludes that transformer outliers have functional roles: together with softmax or RMSNorm, they rescale non-outlier features. Explicit gated rescaling can reduce outliers while preserving or improving performance and quantization robustness.
- Conclusion: Outlier-driven rescaling works with softmax and RMSNorm to adjust the magnitude of non-outlier features and supports stable training and strong performance.
- Conclusion: Providing gating-based rescaling reduces activation outliers while maintaining or even improving performance.
- Conclusion: Gated rescaling produces smoother activations and significantly improves quantization robustness, especially under aggressive low-bit settings.
Limitations
The paper empirically demonstrates the importance of outlier-driven rescaling but does not explain why rescaling is necessary for effective training or representation learning.
- Limitations: Why outlier-driven rescaling is necessary for effective training or representation learning remains an open theoretical question.
A.1 A Brief Calculation On How Outlier Interacts With the Normalization Layer
The calculation shows how a large outlier dimension can rescale the norm of features after normalization. This effect strengthens as the outlier magnitude increases.
- The analysis models an input feature h with normalization parameters λ and a single outlier dimension d.The outlier dimension is assumed to have a very small affine parameter relative to the largest parameter.
- The upper bound on the feature norm after LayerNorm decreases as the outlier becomes larger.Changing the outlier magnitude therefore provides a mechanism for rescaling feature norms.
A.2 Comparison Between Different Outliers
Attention sinks and residual sinks are presented as analogous normalization-associated outliers that rescale non-outlier components despite differing in whether they are token- or dimension-specific.
- Both attention sinks and residual sinks arise at normalization layers and modulate the scale of non-outlier components.The comparison places attention sinks at softmax normalization and residual sinks at RMSNorm.
- Attention sinks are token-specific, whereas residual sinks are dimension-specific, but their functional roles are analogous.
A.3 Efficiency Analysis of GatedNorm
The efficiency analysis evaluates GatedNorm across model scales and architectures. Its relative overhead decreases with larger hidden dimensions and is further diluted in MoE models.
- Efficiency setup: The end-to-end overhead study varies hidden dimension while keeping the gating rank fixed at 16.It uses an 8-layer dense transformer with ZeRO-1, and the PreAffine variant uses a fused Triton kernel.
- Scaling with hidden dimension: GatedNorm overhead becomes relatively smaller at larger hidden dimensions because its cost scales linearly while attention and FFN GEMMs scale quadratically.
- Scaling with hidden dimension: Kernel-launch overhead matters more at smaller hidden dimensions, while larger workloads amortize launch costs and improve pipeline efficiency.
- Architecture effects: GatedNorm has lower relative overhead in MoE models because communication and routing costs dominate the training step.The evaluated suite includes both dense and MoE architectures, including hybrid linear-softmax attention configurations.