The Lazy Neuron Phenomenon: On Emergence of Activation Sparsity in Transformers

Zonglin Li; Chong You; Srinadh Bhojanapalli; Daliang Li; Ankit Singh Rawat; Sashank J. Reddi; Ke Ye; Felix Chern; Felix Yu; Ruiqi Guo; Sanjiv Kumar

Source-linked AI summary

The Lazy Neuron Phenomenon: On Emergence of Activation Sparsity in Transformers

Zonglin Li, Chong You, Srinadh Bhojanapalli, Daliang Li, Ankit Singh Rawat, Sashank J. Reddi, Ke Ye, Felix Chern, Felix Yu, Ruiqi Guo, Sanjiv Kumar

arXiv:2210.06313v2Oct 12, 2022cs.LGcs.CLcs.CVstat.ML

TL;DR

The paper studies why trained Transformers produce sparse MLP activation maps despite using dense computations, and evaluates the phenomenon across models, tasks, data, and training conditions. It finds that sparsity is widespread, may arise partly from training dynamics, and can support efficiency, robustness, and calibration, while practical speedups remain hardware-dependent.

Problem
The paper investigates whether sparse activation maps are a broad Transformer phenomenon, what causes them, and why they matter beyond their biological analogy.
Method
The authors measure ReLU-activated MLP sparsity in T5 and ViT, test varied architectures and datasets including random or infinite-data settings, analyze training dynamics theoretically, and evaluate Top-k thresholding.
Results
Sparsity emerges broadly across tasks, layers, datasets, and configurations; experiments and theory indicate training dynamics contribute, while Top-k Transformers retain comparable performance and improve robustness and calibration.
Takeaways & Limitations
Sparse activations can reduce unnecessary zero-valued computation and provide a practical route toward more efficient and more reliable Transformers.
Takeaways & Limitations
Sparse computation may not benefit every input or training stage, and FLOP reductions may not translate into wall-time gains without suitable hardware support.

Abstract

from arXiv · show

This paper studies the curious phenomenon for machine learning models with Transformer architectures that their activation maps are sparse. By activation map we refer to the intermediate output of the multi-layer perceptrons (MLPs) after a ReLU activation function, and by sparse we mean that on average very few entries (e.g., 3.0% for T5-Base and 6.3% for ViT-B16) are nonzero for each input to MLP. Moreover, larger Transformers with more layers and wider MLP hidden dimensions are sparser as measured by the percentage of nonzero entries. Through extensive experiments we demonstrate that the emergence of sparsity is a prevalent phenomenon that occurs for both natural language processing and vision tasks, on both training and evaluation data, for Transformers of various configurations, at layers of all depth levels, as well as for other architectures including MLP-mixers and 2-layer MLPs. We show that sparsity also emerges using training datasets with random labels, or with random inputs, or with infinite amount of data, demonstrating that sparsity is not a result of a specific family of datasets. We discuss how sparsity immediately implies a way to significantly reduce the FLOP count and improve efficiency for Transformers. Moreover, we demonstrate perhaps surprisingly that enforcing an even sparser activation via Top-k thresholding with a small value of k brings a collection of desired but missing properties for Transformers, namely less sensitivity to noisy training data, more robustness to input corruptions, and better calibration for their prediction confidence.

1 Introduction

The paper shows that trained Transformers develop sparse MLP activations broadly across architectures, tasks, layers, datasets, and configurations. It investigates possible causes, measurement setup, and practical benefits of this sparsity.

1.1 An Intriguing Observation: Activations are Sparse in Trained Transformers: In T5-Base, the average percentage of nonzero activation entries falls from about 50% at initialization to 2.7% after training.The activation maps are measured at the intermediate outputs of two-layer MLPs after ReLU activation.
1.2 Prevalence, Causes, and Benefits of Sparsity: The study asks whether sparsity is broadly prevalent, what causes it, and why it matters beyond its similarity to biological neural activity.These questions organize the paper's prevalence, causal analysis, and benefits sections.
1.2 Prevalence, Causes, and Benefits of Sparsity: Trained Transformers exhibit sparse activations across language and vision tasks, training and evaluation data, layers, and model configurations.The paper reports that larger models are sparser and that sparsity also appears beyond Transformers and with different optimizers.
1.2 Prevalence, Causes, and Benefits of Sparsity: Experiments with random labels, random images, and infinite data show that datasets and data-fitting affect sparsity but do not fully explain its emergence.The authors speculate that training dynamics contribute, supported by theory showing gradients tend to decrease positive activations early in training.
1.2 Prevalence, Causes, and Benefits of Sparsity: Because many activation values are zero, sparsity can reduce inference FLOPs, while Top-k thresholding is introduced to extend sparse computation through training.The paper also reports improved robustness and confidence calibration, but notes that hardware support may limit realized wall-time gains.

2 Prevalence of Sparsity in Learned Transformers

Experiments show activation sparsity is widespread across Transformer tasks, datasets, configurations, and layers, with larger models generally sparser by percentage.

2.1 Sparsity is a Ubiquitous Phenomenon: Sparsity appears in both vision and language Transformers, across varying model configurations, and throughout all layers.The first and last layers tend to be denser than intermediate layers.
2.1 Sparsity is a Ubiquitous Phenomenon: Sparsity generalizes from training data to evaluation data, with closely aligned activation-sparsity curves for trained T5.It also persists across datasets of different scales.
2.1 Sparsity is a Ubiquitous Phenomenon: Individual neurons show heterogeneous activation frequencies rather than a fixed subset that is always active or permanently dead.The activation-frequency distribution has a long tail across 409600 token-level samples.
2.2 The Larger, the Sparser: Deeper and wider T5 Transformers have lower percentages of nonzero activation entries, although wider models have higher absolute nonzero counts.Many middle layers of a 32-layer model have fewer than 1% nonzero entries.

3 Sparsity from Training Dynamic?

Controlled experiments test whether labels, data structure, or data-fitting explain activation sparsity, while theory points toward training dynamics as a contributing mechanism.

3.1–3.3 Controlled Experiments: The experiments separately test hypotheses that sparsity arises from meaningful labels, natural-data structure, or over-parameterized models fitting finite datasets.Random labels, random images, and infinite data are designed to remove or weaken these factors.
3.1–3.3 Controlled Experiments: Random labels, random images, and infinitely many random image-label pairs alter sparsity across layers but do not eliminate it.With random images, every layer has fewer than 10% activated neurons; infinite-data training yields roughly 10%–20% nonzeros in some middle layers.
3.4 Discussion: Sparsity from Training Dynamic?: Random-label, random-image, and infinite-data experiments all retain substantially fewer nonzero activations than the 50% initialization level.These results show that none of the tested factors fully explains sparsity, although each affects its level.
3.4 Discussion: Sparsity from Training Dynamic?: A theorem shows that, at random initialization, the expected gradient for a positive activation is positive under MSE or cross-entropy loss.Negative-gradient training therefore tends to reduce positive activation magnitudes, though the theorem does not apply directly to later iterations.

4 Efficient, Robust, and Calibrated: Sparsity is All You Need?

Activation sparsity can reduce Transformer computation and, through Top-k thresholding, provide training-time sparsity while preserving accuracy and improving robustness and calibration.

4.1 Efficiency for Free: Sparse intermediate activations reduce the second MLP layer’s FLOP count from 2d_model × d_ff to 2d_model × s.With 2.7% nonzeros averaged across layers, the resulting reduction is substantial and likely larger in bigger models.
4.1 Efficiency for Free: Approximate nearest-neighbor search can reduce first-layer MLP computation to sublinear complexity in d_ff.The approach treats selecting nonzero outputs as a maximum inner-product search problem.
Limitations: Efficiency benefits are limited because some inputs may produce denser activations and sparsity appears only after training.Existing hardware also poorly supports unstructured, data-dependent sparse computation.
4.2 Sparsity in Training via Top-k Transformers: Top-k Transformers closely match vanilla accuracy with k = 64 for T5 and k = 256 for ViT, without additional hyperparameter tuning.Figure 6 reports an approximately 0.3% drop for T5-3B with k = 128 and ViT with k = 256.
4.2 Sparsity in Training via Top-k Transformers: Around 10% wall-time reduction is observed for T5-11B with k ≤128 during unbatched greedy decoding on TPUv4.The reduction becomes smaller when k increases to 256.
4.3 Robustness and Calibration: Top-128 ViT remains on par with ViT on natural accuracy while improving robustness to corrupted labels, input perturbations, and calibration.The comparison covers label corruption, Gaussian, impulse, and shot noise, plus ECE.

5 Related Work

Prior work studies sparsity for efficiency, robustness, explainability, data modeling, and theory, while this paper emphasizes naturally emerging Transformer activation sparsity.

Sparsity for Efficiency: Activation sparsity has been explored for efficiency in fully connected, convolutional, and large Transformer models.The cited literature includes sparsity in both weights and activation maps.
Sparsity for Efficiency: The paper argues that naturally sparse Transformer activations and greater sparsity in larger models may guide efficient training of future large models.This contrasts with prior practices that often rely on trial-and-error designs because sparsity may hurt performance.
Sparsity for Robustness: Related robustness work uses compression or sparse data corruption, whereas this paper studies robustness from activation-map sparsity.The distinction is presented as a difference from the cited prior approaches.
Other Uses of Sparsity: Other research uses sparsity for post-hoc interpretability, sparse modeling of natural signals, or theoretical analysis of over-parameterized models.These lines address neuron meanings, data representations, implicit regularization, optimization, and related theory.

6 Discussion: Transformers are Parsimonious Models?

The discussion interprets sparse Transformer activations as a form of parsimony that may help explain generalization, while acknowledging unresolved design and hardware challenges.

Parsimony: Transformers use only a small fraction of their parameters for each input, making them parsimonious models despite dense architecture and computation.The discussion connects this behavior to the law of parsimony.
Generalization: Improved robustness and calibration provide evidence that sparsity may act as a pertinent prior for good generalization.This is presented as evidence rather than a definitive causal explanation.
Outlook: The paper presents Top-k thresholding as a proof of concept rather than the best method for inducing sparsity.It calls for more principled approaches to introducing sparsity in deep networks.
Outlook: Reduced FLOP counts may not translate into proportional wall-time gains because current platforms favor dense computation over sparse computation.The discussion motivates future hardware designed for sparse computation.

Appendices

The appendices provide implementation details, additional experiments, a theorem proof, and further analysis of activation sparsity.

Appendices: The appendices cover implementation details, experiments on other architectures and optimizers, additional Section 4 results, a proof of Theorem 3.1, and insights into sparsity emergence.These materials are organized in Sections A through E.

A Implementation Details

The experiments use standard T5 and ViT training configurations, with model settings summarized in Table A.1.

T5: T5 training uses dropout, Adafactor, an inverse square root schedule, warm-up, span corruption, and 100,000 training steps.The batch size is 256, and the fixed warm-up learning rate is 0.01 for the first 10,000 steps.
ViT: ViT training uses ADAM with β1 = 0.9 and β2 = 0.999 and runs for 180 epochs on ImageNet-1k.Other settings follow the referenced ViT configuration.
Model configurations: Table A.1 summarizes the varying T5 and ViT configurations used in the experiments.The table includes model dimensions and layer counts for the architectures.

B.1 Sparsity and Network Architecture

Activation sparsity appears across Transformer, MLP-Mixer, and convolutional architectures, while its level and layerwise pattern depend on architecture, width, depth, normalization, and activation function.

BERT: BERT Base and Large show high sparsity below 10%, while negative shifts in pre-activation means emerge during training.The result holds across BERT’s intermediate MLP layers, with histograms shown for early and late layers.
Activation functions: GeLU and Sigmoid show pre-activation behavior similar to ReLU, whereas Tanh lacks sparsity and has significantly worse accuracy.This comparison links the activation-function distribution to both sparsity and accuracy outcomes.
MLP-Mixer: Channel-mixing MLPs in MLP-Mixer become denser than comparable ViT layers after the first four layers, whereas token-mixing MLPs exceed 50% nonzero entries.The first four channel-mixing layers and ViT have nearly identical sparsity levels.
ConvNets: In ResNets, output-near layers are sparser, while within-stage sparsity follows distinct patterns for first, second, and third layers.First-layer nonzeros decrease within stages and jump at boundaries; second-layer values decrease then stabilize; third-layer values increase slightly before boundary drops.
ConvNets: ResNets produce denser activation maps than Transformers, with more than 10% nonzero entries in every examined layer.The paper suggests batch normalization before activation as one possible explanation and tests its effect separately.
Width and architecture: Wider models generally have lower nonzero percentages but higher nonzero counts, and this width trend holds for both Transformers and ResNets.For ResNets, the exception is the very last layer in the examined configuration.

C.1 Top-k Does not Significantly Affect Training Convergence

The paper evaluates whether enforced sparsity changes optimization and predictive behavior, finding that Top-k training converges similarly while moderate L1 regularization improves robustness and calibration.

Top-k convergence: ViT and Top-k ViT have similar convergence speed during ImageNet-21K training.The learning-curve comparison covers k ∈ {64, 128, 256}.
L1-induced sparsity: The paper also tests L1-ViT by adding a weighted sum of activation-map L1 norms across all ViT layers.The regularization weight λ is varied over {0.001, 0.01, 0.1, 1.0}.
L1-induced sparsity: L1 regularization lowers activation nonzero percentages, while λ = 0.001, 0.01, and 0.1 improve robust accuracy and calibration without reducing natural accuracy.At λ = 1.0, sparsity increases sharply but natural accuracy declines, with robust accuracy and ECE comparable to ViT.
L2 comparison: L2 regularization also reduces nonzero percentages and improves robustness and calibration, but less strongly than L1 regularization.The paper attributes the difference to L1’s relatively greater emphasis on suppressing small activation entries.

D Proof of Theorem 3.1

This section proves Theorem 3.1 through expectation-based algebra under stated distributional assumptions, concluding with a strict inequality used to finish the argument.

Proof setup: The proof starts from derivatives of an arbitrary loss and substitutes expressions into the theorem’s defining equations.Several steps evaluate numerator and denominator terms before taking expectations.
Expectation argument: The argument takes expectations over one selected variable and all remaining entries of V, using independence and a zero-mean assumption.The proof explicitly identifies which expectation applies to the selected entry and which applies to the other entries.
Auxiliary lemma: Lemma D.1 supplies the strict inequality needed by assuming a non-degenerate random variable and positive constants.The lemma’s proof uses monotonicity and an interval with positive probability.
Conclusion: The proof concludes that the inequality in D.20 is strict and states that the MSE-loss proof is complete.The strict conclusion is reported immediately before the proof-ending statement.

E Insights from Sparsity in MLPs

Two-layer MLPs also exhibit activation sparsity, with sparsity varying systematically with model width, data regime, and learning rate. The results suggest that activation sparsity is associated with parameterization and is stronger than ordinary ReLU sparsity.

Sparsity with Increased Learning Rate in 2-Layer MLPs: Sparsity also appears in convolutional networks and pure MLPs, although the reported convolutional-network activations are less sparse.The cited discussion specifically mentions ResNet and two-layer MLPs as architectures exhibiting sparsity.
Sparsity and Over/Under-Fitting in 2-Layer MLPs: Sparsity in two-layer MLPs follows a unimodal pattern with width for random data, increasing during fitting and decreasing after perfect fitting.The reported relationship places the highest nonzero rate near the interpolation threshold and lower rates in both under- and over-parameterized regimes.
Sparsity and Over/Under-Fitting in 2-Layer MLPs: For infinite data, the nonzero rate is close to 0 when the model cannot fit the data, while MNIST models show decreasing nonzero rates as width increases.Infinite-data models have training accuracy of 0.1, matching random guessing; MNIST models in the experiment can fit their training data.
Sparsity with Increased Learning Rate in 2-Layer MLPs: Larger learning rates produce sparser activation maps in two-layer MLPs.This comparison is reported using the larger-learning-rate setting corresponding to Figure E.2.
Sparsity with Increased Learning Rate in 2-Layer MLPs: Trained ReLU activations are far sparser than the roughly 50% nonzero outputs typical of randomly initialized networks, reaching 3% nonzeros in T5.ReLU yields exact zeros for non-positive inputs, allowing sparsity to be measured directly without thresholding.
Sparsity with Increased Learning Rate in 2-Layer MLPs: Per-input sparsity does not imply that neurons can be removed, because different inputs can activate different neurons and collectively reach all neurons.The paper reports a long-tail distribution of neuron activation across inputs.

Q. Is sparsity unique to Transformer? Or is it present in other architectures as well?

Activation sparsity is not unique to Transformers: the paper reports it in convolutional networks and pure MLPs as well. The cited comparison indicates that ResNet is less sparse, while pure MLPs also exhibit sparsity.

Q. Is sparsity unique to Transformer? Or is it present in other architectures as well?: Activation sparsity also emerges in convolutional networks, specifically ResNet, and in pure MLPs.The paper notes that the convolutional-network activations are less sparse than those observed in the main Transformer setting.