PaperlayerView exact revision on arXiv

Source-linked AI summary

Bug or Feature$^2$: Weight Drift, Activation Sparsity and Spikes

Egor Shvetsov, Aleksandr Serkov, Shokorov Viacheslav, Redko Dmitry, Vladislav Goloshchapov, Evgeny Burnaev

arXiv:2605.17659v2cs.LG

TL;DR

The paper investigates how loss functions, activation asymmetry, and normalization shape early training dynamics and activation sparsity. It combines theoretical analysis with experiments across architectures and controlled sparsification studies. The results identify negative weight drift and an architecture-dependent sparsity cliff, while showing that some squared activations improve GPT-nano performance but require careful handling of spikes.

  • Problem

    The training mechanisms behind activation sparsity and its relationship to model quality remain only partially understood.

  • Method

    The paper proves expected positive gradients for positive pre-activations and empirically evaluates drift, controlled sparsity, activation functions, and computational strategies across architectures.

  • Results

    The study finds that sparsity is largely preserved through moderate levels before an architecture-dependent accuracy cliff, while ReLU2 achieves the best GPT-nano validation loss among tested activations.

  • Takeaways & Limitations

    Activation sparsity can be treated as a controllable consequence of the loss–activation–normalization interaction rather than solely as a data-emergent property.

  • Takeaways & Limitations

    The formal results assume zero-mean i.i.d. weights and hold strictly at initialization, while smooth-activation extensions are empirical; language-model experiments use only GPT-nano on FineWeb.

Abstract

from arXiv · show

The design of modern neural architectures has converged through incremental empirical choices, yet the mechanisms governing their training dynamics remain only partially understood. We identify and analyze a negative weight drift induced by the interaction between standard losses and positively biased activation functions. We prove that under MSE or cross-entropy loss, the gradient with respect to positive pre-activations is non-negative in expectation at initialization, driving downstream weights toward negative values during early training. The drift is intrinsic to optimization rather than data, and persists across architectures (MLP, ResNet, ViT, GPT-nano, MP-SENe) and asymmetric activation functions (ReLU, GELU, SiLU). Coupled with ReLU, weight drift produces activation sparsity reaching up to 90\% in GPT-nano. We characterize the sparsity-accuracy tradeoff across 79 configurations and identify a sharp accuracy cliff above $\sim$70\% activation sparsity. While ReLU$^2$ achieves a good sparsity--accuracy ratio in GPT-nano, it pathologically amplifies identified activation spikes in intermediate transformer layers. Clipping resolves this while preserving the representational benefits of squaring: clipped ReLU$^2$ outperforms its unclipped version, and GELU$^2$ achieves the lowest validation loss on GPT-nano. Code is available at https://github.com/On-Point-RND/BugOrFeature.

1. Formal Illustration of Negative Weight Drift

The paper proves that standard MSE and cross-entropy losses produce non-negative expected gradients for positive ReLU pre-activations at initialization, causing negative weight drift throughout intermediate layers. The proof relies on effective downstream weight matrices whose rows have zero mean and non-negative expected cross-correlations under ReLU survival conditioning.

  • Effective weight matrix: ReLU networks represent each activation gate as a binary diagonal matrix that selects active neurons, enabling analysis through an effective downstream weight matrix.For a fixed input, the activation pattern is fixed and subsequent layers can be folded into V_eff.
  • Effective weight matrix: Zero-mean initialization makes each row of the effective weight matrix have zero expectation.The expectation factors through the outermost random weight matrix.
  • Scope: The smooth-activation extension is supported empirically rather than formally, while the theoretical argument assumes zero-mean i.i.d. weights and applies strictly at initialization.The paper explicitly limits its formal proof to ReLU.
  • Effective weight matrix: Conditioning on ReLU survival induces non-negative expected inner products between effective-weight rows, because surviving random vectors share a positive component along the input direction.The remaining coordinate contributions cancel under the zero-mean i.i.d. assumption.
  • Positive expected gradients: At initialization, gradients with respect to positive pre-activations are non-negative in expectation under both MSE and softmax cross-entropy loss.The formal results are stated in parallel for regression and classification.
  • Propagation through depth: The positive-gradient property holds at every intermediate layer, producing non-positive expected weight updates that shift pre-activations downward and reinforce the cycle.The formal extension is local to contiguous stacks of linear layers and ReLU gates within larger architectures.

2. Empirical Results for Negative Weight Drift

Experiments confirm negative weight drift across optimizers, data conditions, architectures, and activation functions. Drift is fastest with momentum and larger learning rates, appears even on random data, and commonly produces substantial negative pre-activation fractions.

  • Optimizer and learning rate: Momentum substantially accelerates drift, while higher learning rates produce faster and larger drift within each optimizer.Momentum-based optimizers show a rapid initial surge followed by a plateau; plain SGD progresses more slowly and near-linearly.
  • Data independence: Negative weight drift arises across all activation functions when the MLP is trained on entirely random data, showing that the effect is intrinsic to optimization rather than data.For ReLU, deeper layers accumulate more negative weight means; GELU and SiLU show weaker depth ordering but retain overall drift.
  • Architectural breadth: The positive-gradient property is observed across MLP, MaxViT-Tiny, MP-SENet, and ResNet-18, with covariance terms remaining orders of magnitude smaller than weight means.This supports the assumptions used in the formal analysis across multiple architectures.
  • Activation consequences: Negative pre-activations typically comprise 60% to 80% of values across MLP, ResNet, ViT, and GPT configurations.ResNet with batch normalization is the exception because mean-centering directly disrupts the drift.

3. Post-activation Sparsity and Performance

The paper treats drift-induced sparsity as controllable and evaluates its effect across 79 model–sparsity configurations. Performance remains stable through moderate sparsity but collapses beyond an architecture-dependent cliff, with sparsification mechanism less important than sparsity level.

  • Experimental design: The study evaluates controlled post-activation sparsity across four architectures and 79 model–sparsity pairs.It compares Top-K sparsity with Percentile Centering and distinguishes unstructured from channel-wise sparsity in ResNet-18.
  • Sparsification mechanisms: Percentile Centering shifts pre-activations by a selected percentile so that a controlled fraction falls below zero before ReLU.Top-K instead retains the k% largest activations and hard-zeros the rest.
  • Sparsity–performance curve: Performance remains nearly flat for sparsity s ≲0.7 before degrading sharply beyond a critical threshold.The fitted power law gives A = 0.978, B = 0.635, and N = 16.72, quantifying the sharp transition.
  • Architecture dependence: At 85% sparsity, a plain MLP collapses to near-random performance at ≈10% accuracy, whereas skip connections preserve 74.0% of peak accuracy.GPT-nano validation loss remains nearly flat up to s ≈0.91, illustrating strong architecture dependence.
  • Mechanism comparison: The sparsity level dominates predictive accuracy more strongly than whether sparsification uses Top-K or Percentile Centering.Adding a mechanism indicator raises R2 only marginally from 0.565 to 0.618.

4. Activation Functions and the Sparsity–Accuracy Tradeoff

The paper compares activation functions and sparsification strategies across architectures, finding that sparsity–accuracy tradeoffs depend strongly on normalization and activation choice. ReLUfication offers broad sparsity with minimal accuracy loss, while ReLU2 is particularly effective on GPT-nano but normalization-sensitive.

  • Experimental setup: Across four architectures, the study evaluates GELU and ReLU baselines alongside NoisyReLU, SUGARBSiLU, ReLU2, and ReLUfication.The comparison targets alternative mechanisms for achieving sparsity while retaining model performance.
  • Results: GELU is the strongest general-purpose baseline, achieving the highest accuracy in four of five classification settings and the second-best GPT-nano loss.GELU produces essentially no natural sparsity in these experiments.
  • Results: ReLU2 is normalization-sensitive, collapsing to 10% accuracy in unnormalized MLP and batch-normalized ResNet, but achieving 48.0% with RMSNorm on MLP and the best GPT-nano loss of 3.250.The GPT-nano result compares against GELU’s loss of 3.260.
  • Results: SUGARBSiLU averages 68.0% negative pre-activations but underperforms every other activation, whereas NoisyReLU reaches 65.2% sparsity versus ReLU’s 66.2% with competitive accuracy.NoisyReLU’s average accuracy is 60.3%, compared with 59.1% for ReLU.
  • Results: ReLUfication yields 55–74% post-activation sparsity after one fine-tuning epoch, while accuracy decreases by less than 1 percentage point.The conversion changes the activation thresholding behavior without substantially changing the fraction of negative pre-activations.

5. Pathological Spikes Amplification with Squared Activation Functions

The study finds that intermediate GPT-nano activation spikes arise before the down-projection and are amplified by squared nonlinearities. Clipping suppresses harmful extremes while retaining the benefits of squaring, with GELU2 achieving the best reported overall performance.

  • Spike origin: Layer-wise analysis finds consistent maximum-activation spikes between layers 2–4, measured by the input range to the down-projection.The MLP analysis locates the spikes after activation and quantifies them as maximum minus minimum.
  • Spike origin: The spikes emerge after activation: anomalously large up-projection pre-activations are further amplified by the nonlinearity before entering the down-projection.The analyzed MLP block contains up-projection, activation, and down-projection stages without a gating interaction.
  • Normalization: Normalization with centering or quantile shift shrinks spike ranges and also reduces input ranges to the up-projection.The findings indicate an interaction between normalization, activation functions, and the weight-drift mechanism.
  • Squared activations: ReLU2 produces Layer 2 values exceeding 1000, compared with moderate baseline spikes of roughly 20–43 for ReLU and GELU.Squaring large pre-activation values amplifies an already-existing tendency rather than introducing the spikes.
  • Clipping: ReLU2 clip50 improves validation loss to 3.236 from 3.251 for unclipped ReLU2, while clip15 degrades performance relative to ReLU2.The result identifies the most extreme spike values as harmful rather than informative.
  • Clipping: GELU2 achieves the best overall performance at 3.233 while producing substantially lower spikes than ReLU2.The study presents GELU2 as a possible starting point for ReLUfication toward ReLU2.
  • Scope: On MaxViT, no squared or clipped activation outperforms GELU, which reaches 70.30% performance.This result suggests the benefits of squared activations may depend on model setting or clipping threshold.

6. From Weight Drift to Computational Efficiency

The paper exploits early stabilization of weight drift to freeze normalization and percentile statistics during training. Accumulation Stop recovers throughput after warm-up while preserving training stability and model quality in the tested vision models.

  • Motivation: Dynamic percentile and centering statistics impose non-negligible overhead, motivating replacement with fixed values after the network settles.The paper notes that quantile algorithms are poorly parallelisable on GPUs and that LayerNorm can also bottleneck throughput.
  • Accumulation Stop: During warm-up, exponential moving averages track statistics; after Twarm steps, Accumulation Stop freezes the buffers and removes dynamic computation.The reported EMA uses γ = 0.9999.
  • Throughput: Dynamic computation reduces throughput by approximately 25–30% during warm-up, but throughput returns near optimized LayerNorm baseline after statistics are frozen.The throughput measurements use a non-optimized PyTorch implementation.
  • Quality: For DiT-S/2, Percentile Shift lowers FID from 49.40 to 48.21 relative to standard LayerNorm.The paper reports this improvement without a full explanation.
  • Quality: For MaxViT-Tiny, quality changes remain within 2% relative change, from the 70.30% baseline to 69.75%.The reported metric is top-1 ImageNet-1K validation accuracy.
  • Stability: Before Accumulation Stop, all configurations converge to approximately 0.40 average absolute Z-score by step 100, and frozen-statistics trajectories remain continuous afterward.Accumulation Stop and no-Accumulation-Stop trajectories remain closely aligned across the tested activation and normalization configurations.

7. Limitations and Discussion

The paper identifies scope boundaries for its theory and experiments, including initialization assumptions, ReLU-only formal proofs, and architecture-dependent empirical settings.

  • The theoretical results assume zero-mean i.i.d. weights and hold strictly only at initialization.
  • The formal proof covers ReLU, while extensions to smooth activations are supported empirically rather than formally.
  • Language modeling is evaluated only on FineWeb with GPT-nano, and squared-activation clipping thresholds appear architecture-dependent.
  • Squared activations improve autoregressive GPT-nano results but do not outperform plain GELU on MaxViT, suggesting modality- or objective-specific interactions remain uncharacterized.
  • The robustness of ViT and GPT-nano to aggressive sparsification is not explained, and skip connections alone do not fully account for it.

8. Conclusion

The paper reframes activation sparsity as a controllable consequence of loss, activation, and normalization choices rather than data, while identifying a sharp quality boundary for aggressive sparsity.

  • Activation sparsity is presented as a controllable consequence of the loss–activation–normalization triple rather than an emergent property of data.
  • The sharp ∼70% sparsity cliff shows that aggressive activation sparsity can be achieved without significantly degrading quality.
  • Understanding these interactions offers practical insights for future architecture development, particularly as modern LLMs increasingly use non-centering normalization.

9. Related Work

Related work connects activation sparsity and internal representation dynamics to optimization, normalization choices, and computational acceleration, while this paper broadens the analysis across architectures and activations.

  • Batch Normalization was proposed to normalize layer inputs to zero mean and unit variance in response to changing layer-input distributions during training.
  • RMSNorm rescales activations by their root-mean-square without subtracting the mean, and is widely used in contemporary large language models.
  • Activation sparsity has been observed in trained transformers and on random data, implicating optimization rather than the data distribution as the causal factor.
  • The paper extends this phenomenon across MLP, ResNet, MP-SENet, ViT, and GPT architectures and across ReLU, GELU, and SiLU activations.
  • Activation sparsity has motivated decoding acceleration through sparse vector–dense matrix multiplication, with specialized kernels achieving up to 2× speedup.
  • Intermediate-layer spikes occur in both gated and non-gated transformer blocks; normalization reduces but does not eliminate them, while controlling amplitude improves downstream performance.

A. Theorem & Proof: Positive Expected Gradient for MSE loss

The theorem analyzes a ReLU multilayer perceptron at initialization by differentiating the MSE loss through an intermediate layer and averaging over downstream weights. It shows that positive pre-activations have non-negative expected gradients, with strict positivity under active conditions.

  • At initialization, downstream effective weights are treated as independent of the intermediate pre-activation and target, enabling expectation over downstream weights.
  • The MSE proof expands the downstream matrix–vector product columnwise, where only the matching activation term depends on the selected pre-activation.
  • For a positive ReLU pre-activation, the derivative of the network output with respect to that pre-activation is the corresponding downstream weight column.
  • For ReLU, every activation is non-negative and expected downstream column inner products are non-negative, making each gradient summand non-negative.
  • The expected gradient is non-negative, with strict inequality whenever the selected pre-activation is positive.
  • An empirical covariance term involving the input and pre-activation oscillates around zero during early iterations.

B. Theorem & Proof: Positive Expected Gradient for Cross-Entropy Loss

The cross-entropy analysis decomposes the softmax gradient to establish non-negative expected gradients for positive pre-activations, implying non-positive expected weight updates. Experiments then show that drift is broadly robust but can vary with architecture and activation.

  • Cross-entropy derivation: The softmax gradient is decomposed into a constant component and a centered component because the probability simplex prevents direct application of the zero-mean theorem.The centering projection removes the constant component, leaving the term analyzed for expected sign.
  • Cross-entropy derivation: To leading order, every cross-entropy gradient summand is non-negative by the zero-mean effective-weight result and ReLU non-negativity.The derivation includes the centering factor induced by the softmax constraint and higher-order corrections.
  • Cross-entropy derivation: The expected weight update is non-positive because the activation-gradient and input factors are both non-negative in expectation.Strict negativity holds under the stated positive-pre-activation conditions, up to higher-order softmax-linearization corrections.
  • Extended empirical validation: At K = 0.10, retaining 10% of activations collapses the gradient mean and leaves weight means nearly flat across layers.The result comes from a three-layer GELU MLP trained on random inputs, indicating that extreme sparsity can starve informative updates.
  • Extended empirical validation: Negative weight drift accumulates across GELU, ReLU, and SiLU in ResNet-18 and appears across diverse architectures, though MP-SENet shows activation-dependent exceptions.In MP-SENet, ReLU follows the expected negative drift while GELU and SiLU show positive weight-mean drift despite positive gradients.
  • Extended empirical validation: In an unnormalized MLP, ReLU and NoisyReLU produce the strongest drift and reach approximately 60% sparsity, while skip connections reduce sparsity to approximately 40%.The skip connection does not qualitatively change the drift trajectory, suggesting activation bias is the primary driver in this experiment.

D. Extended Results on Controllable Sparsity

The extended results compare sparsity mechanisms and architectures, showing that sparsity level largely predicts performance while architecture determines tolerance. They also connect squared activations to transformer spikes and clipping-based stabilization.

  • Controllable sparsity: Percentile Centering and Top-K produce closely matching sparsity and negative-value fractions at equivalent targets, indicating that sparsity level dominates mechanism for performance.The comparison covers accuracy or validation loss, pre-activation negativity, and post-activation sparsity across architectures.
  • Architecture-dependent tolerance: GPT validation loss stays nearly flat from 11% to 77% sparsity, increasing by less than 0.002 nats, while the plain MLP collapses beyond 80% sparsity.Residual paths improve the plain MLP's tolerance, retaining 32.7% accuracy at 90% sparsity.
  • Architecture-dependent tolerance: ResNet retains 94.3% accuracy at 75% per-activation sparsity and 90.4% at 90%, whereas 95% structured sparsity reduces accuracy to 74.7%.ViT accuracy varies by less than 2% across evaluated sparsity levels under both sparsification methods.
  • Top-K sparsity: At K = 0.10, gradient signal collapses and weight means remain flat, whereas increasing K makes gradient bias and weight drift more pronounced.The Top-K experiment uses four retention levels and averages trajectories across 20 random-data runs.
  • Generative models: GELU with Top-K sparsity preserves sharper structural details and textural realism than baseline ReLU in DiT-S/2 image generation.All configurations reach plausible overall composition, with differences emerging mainly in high-frequency detail.
  • Activation spikes: Activation spikes originate in the MLP block: down-projections receive one-sided inputs with maxima around 39–42 and minima near 0, despite stable weight standard deviations.The evidence rules out weight instability as the source of spikes in the examined layers.
  • Squared activations in ViT: Squared activations improve accuracy over base activations in MaxViT, while clipping stabilizes training and further improves performance.The reported comparison is aggregated over accuracy and sparsity metrics.

H. Extended Limitations and Discussion

The paper’s extended discussion identifies theoretical assumptions, empirical scope limits, and architecture-dependent behavior that constrain how broadly its findings should be interpreted.

  • Theoretical assumptions: The formal theorems rely on zero-mean i.i.d. weights, independence between effective downstream weights and pre-activations, and cross-entropy softmax linearization near f = 0.Training progressively violates these conditions, although the authors argue the early drift phase occurs while they are best satisfied.
  • Theoretical assumptions: The formal proof covers ReLU only because smooth activations lack its binary survival-conditioning decomposition, despite empirical drift patterns across several alternatives.The reported empirical pattern includes GELU, SiLU, NoisyReLU, and SUGARBSiLU, with one model excepted.
  • Empirical scope: The ∼70% sparsity cliff and clipped squared-activation benefits remain untested at frontier-scale language-model sizes and longer, larger training regimes.The experiments use CIFAR-10, ImageNet-1K, and FineWeb with GPT-nano, while the cliff is fitted from N=79 configurations.
  • Empirical scope: Clipping thresholds were tuned empirically on GPT-nano, and ViT results suggest fixed numerical thresholds may not transfer directly across architectures.Adaptive or layer-wise clipping schedules were identified as unexplored extensions.
  • Efficiency and transfer: Throughput gains from Accumulation Stop are conservative lower bounds, with evaluation focused on fixed-size-input architectures and limited applicability to autoregressive models.The reported comparison also uses a naive percentile-centering implementation against optimized normalization kernels.
  • Efficiency and transfer: Attention-based architectures tolerate substantially more sparsity than MLPs, but the paper does not fully explain the mechanisms behind this robustness.GPT-nano validation loss remains nearly flat up to s ≈0.91, while skip connections improve but do not fully explain the difference.
  • Architecture-specific findings: On DiT-S/2, GELU with 50th-percentile Percentile LayerNorm improves FID to 48.21 and IS to 31.41 versus the centered LayerNorm baseline’s 49.40 and 29.85.The authors do not characterize this effect mechanistically.
Loading 2605.17659v2…