PaperlayerView exact revision on arXiv

Source-linked AI summary

Transformer MLP Gate Thresholds Are Couplings to a Carried Reference Direction

Olli Tuomi

arXiv:2609.31956v1cs.CLcs.LG

TL;DR

The paper asks what computational role the commonly mean-centered corpus-mean residual-stream direction plays. It tests whether that direction sets the MLP gate population’s operating point, finding that it carries resting inhibition and that manipulating its projection changes gate firing and predictions.

  • Problem

    Shared residual-stream components are known to matter for quantisation and attention, but their computational role is less settled.

    Source: §1 · p. 1

  • Method

    The paper decomposes resting gate inhibition and manipulates the stream’s projection on the corpus-mean direction, using norm-matched controls and random-initialised twins.

    Source: §1 · p. 2

  • Results

    Removing the reference releases gate firing dose-monotonically far beyond norm-matched controls, and an exact decomposition identifies w · bL rather than the bias parameter as the source of resting inhibition.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The corpus-mean component is the reference against which the gate population’s default-off state is defined, so mean-centering sets aside a term relevant to why a unit is off.

    Source: §8 · p. 12

    The causal results concern only the rank-1 corpus mean, whose hidden token-class structure accounts for 2–4% of variance around it; the entropy response rests mainly on release because amplification is shallow in Phi-2 and varies across families.

    Source: §10 · p. 14

Abstract

from arXiv · show

The corpus-mean direction of a transformer's residual stream is a component shared across all inputs, and is commonly removed by mean-centering before representational analysis. We present evidence that it is a functional component: the reference against which the MLP gate population sets its operating point. In Phi-2, an exact decomposition of resting gate pre-activations shows that at mid-stack layers the resting inhibition of >99.9% of gates is carried by the coupling $w \cdot b_L$ to the carried mean direction, at 48-56$\times$ the explicit bias parameter, and the coupling is direction-specific: a random direction at matched norm orders the population's firing rates at Spearman $\rho \leq 0.09$ where the reference reaches 0.95. (That 0.95 is near-tautological on its own; the paper derives its null.) Causally, removing the stream's projection on the reference multiplies above-threshold firing by about 9$\times$, dose-monotonically, at 23-59$\times$ a norm-matched control; a random-initialised twin is flat, and replacement tests show that the direction carries the function and the magnitude does not. A direction-matched control makes the same point: noise injected along the reference costs 8-42$\times$ the same energy along a random direction. The decomposition replicates on four further families spanning both gate types (GELU with an explicit gate bias, bias-free SwiGLU), and the dose-response on two of them. Across an eight-model scan the mechanism is present in every GELU and SiLU family and absent only in OPT, where an opposing LayerNorm bias cancels the carried reference. Gate thresholds are implemented as couplings to a constant the network builds for the distribution it is reading, with the bias parameters contributing little; how much of that constant is carried in the stream and how much in parameters depends on the architecture.

1 Introduction

The paper asks what shared residual-stream components do computationally and tests whether the corpus-mean direction sets the MLP gate population’s operating point. Its evidence links gate firing to that reference, identifies how the function is implemented, and examines its persistence and development.

1 Introduction

  • On Phi-2, the corpus-mean direction is constant across text for a given input distribution, remains a fixed fraction of the growing stream, and rotates slowly with depth.It forms a band of related directions.Source: §1 · p. 1
  • Removing or amplifying the stream’s projection on the corpus-mean direction changes gate firing dose-monotonically, far beyond norm-matched controls, while a random-initialised twin is flat.The function is carried by the direction, not its magnitude.Source: §1 · p. 2
  • An exact decomposition identifies the coupling w · bL, rather than the bias parameter, as the source of resting gate inhibition.The decomposition is exact, not a fit.Source: §1 · p. 2
  • The direction is continuously re-supplied by aligned per-layer mean writes, and co-removing those writes with the direction is near-catastrophic for prediction.The direction re-accumulates after local removal.Source: §1 · p. 2
  • The operating point forms early in training, during the induction-transition window, and appears even with attention disabled in one 160M seed.A finer 160M checkpoint grid shows it forms first; the authors conclude that the token-local prediction task alone is enough to build it.Source: §1 · p. 2
  • The static decomposition and causal dose-response are tested across further model families, while the remaining causal tests are Phi-2-only.The broader scan includes GELU, ReLU, and SiLU models.Source: §1 · p. 2

2 The object

The corpus-mean direction is a stable, distribution-specific component of the layer-normalised MLP input, maintained as the residual stream grows. LayerNorm and aggregation account for its stability, while aligned per-layer writes construct it and its direction drifts gradually with depth.

2 The object

  • The corpus-mean direction remains stable for a given input distribution: split-half cosine is 0.94–0.99, while prose-to-code cosine is 0.76–0.84 and prose-to-repeated-token cosine is 0.02–0.05.Token-class means add finer structure, but account for only 2–4% of variance around the corpus mean.Source: §2 · p. 3
  • Across L2–L28, the mean’s share of the stream stays at 0.33–0.44 as stream norm grows 5.1×, while LayerNorm keeps the gate-read bL norm at 6.6–10.1.The resting term remains within a band of about 0.7; the random-initialised twin does not stabilise the share.Source: §2 · p. 3
  • LayerNorm’s per-position gain control and aggregation suppress variation in the reference component, with the summed sources steadier than their individual contributions.The decomposition attributes 85%/77%/66% variance reduction to gain control at L8/L16/L24; covariance adds 34–72% of total variance.Source: §2 · p. 3
  • The reference rotates gradually with depth: directions four layers apart have cosine ≈0.97, whereas cos(ˆb6,ˆb22) is 0.81 and cos(ˆb6,ˆb30) is 0.24.The cross-layer geometry is specific to Phi-2 and varies across replication families.Source: §2 · pp. 3–4
  • Every layer’s attention and MLP mean writes are aligned with the deep mean, and their small contributions sum coherently to construct the reference.The largest and steadiest single contributor is the layer-0 block, supplying 28% of the reference component at L8.Source: §2 · p. 4

3 Function: the gate population thresholds against it

The paper tests whether the corpus-mean direction functions as the reference for gate firing by manipulating its projection and comparing with norm-matched controls. Removing it releases firing and changes predictions, with effects that are direction-specific and sensitive to context and intervention scope.

3 Function: the gate population thresholds against it

  • The intervention scales the stream’s projection on ˆmL at L6, L10, or L14, with α=1 as an exact no-op and a same-magnitude random-direction control.Firing is measured as the fraction of post-GELU activations above 0.5; output damage is final-position next-token KL.Source: §3 · p. 4
  • Single-site reference removal costs KL = 1.43/0.52/0.27 at L6/L10/L14, against 0.020–0.025 for the norm-matched control.The twin loses only 0.32–0.38 KL under the same removal despite a raw mean 8–10× larger in norm.Source: §3 · p. 4
  • At L10, removing the reference raises above-threshold firing about 9× from the clean baseline of 0.0105; firing changes monotonically across the dose-response and is 23–59× the norm-matched control.Table 1 reports the dose-response; the random-initialised twin is flat.Source: §3 · p. 4, Table 1 · p. 5
  • Reference-axis noise costs 8–42× the same energy along a matched random direction, and removing the reference costs 127×/52× matched-energy isotropic noise at L6/L10.Enforcing a near-zero empirical mean on injected noise leaves the direction-axis cost ratio unchanged.Source: §3 · p. 5
  • At L16, removal costs ∆NLL = +0.493 and ∆H = +0.379 while top-1 stays 0.70, with loss concentrated at high-entropy positions (+0.94 versus +0.04 at low-entropy positions).Replacing massive-activation coordinates with corpus constants is milder: ∆NLL = +0.086 and top-1 0.87.Source: §3 · p. 5
  • A real repeated-token input divides out the reference and raises duty from 0.087 to 0.791 at L24, but the effect requires extreme degeneracy and recovers within two or three clean-context tokens.Tiling four tokens or an i.i.d. draw from four token types is enough to avoid the extreme-denominator regime; natural text is nowhere near it.Source: §3 · pp. 5–6
  • Removing the reference in an eight-position window at L6 still alters the next-token distribution 32 tokens later, at KL = 0.098 and 54× the norm-matched control.Greedy continuations diverge within a median of two steps, though the true-token log-probability change is inconsistent in sign.Source: §3 · p. 6
  • The manipulation reproduces monotone firing changes and control separation in Pythia-1.4B and Qwen2.5-1.5B; removal costs final-position KL 3.60 and 0.59, respectively.Qwen’s SwiGLU gate has no bias parameter, so the carried reference is its entire resting inhibition.Source: §3 · p. 6

4 Identity: the direction, not the magnitude

Replacement tests show that preserving baseline firing depends on transferring the stream projection to a direction aligned with the reference, not merely preserving its magnitude. Random-direction replacements release firing much like removal, while a near-aligned layer reference preserves baseline firing.

4 Identity: the direction, not the magnitude

  • Replacement preserves projection magnitude, sign pattern, and timing while changing direction and approximately preserving total norm.The test addresses both the possibility that gates need a full stream and the possibility that norm shrinkage boosts orthogonal components through LayerNorm.Source: §4 · p. 6
  • A replacement direction aligned with the reference preserves baseline firing, with preservation tracking cosine to the reference monotonically.At cosine 0.94, another layer’s mean direction yields 31× less release and 130× less KL than removal.Source: §4 · p. 6, Table 2 · p. 7
  • Random-direction replacement releases firing as if the projection had been removed, even with the pre-norm magnitude unchanged.This excludes the LayerNorm gain-boost alternative; layer 18’s direction under the same norm preserves baseline firing.Source: §4 · p. 7

5 Implementation: the threshold is the coupling

An exact decomposition attributes most resting gate inhibition to the coupling between each read row and the carried mean direction, rather than to explicit gate bias. Direction controls, cross-model replications, and a counterexample in OPT qualify how this operating point is implemented.

5 Implementation: the threshold is the coupling

  • At mid-stack Phi-2 layers, w · bL carries resting inhibition for >99.9% of gates at 48–56× the explicit bias magnitude.The exact decomposition is ⟨w · xln + bn⟩ = w · bL + bn.Source: §5 · p. 7
  • The raw ρ≈0.95 resting-term–duty association is near-tautological, while a matched-norm random direction orders duty at ρ=0.058–0.085 versus 0.946–0.964 for the reference.The paper treats the correlation as descriptive and identifies direction specificity through the random-direction comparison.Source: §5 · p. 8
  • The reference coupling is a small read-row slice: |cn|/∥w∥ has median 0.07–0.12 and accounts for 0.5–1.4% of read-weight energy.The coupling alone predicts duty ordering as well as the full resting inhibition; bn adds nothing once the coupling is known.Source: §5 · p. 8
  • The reference coupling carries resting inhibition for ≥99.6% of gates at every checkpoint after initialisation, when bn is exactly zero; there is no phase in which explicit bias leads.The coupling builds through the step 512–1000 window (0.16→0.38).Source: §5 · p. 9
  • Across four further architectures spanning biased GELU and bias-free SwiGLU gates, resting inhibition is negative and coupling-dominant; where bias exists, the coupling is 15–43× the bias parameter.Qwen’s shallowest sampled layer is the exception to the otherwise ≥0.91 duty rank correlation; random-init twins remain floored.Source: §5 · p. 9, Table 4 · p. 9
  • The mechanism appears in all six non-OPT models, while OPT’s opposing LayerNorm-bias excitation nearly cancels its residual-carried reference at the operating point.Across architectures, the input-independent share of the reference component ranges from 0% in three bias-free RMSNorm models to 52–84% in Pythia-410M.Source: §5 · pp. 9–10

6 How the reference persists, and the cost of losing it

Attention sustains the carried reference, which re-accumulates after local removal. Removing it across the middle layers releases firing throughout the network and substantially harms prediction.

6 How the reference persists, and the cost of losing it

  • At L10, attention’s ˆmL channel has a constant share of 0.452 versus 0.014 for its orthogonal channel, a 33× dissociation.The MLP shows no such dissociation (0.003 vs. 0.003); its time-averaged write is reference-aligned, while attention sustains the reference during operation.Source: §6 · p. 10
  • After L6 removal, cos(resid, ˆmL) falls by 0.16 at L10 and L14, then recovers to baseline by L26–L30.The stream re-accumulates its mean over ∼12–16 layers, and deeper removal sites restore more slowly.Source: §6 · p. 10
  • KL = 4.38 and ∆NLL = +4.75 nats after co-removal at all measured layers ≥6, about 8.4× the L10 single-site removal’s KL of 0.52.Late-only co-removal (≥14) is milder (KL 0.55); the twin loses 0.65 KL with flat firing.Source: §6 · p. 10

7 Development

The gate operating point forms early in training, through coupling to the carried reference. It appears before the induction transition in a finer-grid model and also forms when attention is disabled.

7 Development

  • In Pythia-410M, the reference coupling forms during the step 512→1000 induction-transition window, but the checkpoints cannot order the events within it.Across that interval, induction-head score rises 0.03 →0.88, median duty cycle falls 0.42 →0.28, and reference share deepens −0.06 →−0.34.Source: §7 · p. 11
  • By step ≈800, the 160M model’s median mid-stack duty falls from 0.50 to 0.075, while its induction-head score remains near its floor.The resting depth z reaches ≈1.5 in the same window, and the mean-input term carries ≈0.99 of resting pre-activation; the induction score rises later, over steps ≈3000–12000.Source: §7 · p. 11
  • With attention disabled, the 160M model still forms the operating point: mid-stack duty settles at 0.077 versus 0.067 with attention enabled.The reference coupling is unchanged, carrying ≈0.99 of resting pre-activation in both models; the attention-free model has one seed.Source: §7 · p. 11
  • Across the three developmental settings, the operating point is an early, token-local component of a trained MLP.The authors conclude that the token-local next-token prediction task alone can build it, while noting each setting uses one seed.Source: §7 · pp. 11–12

8 Discussion

The carried mean is the reference that defines the gates’ default-off state, so mean-centering removes a term relevant to explaining gate behavior. The discussion also connects the coupling to autoencoder features and gated linear units.

8 Discussion

  • Removing the mean-direction reference costs KL 0.27–1.43 at a single Phi-2 site and 4.38 when removed at every measured layer from L6 onward.Mean-centering leaves computation untouched, but an account of why a unit is off needs the term that sets the gates’ resting point.Source: §8 · p. 12
  • The same mean-alignment effect shifts sparse-autoencoder encoder pre-activations, while bias-free RMSNorm families assemble the reference from computed writes.The authors suggest bias parameters contribute little to the gate operating point outside OPT, where the reference does not reside in the bias parameter.Source: §8 · p. 12
  • In GLU MLPs, the coupling sits in the gate branch and places units on a continuum from thresholded switches to bilinear multipliers.This analysis is reported separately.Source: §8 · p. 12

9 Related work

The discussion distinguishes the studied mean direction from massive activations, attention sinks, and outlier dimensions by its geometry, site, and role. It also frames the carried mean as a constant that can supply a bias-like function.

9 Related work

  • Outlier dimensions track token frequency and can dominate cosine similarity, while last-layer outlier dimensions can add a constant positive logit contribution toward frequent tokens.The paper situates its object in this broader family while distinguishing its direction, MLP gate-input site, and exact decomposition.Source: §9 · pp. 12–13
  • Unlike massive activations and sinks, the studied object is a direction read at mid-stack MLP gate inputs and assembled by distributed aligned writes.The authors distinguish it by object and site, while noting that reference removal costs most at function-word and high-entropy positions.Source: §9 · pp. 12–13
  • In Phi-2, the mid-stack reference is geometrically distinct from first-token, residual-sink, and massive-coordinate directions, with cosines of 0.04–0.17 and ≤0.27, respectively.The reference carries 10–23% of squared norm on the twenty largest last-layer outlier coordinates, a minority overlap; top-20 Jaccard is 0.05–0.14.Source: §9 · p. 13
  • Under covariance-weighted cosine, bL aligns with the sink, first-token, and massive directions at −0.23 to +0.01, versus a random-pair null of 0.03 median and 0.11 at the 95th percentile.The positive control, covariance-weighted cos(bL, mL), is 0.91–0.95; in the twin, where bL is the sink direction, every metric returns 0.83–0.99.Source: §9 · p. 13
  • The trained network spontaneously supplies a constant through its carried mean, implementing the homogeneous-coordinates construction rather than relying on a dedicated input.On the attention side, Sun et al. (2024) report GPT-2 models with explicit learnable bias parameters do not form massive activations; the authors identify OPT as the parameter-based strategy on the gate side.Source: §9 · pp. 13–14

10 Limitations

The paper’s claims have distinct scope boundaries: several causal and geometric tests are Phi-2-only, while development evidence is trend-level and single-seed. Other limits concern the operating-point statistics, rank-1 treatment, response asymmetry, manipulation design, and proxy-based geometric comparisons.

10 Limitations

  • The static decomposition replicates across an eight-model scan, but the causal dose-response and entropy response extend only to Pythia-1.4B and Qwen2.5-1.5B; replacement and co-removal remain Phi-2-only.Development findings are trend-level and single-seed; the depth profile, context-resolved removal, finer-grid study, and attention-disabled study are also limited to single models or modest samples.Source: §10 · p. 14
  • Cross-layer geometry varies across families: the rotating-band account is Phi-2-only, and the single-carried-direction geometry does not replicate, despite broader replication of static coupling and dose-response.The reported mid-stack cosine ranges from 0.95 in GPT-2-medium and 0.96 in Qwen2.5-1.5B to 0.74 in TinyLlama-1.1B and 0.41 in Pythia-1.4B, below its twin floor.Source: §10 · p. 14
  • The resting-term–duty correlation is near-tautological, and duty ordering is largely norm-scaled; the reference’s residual ranking contribution remains undecomposed.The read norm alone yields ρ = −0.89; functional claims instead rely on causal tests.Source: §10 · p. 14
  • Every causal result concerns only the rank-1 corpus mean; token-class structure accounts for 2–4% of variance around it.The authors describe rank-1 treatment as a good approximation for operating-point claims.Source: §10 · p. 14
  • The entropy response to weakening the reference rests mainly on release, because amplification is shallow in Phi-2 and varies across families.Source: §10 · p. 14
  • The corpus-mean reference is not position-resolved, manipulations leave positions < 16 untouched, and α-scaling changes the LayerNorm input distribution.A norm-preserving replacement excludes a pure gain-boost account only at L10; function-word replacement is one naive content-direction control.Source: §10 · pp. 14–15
  • Geometric comparisons use activation-based proxies on one model rather than each prior work’s exact construction; the variance-aware recomputation is Phi-2-only.The raw-residual-covariance metric cannot discriminate among proxies, and the whitening variant fails its positive control at this sample size.Source: §10 · p. 15

11 Reproducibility

The paper lists scripts and JSON artifacts for its experiments, identifies the analyses each supports, and provides repository and archive locations.

11 Reproducibility

  • Scripts and JSON artifacts are listed by analysis, including background reference, identity, mechanism, development, geometry, and cross-model tests.The inventory also names scripts for operating-point measurements, controls, noise tests, and bias-migration analyses.Source: §11 · pp. 15–16
  • Code and data are available in the background GitHub repository and archived on Zenodo.Source: §11 · p. 16

Related papers

Loading 2609.31956v1…