PaperlayerView exact revision on arXiv

Source-linked AI summary

Off-Axis, On Purpose: Where a Transformer Computes Concepts and Why it Does So

Mark Oskin

arXiv:2608.10251v1cs.CLcs.LG

TL;DR

The paper investigates why transformer concepts are computed off the unembedding read-out axis and how answers eventually reach that axis. It measures the geometry, tests its functional role, and compares loss-based and frame-based ways to impose it. The results show that off-axis computation protects cross-token composition, answers are added late, and a prescribed frame can control the geometry without changing benchmark quality.

  • Problem

    The paper asks why intermediate transformer content lies off the vocabulary read-out axis and how the answer reaches that axis.

    Source: §1 · p. 2

  • Method

    The paper measures depth-wise write angles, cross-token insulation, rotational versus additive changes, workspace dimensionality, and convergence under loss-based or fixed-frame interventions.

    Source: §1 · p. 2, §5 · p. 16, §5.1 · p. 17

  • Results

    The off-axis phase protects cross-token composition, the answer is written late and additively, and prescribed frames achieve baseline quality while controlling the geometry.

    Source: §11 · p. 33

  • Takeaways & Limitations

    The off-axis frame is functionally useful and sufficiently free that researchers can prescribe it without sacrificing downstream quality.

    Source: §11 · p. 33

    The evidence comes from one architecture family, scale, and corpus, with only eight- and nine-seed convergence flocks; larger-scale generality remains open.

    Source: §9.4 · p. 30

Abstract

from arXiv · show

A transformer's answer lives on one axis: the direction its unembedding reads. Its intermediate states largely do not, and that off-axis position is usually treated as an obstacle to interpretation. We show it is functional. A 12-layer model computes in two phases. Through the first, every sublayer writes into a subspace held near-orthogonal to the read-out, attention 75 to 96 degrees off it at every depth. Moving attention's values onto the read-out is 64 to 84 times more damaging than a matched random rotation, and the damage is entirely in cross-token mixing: the subspace insulates composition from the vocabulary. Beneath it the frame itself turns rigidly with depth. In the second phase the answer arrives on-axis, late, and by addition rather than by turning accumulated content onto the read-out. Pressing every layer onto the read-out instead, as training for early exit does, matches the baseline on perplexity, LAMBADA and BLiMP while cutting the concept-phase workspace from about twenty-five effective dimensions to fourteen, a change none of those benchmarks register. The geometry can also be imposed, though not by asking for it. Prescribing it through the loss is a lottery: six of eight seeds collapse, because a model told to null its read-out projection obeys most cheaply by discarding dimensions. Inserting one fixed rotation at the phase boundary lands it instead, at baseline quality. A sparse rotation the surrounding weights can absorb converges on all nine seeds, against five of nine for ordinary training. Which rotation is immaterial: twenty-five runs across thirteen distinct ones reach the same quality, and two baselines from different seeds hold their concepts in near-orthogonal frames while agreeing on their read-outs. That freedom is usable: a basis drawn at random and prescribed before training is adopted across the concept phase, with quality unchanged.

1 Introduction

The paper asks why transformer representations remain off the read-out axis and how answers reach that axis. It argues that off-axis computation insulates cross-token composition, while the final answer is added late, and shows that the geometry can be imposed through a prescribed frame.

Functional role of the off-axis phase

  • 64–84× more damage than a matched random rotation occurs when attention values are moved onto the read-out, entirely through cross-token mixing.The off-axis subspace lets attention average values without blurring vocabulary predictions.Source: §1 · p. 2

How the answer arrives

  • 64% of the committed state is orthogonal to any rotated earlier content, while the residual norm grows 1.74×, showing that the answer is added rather than turned.A best-fit rotation captures only R2_rot = 0.38 of the change.Source: §1 · p. 2
  • The final prediction first becomes and remains the top decoded token at layer 9.2 on average, with 39% of positions reaching that status only at the last layer.The model therefore separates an off-axis concept phase from a late on-axis writing phase.Source: §1 · p. 2

Imposing the phase structure

  • Loss-only geometric prescribing is unreliable: six of eight seeds collapse, whereas a fixed boundary rotation reaches baseline quality and raises convergence from two of eight to seven of eight.The loss-only model can satisfy null read-out projections by discarding dimensions rather than building a stable frame.Source: §1 · p. 2

Frame freedom and control

  • A sparse absorbable rotation converges on all nine seeds against five of nine for ordinary training, while twenty-five runs across thirteen rotations reach the same quality.Different seeds can hold concepts in near-orthogonal frames, and a randomly prescribed basis is adopted without changing downstream quality.Source: §1 · pp. 2–3

Cost of on-axis computation

  • Forcing every layer onto the read-out preserves perplexity, LAMBADA and BLiMP within baseline ranges but shrinks the concept workspace from about twenty-five to fourteen effective dimensions.The benchmark parity does not reveal the geometric cost.Source: §1 · p. 3

2 Experimental setup

The study evaluates decoder-only 12-layer GPT-2-small transformers trained on OpenWebText, using matched geometric nulls and benchmark measurements. It reports both converged-run performance and convergence failures, while measuring effective dimensionality, rotations, and write angles across depth.

2 Experimental setup

  • The experiments use 12-layer GPT-2-small transformers with 768-dimensional residuals, 12 attention heads, 3072-wide feed-forwards, and 125,143,296 parameters.Models use pre-layer normalization, learned positional embeddings, tied input/output embeddings, and a 50,257-token GPT-2 byte-pair vocabulary.Source: §2 · p. 3
  • Models train for one epoch on approximately 8.9 billion OpenWebText tokens with AdamW, a 3 × 10^-4 peak learning rate, 2000 warmup steps, cosine decay, and bf16 precision.Runs use batch size 16 at 2048-token context length and differ only by training seed unless otherwise stated.Source: §2 · pp. 3–4
  • Perplexity, zero-shot LAMBADA accuracy, and BLiMP are evaluated with uniform right-padding and attention masking across models.The harness excludes padded positions to avoid changing model rankings through batching artifacts.Source: §2 · p. 4
  • Averages include only converged runs, report their ranges, and screen final residual participation ratio alongside perplexity to exclude collapsed representations.Convergence rate is reported separately rather than repaired or hidden.Source: §2 · p. 4
  • From-scratch instability is part of the measured setting: a matched GPT-2 124M study diverges on about one run in ten, reported as a lower bound.The paper does not manually repair failures, so convergence rates remain visible in the results.Source: §2 · p. 4

2.1 Geometric measurements

  • The study uses matched nulls: equal-magnitude random rotations for insulation, token-stream-only fits for rigidity, and foreign bases for frame prescriptions.These controls are designed to isolate the relevant geometric comparison rather than use generic references.Source: §2 · p. 4
  • Participation ratio estimates effective dimensionality, orthogonal Procrustes separates rotation from new content, and write angles quantify alignment with final logits.Participation ratio equals d for isotropic activity and 1 for rank-one activity; the Procrustes control reaches R2_rot = 1.00 for an explicit rotation.Source: §2.1 · p. 5

3 The GELU baseline

Across five converged baselines, the transformer computes concepts in an off-axis workspace, then writes the answer onto the read-out late by addition. The workspace insulates cross-token mixing, its frame rotates rigidly with depth, and usable convergence occurs in only five of nine seeds.

3.1 Concepts are computed off the read-out

  • Attention writes remain 75–96° off the final logits at every layer, while feed-forward writes stay near 85° through the network body and swing toward the read-out in the last four layers.The feed-forward write reaches about 55° at the output, while residual participation expands sharply toward the output, exceeding seventy at the final layer.Source: §3.1 · pp. 6–7
  • 64× and 84× larger next-token-loss damage from read-out-directed value rotations at 10◦ and 20◦ disappears to about 1× without cross-token averaging, showing insulation is functional.At 40◦ the ratio falls to 16× because large rotations degrade the model in either direction, so the strongest comparison concerns small perturbations.Source: §3.1 · p. 7, Figure 3 · p. 8

3.2 The answer is written to the read-out, rather than turned from the concept space

  • Across the commit span, rotation explains only R2_rot = 0.38 of the change, 64% of committed content is orthogonal to any input rotation, and the residual norm grows 1.74×.The answer therefore arrives as a separate on-axis write rather than by turning the accumulated off-axis representation; prediction surfaces at mean layer 9.2, with 39% resolving only at the final layer.Source: §3.2 · pp. 8–9, Figure 5 · p. 10
  • The final answer is formed by coherent superposition: last feed-forward layers provide almost all answer-logit contribution, the final layer about a third, while attention and embeddings contribute essentially nothing.Answer-aligned components accumulate, whereas orthogonal other-token residue cancels to 0.18 times the random-walk level at the final write.Source: §3.2 · p. 9, Figure 6 · p. 11, §3.3 · p. 13

3.3 Why the frame rotates

  • The frame rotates about 29◦ per layer within the off-axis subspace, with a token-stream-only rotation achieving R2 = 0.93 versus 0.76 for leaving the constellation unchanged.The turn is the orthogonal factor of the layer’s linear response; its antisymmetric energy share, 0.675–0.687, is close to 0.7064 for a random matrix.Source: §3.3 · pp. 11–12, Table 2 · p. 12

3.4 A usable baseline is roughly a coin flip

  • Five of nine seeds converged, with usable runs showing moderate-to-expanding dimensionality and no anti-read-out writes, while failures collapsed, inflated, or turned obtuse.The late feed-forward swing toward the read-out survived failures, so the distinguishing signature was the intact workspace geometry rather than a missing commit.Source: §3.4 · p. 13, Figure 7 · p. 13, Table 3 · p. 13

4 Forcing GELU on-axis

Forcing every layer toward the read-out tests whether the transformer's default off-axis computation is necessary. The model can approach the axis, but quality and ordinary computation degrade as it does so.

4.1 Pressure toward the read-out

  • A distribution-matching penalty pushes layers on-axis, but its coefficient controls a monotonic quality loss and never reaches the target angle.The penalty uses a stop-gradient final prediction and averages divergences between intermediate and final decoded distributions.Source: §4.1 · p. 14, Figure 8 · p. 14

4.2 Closer to the axis, at a rising cost

  • LAMBADA falls from 0.271 to 0.142 and perplexity rises from 19 to 28 as pressure draws computation toward the read-out.The residual angle saturates near 13°, while the baseline remains the best point on the sweep.Source: §4.2 · p. 14, Figure 9 · p. 15

4.3 The answer survives as a residue of cancellation

  • Angle-constrained training reaches near-baseline quality at the same ≈13° floor, with six usable seeds averaging LAMBADA 0.258 and perplexity 19.5.Fifteen seeds were trained, eight reached the end, and six produced usable models; the distributional penalty at the same coefficient averaged 0.191 and 22.0.Source: §4.3 · p. 15
  • The apparent success is a cancellation: feed-forward writes +9103 toward the answer while attention writes −8113 against it, leaving a sane residual prediction.Across six seeds, attention contributes about −0.9 times the feed-forward answer mass, and the opposing contributions inflate by 36× relative to baseline.Source: §4.3 · p. 15, Figure 10 · p. 16
  • Attention cancels its own on-axis blur because cross-token mixing would smear predictions, so forced on-axis models reach answers through contorted rather than normal computation.The residual workspace remains well above rank collapse in working models, making the result a caution rather than a broken-model artifact.Source: §4.3 · pp. 15–16
  • Forcing every layer onto one axis is therefore achievable, but the paper treats it as a caution and next tests a structured off-axis phase followed by an on-axis phase.The imposed geometry preserves quality only by changing how the sublayers compute and cancel their contributions.Source: §4.3 · p. 16

5 Prescribing the phase structure

The frameless step imposed the desired off-axis-to-on-axis phase structure through the loss, but it usually failed because the loss specified angles without supplying a computational frame.

5.1 The step schedule

  • The angle schedule requested 90° through layers 0–5 and 0° through layers 6–11, with no architectural rotation inserted.The phase structure therefore had to be built entirely by the weights under the loss.Source: §5.1 · pp. 16–17, Figure 11 · p. 17

5.2 A lottery

  • Six of eight frameless-step seeds diverged, while only two completed; the prescription reached the end of training about a quarter of the time.Failures had instantaneous perplexities of roughly 110–3000, whereas completions were near perplexity 20–33.Source: §5.2 · pp. 17, 19
  • Both completed seeds matched the prescribed geometry, yet one reached baseline quality while the other collapsed to LAMBADA 0.161 and perplexity 33.0.The survivor achieved LAMBADA 0.237 and perplexity 20.6; geometry alone did not distinguish the runs.Source: §5.2 · p. 17, Figure 12 · p. 18
  • The collapsed run’s residual norm grew to about 114× the baseline’s before the boundary, showing that the strain was spent during the concept phase.The survivor absorbed a first-layer blast and then tracked the baseline, whereas the collapsed run peaked near 580 before recovering.Source: §5.2 · pp. 17–18, Figure 12 · p. 18
  • The collapsed run’s concept-phase workspace stayed near rank 2, while the survivor expanded its workspace like the baseline.The angle loss specified where representations should point but did not preserve room to compute.Source: §5.2 · p. 18, Figure 13 · p. 19
  • The frameless step therefore reproduced the phase geometry only by luck: it named a direction but did not provide the stable off-axis frame needed for computation.The next section addresses this failure by supplying the frame directly.Source: §5 · p. 16, §5.2 · p. 19

6 Supplying the frame

A single fixed rotation supplied the missing off-axis frame at the phase boundary, making the prescribed geometry reliable while preserving a high-dimensional workspace and additive commitment.

6.1 The device

  • The device applied one fixed orthogonal rotation at layer 6, preserving residual norm and participation ratio while handing the network its boundary frame.The rotation was fixed, input-independent, and not learned; the step schedule remained unchanged.Source: §6.1 · pp. 19–20, Figure 14 · p. 20

6.2 The lottery is gone

  • Seven of eight dense-device seeds converged to baseline-quality models, compared with two of eight frameless-step seeds.The dense device achieved LAMBADA 0.257, perplexity 19.3, and BLiMP 0.798 within the converged baseline range.Source: §6.2 · p. 20, Table 4 · p. 21

6.3 The geometry, with room to compute

  • Across converged device runs, the residual stayed about 90° off-axis through the concept phase and turned onto the read-out at the boundary.Unlike the frameless step, the device reproduced this geometry across the converged set rather than only for lucky seeds.Source: §6.3 · p. 21, Figure 15 · p. 22
  • The supplied frame preserved a richly dimensional concept workspace, with mean participation ratio about 54 and peaks above 100.The frameless collapsed run had rank about 2, while the device maintained full-dimensional off-axis computation.Source: §6.3 · p. 21, Figure 15 · p. 22

6.4 The device supplies the frame

  • The device did not perform the commit: across the boundary, rotation explained only R2_rot ≈0.38, matching the baseline’s 0.36–0.39.The answer remained freshly written additively rather than being rotated onto the read-out by the device.Source: §6.4 · p. 22
  • The concept frame continued turning within the phase, with 28.7° under the device versus 29.0° for the baseline.The boundary rotation supplied the frame but did not remove the layer-wise motion produced by the network’s own responses.Source: §6.4 · p. 23, Figure 16 · p. 23

7 A cheaper frame

A sparse, absorbable rotation achieved the same geometry and quality as the dense device while using a smaller concept-phase workspace and training more reliably.

7.1 The sparse device

  • The sparse device used adjacent coordinate-pair rotations that surrounding weights and layer normalization could absorb nearly for free.The relative residual after fitting a per-channel affine was 0.05 for sparse versus 0.34 for dense; under RMSNorm it fell to 0.002 for sparse.Source: §7.1 · p. 24

7.2 The same value, in a smaller workspace

  • Sparse-device models matched dense-device and baseline quality, with LAMBADA 0.260 versus 0.257 for dense and 0.266 for baseline.Perplexity and BLiMP remained flat across the three regimes.Source: §7.2 · p. 24, Table 5 · p. 25
  • The sparse and dense devices realized the same 90°-then-commit geometry, but sparse models used a concept-phase participation ratio of about 30 versus about 54 for dense models.The sparse workspace was close to the baseline’s participation ratio of 24, producing roughly a twofold difference in effective dimensionality without a quality change.Source: §7.2 · p. 24, Figure 17 · p. 25
  • The frame’s form controlled how widely concepts were spread, but richness beyond the collapse-avoiding floor did not register in quality.The smaller sparse workspace was sufficient for the same model value.Source: §7.2 · p. 24, §7.3 · p. 25

7.3 The consistency edge

  • Every sparse seed reached a usable model, whereas one dense seed diverged late and dense training showed unusually high perplexity spikes.The absorbable frame was therefore gentler and somewhat more reliable to train.Source: §7.3 · p. 25

8 Which 90◦?

The particular 90° frame was functionally free: different devices and baseline seeds used unrelated frames at the same quality, and a randomly chosen frame could be imposed directly.

8.1 Any device, the same model

  • Twenty-five runs across thirteen distinct rotations reached baseline-range quality, with LAMBADA 0.226–0.287 and BLiMP 0.788–0.810.The set included eight dense Haar-random rotations and five sparse signed permutations; no device stood out.Source: §8.1 · p. 26, Figure 18 · p. 28

8.2 The gauge in the wild

  • Two baselines aligned by their read-outs held concept frames about 90° apart, with cross-model angle 89.7° versus 91.4° for random orientation.Their read-outs agreed at R2 = 0.81, while within-model measurement noise was near 20°.Source: §8.2 · p. 26, Figure 18 · p. 28

8.3 Prescribing the exact frame

  • A randomly drawn basis could be prescribed in advance, selecting which 90° frame the model adopted without changing the phase geometry or quality.The prescription targeted the concept-phase second moment while retaining the step schedule and boundary device.Source: §8.3 · pp. 26–27
  • The frame prescription used per-pair normalization, per-layer penalties, and a randomly drawn orthogonal basis to avoid rank-one and canonical-basis degeneracies.These design choices ensured that the loss named a distributed frame rather than an inflated direction or a signed permutation symmetry.Source: §8.3 · p. 27
  • In the prescribed basis, concept-phase off-diagonal energy was 3–28 times lower than in comparison bases at every layer, with no overlap between runs.After removing the top five eigendirections, the prescribed basis remained four times lower, ruling out a single inflated direction.Source: §8.3 · p. 27, Figure 18 · p. 28
  • Seven of eight prescribed-frame seeds converged with downstream quality inside the baseline range: LAMBADA 0.265, BLiMP 0.796, and perplexity 19.5 versus 19.1.The alignment held across layers 0–5 and survived deflation.Source: §8.3 · p. 27

9 Discussion

The discussion argues that supplying a frame improves trainability without changing final quality, while forcing decodability preserves benchmark scores but substantially alters internal geometry. It further treats the concept-phase frame as a freely prescribable gauge, with conclusions bounded by the study’s scale, architecture, corpus, and small seed counts.

9.1 Supplying a frame increases training reliability

  • Under the same step schedule, data, and budget, 2/8 loss-only, 7/8 dense-rotation, and 9/9 sparse-rotation seeds reached usable models, compared with 5/9 baseline seeds.The device adds no capacity; a fixed, untrained, input-independent rotation produced a more reliable training process than the unconstrained baseline.Source: §9.1 · p. 28
  • 25 runs across 13 rotations reached the same model, and different-seed baselines used unrelated concept frames despite agreeing on their read-outs.The frame is therefore a gauge freedom that affects optimization but not the converged function.Source: §9.1 · pp. 28–29

9.2 What asking for decodability costs

  • Forcing every layer onto the read-out preserved perplexity, LAMBADA, and BLiMP within baseline range while shrinking the concept workspace from about 25 to 14 effective dimensions.The representation changed substantially even though these benchmarks did not detect the change.Source: §9.2 · p. 29

9.3 Placing a representation rather than finding one

  • The paper proposes prescribing representations rather than only recovering them: models adopted a named basis across the concept phase with unchanged downstream quality.A shared prescribed basis could improve internal comparability without requiring post hoc alignment.Source: §9.3 · p. 29

9.4 Limits

  • The evidence is limited to one architecture family, scale, and corpus, uses strong interventions and small seed flocks, and treats the insulation mechanism as an inference beyond its causal measurement.Whether the same lottery and fix hold at larger scale remains open.Source: §9.4 · p. 30

10 Related Work

Related work provides tools and neighboring accounts of intermediate representations, rotations, early exit, and gauge freedom. This paper distinguishes itself by measuring geometry against the fixed unembedding, explaining off-axis computation functionally, and prescribing the frame during training.

Reading intermediate states in vocabulary space

  • The tuned lens reads latent predictions more faithfully than the logit lens, but neither establishes whether the model reaches its answer by rotation or fresh addition.The paper uses the tuned lens as an instrument rather than a competing explanation.Source: §10 · p. 30

The residual-stream gauge and prescribed geometry

  • The paper uses residual-stream gauge freedom differently from prior prescribed-geometry methods: a fixed, read-out-oriented rotation is installed before training to make the phase structure legible.Unlike learned rotations or post-training frames, the intervention is fixed in advance and oriented toward interpretability.Source: §10 · pp. 31–32

Workspaces and off-readout structure

  • Prior workspace, bottleneck, and dictionary methods find structured off-readout representations post hoc, whereas this paper explains their insulation function and supplies the workspace during training.The distinct claim concerns functional cross-token composition rather than merely the existence of structure.Source: §10 · pp. 31–32

The layer-to-layer rotation

  • Prior geometric analyses measure layer rotations or Jacobians, but this paper measures rotation relative to the fixed read-out and connects best-linear-predictor departures to the concept-phase frame.Layer-relative similarity cannot express angle to an external unembedding axis.Source: §10 · pp. 31–32

Deep supervision, early exit, and readout-alignment pressure

  • Early-exit and deep-supervision methods push intermediate states toward a shared read-out, while this paper adds that parity can conceal a workspace reduction from about 25 to 14 effective dimensions.The representation-level effect complements their quality evaluations.Source: §10 · pp. 30–33

11 Conclusion

The paper argues that off-axis computation protects cross-token composition, while the final answer is written late and additively on-axis. It shows that prescribed frames preserve quality and can improve trainability, whereas decodability pressure changes geometry without benchmark-visible losses.

11 Conclusion

  • The off-axis subspace insulates attention’s cross-token mixing from vocabulary predictions, while the answer is written fresh, late, and additively.Moving attention values onto the read-out is 64–84× more damaging than a matched random rotation.Source: §11 · p. 33
  • A fixed phase-boundary rotation preserves baseline quality and raises convergence from 5/9 ordinary runs to 9/9 sparse-frame runs.The intervention specifies the geometry in advance without adding capacity or raw quality.Source: §11 · p. 33
  • Read-out pressure keeps perplexity, LAMBADA, and BLiMP within baseline range but reduces the concept workspace from about 25 to 14 effective dimensions.Standard downstream metrics therefore miss this internal geometric change.Source: §11 · p. 33
  • The concept frame is largely free: 13 distinct rotations yield the same model, random prescribed bases are adopted, and downstream quality remains unchanged.The task fixes the answer axis while leaving the concept-phase frame available for deliberate choice.Source: §11 · p. 33
  • Gauge freedom can turn representation analysis from post hoc recovery into deliberate placement, enabling shared prescribed bases for direct internal comparison.This consequence follows from the frame’s indifference to the task’s output.Source: §11 · p. 33

Related papers

Loading 2608.10251v1…