PaperlayerView exact revision on arXiv

Source-linked AI summary

Gathered, Not Admitted: How Attention Brings a Latent Variable into Verbalizable Form

Parsa Mazaheri

arXiv:2608.15022v1cs.AIcs.CL

TL;DR

The paper asks whether latent variables enter a readable workspace through an admission gate. Using Jacobian lenses and paired five-arm benchmarks, it finds that demand changes visibility without changing availability, while attention transports the variable into the queried position within a mid-depth window.

  • Problem

    It is unclear whether demand makes a latent variable readable through a workspace admission gate or another mechanism.

  • Method

    The study uses Jacobian lenses and paired five-arm benchmarks with identical contexts to compare demand, operator use, and latent-variable availability.

  • Results

    Demand increases visibility across checkpoints, while attention transports the variable into the queried position at least 17× more strongly inside a mid-depth window than below it.

  • Takeaways & Limitations

    The findings refute admission at the query position and identify demand-specific, attention-mediated gathering as necessary but not sufficient for the later readable form.

  • Takeaways & Limitations

    The mechanism is characterized on only two architectures and largely one checkpoint and language task family, while the computation feeding the gather remains unlocalised.

Abstract

from arXiv · show

Language models hold latent quantities in a form they can report on, and more of a quantity is present in that form when the task requires reusing it flexibly. What causes a representation to enter that form is open, and the word workspace invites an admission story: a gate that decides what gets in. Testing it on open-weight models with Jacobian lenses, over a benchmark whose five arms share an identical context, we find no gate where it predicts one. Demand raises a concept's lens visibility beyond what applying an operator to a supplied value produces: +0.050 [+0.045, +0.057] in percentile rank on our primary checkpoint, positive on all four we measure, though that arm answers at ceiling and the accuracymatched contrast is stronger under that readout. At the same time one shared linear map decodes the variable from every arm, the control included, at 6.4-9.0x its selection-corrected floor. What produces the later readable form at the queried position is attention-mediated gathering inside a mid-depth window: separating patch depth from readout depth puts transport there at least 17x above anywhere shallower under non-saturating readouts, with no tested MLP output contributing positively inside it. Under the saturating percentile rank the same grid does not localise the window, which is a fact about that measure. An arm that needs the variable for nothing concentrates sevenfold less, so the window is demand-specific. That window has two measured edges, a survival failure below and destruction above, and it falls at the same fractional depth in a 64-layer hybrid and a 62-layer dense model from another family. We localise where the variable is installed and read, not the route from the passage, which transports nothing. But the readout is not a calibrated measure of use: three components move it to within 12% of one another and differ 7.4x in what they do to the answer.

1 INTRODUCTION

The paper argues that flexible reuse makes a latent variable more readable by gathering it into the queried position, not by admitting a variable already present there. Attention, rather than tested MLP outputs, performs this transport within a mid-depth window whose edges reflect survival failure below and destructive intervention above.

  • Motivation and mechanism: At least 17x less of the variable is transportable early at the query position than inside a mid-depth window.Across the twelve layers immediately below that window, no component transports it in every donor pairing.
  • Motivation and mechanism: Attention carries the variable into the queried position, while no tested MLP output inside the window contributes positively.This supports transport into the measurement position rather than unmasking a variable present there at every depth.
  • Motivation and mechanism: The window’s lower edge marks failure of an installed value to survive to readout, while its upper edge marks intervention shifting from substitution to destruction.Neither edge represents a capability appearing.
  • Contributions: Demand raises visibility against a prompt-format- and accuracy-matched control, while one shared linear map reads z from every arm, including the arm needing it for nothing.The visibility effect survives holding operator use fixed across four checkpoints from four families.
  • Limitations: The study localises a gather that is selectively necessary without being sufficient, rather than claiming a write circuit.What is localised is transport into the queried position, not the route from the passage.

2 SETUP

The study uses Jacobian lenses with a final-normalizer readout, while controlling for depth-dependent residual rescaling and percentile-rank saturation. JGateBench provides paired five-arm prompts over identical contexts on Qwen3.6-27B and four additional checkpoints.

  • Jacobian lens: The Jacobian lens reads layer ℓ as softmax(WU norm(Jℓhℓ)), with Jℓ = E[∂hfinal/∂hℓ] and Jℓ = I recovering the logit lens.J-space is defined as a union of cones formed by sparse nonnegative combinations of vocabulary-indexed lens vectors.
  • Invariance: Depth-dependent residual rescaling changes homogeneous nonzero-degree matrix statistics while leaving the normalized readout unchanged.A global rescaling is insufficient to produce this issue; the rescaling must vary with depth.
  • Readouts: 92% of flexible-arm cells at L40–L44 read above 0.999 in percentile rank, compressing depth profiles where effects are largest.The percentile rank separates rank 1 from rank 25 by 0.0001, so Lz = −log10(1+|{v : λv > λz}|) is also recorded for top-vocabulary resolution.
  • Models: Qwen3.6-27B is the primary checkpoint, with 64 layers, d = 5120, vocabulary 248,320, and a published lens over all 63 source layers.It is a hybrid with attention every fourth layer and gated delta nets in the other 48; four additional checkpoints test generality.
  • Benchmark design: Each JGateBench instance yields five paired prompts over an identical context, contrasting flexible with a format-matched control that needs z for nothing.The other arms are automatic, report, supplied, and flexible; supplied applies the same operator to a value given in the prompt.

3 DEMAND CHANGES VISIBILITY, NOT AVAILABILITY

Demand increases a concept’s lens visibility even when accuracy is matched, while a shared probe decodes the variable across all conditions. The result indicates a change in readable form rather than availability, with operator effects varying by checkpoint and readout.

  • 3 DEMAND CHANGES VISIBILITY, NOT AVAILABILITY: ∆Rz = +0.0891 [+0.0799, +0.0983] against the format-matched control, while both arms have accuracy 0.940, showing visibility rises without a difficulty difference.The non-saturating margin contrast is +2.79 [+2.57, +3.01].
  • 3 DEMAND CHANGES VISIBILITY, NOT AVAILABILITY: 0.575 to 0.808: one jointly fitted multinomial probe decodes z above chance in every arm, including control, across disjoint semantic instances.The probe uses a single weight matrix with no per-arm adjustment and selects layers on inner training splits.
  • 3 DEMAND CHANGES VISIBILITY, NOT AVAILABILITY: +0.0504 [+0.0445, +0.0567] visibility for inference-plus-operator use versus supplied-value operator use, positive across four checkpoints.The supplied arm receives a dummy candidate rather than the true value, preventing visibility from being obtained by copying its rank.
  • 3 DEMAND CHANGES VISIBILITY, NOT AVAILABILITY: +0.039, +0.178, +0.029 and −0.059 for the operator across four checkpoints, so its contribution is not stable in sign.On the primary checkpoint, the operator is +0.0387 in percentile rank but −1.26 [−1.37, −1.15] in the margin.
  • 3 DEMAND CHANGES VISIBILITY, NOT AVAILABILITY: +0.1792 [+0.1625, +0.1961] in tracking and +0.2931 [+0.2775, +0.3084] on Qwen3.5-9B replicate the demand effect across task family, model, and lens.The Qwen3.5-9B report condition’s gold concept is also its next token, so some visibility reflects imminent production.

4 THE VARIABLE IS TRANSPORTED, NOT UNMASKED

The later readable, causally substitutable variable is transported to the query position rather than unmasked there. Transport concentrates in a demand-specific attention-mediated window, with no tested MLP output contributing positively inside it.

  • Scope: The result concerns later readable, causally substitutable content at the query position, not whether the quantity was absent earlier or how it arrived from the passage.Early nonlinear, lens-misaligned, or substitution-destroyed encodings can nevertheless appear weak under this readout.
  • Transport window: At least 17.0× separates transport in the L39 window from the largest shallow cell under Lz, while Rz gives 1.1–2.7× with every interval covering 1.Under Mz, the corresponding ratio is 51.6–62.6×; the shallow maximum is selected within each resample, making the ratio conservative.
  • Demand specificity: 30.3× [18.6, 53.4] is flexible’s window concentration versus 4.2× [2.9, 6.6] for the control arm.The non-overlapping intervals show that the window is demand-specific rather than determined solely by shared passage geometry.
  • Branch mechanism: At L39 under Lz, attention contributes +1.618 versus +1.908 for the residual stream, corresponding to 85% of the stream’s effect.Under the saturating Rz metric, attention and residual effects are nearly equal: +0.0135 versus +0.0137.
  • Branch mechanism: No tested MLP output contributes positively inside the window, while attention carries most of the measured transport.The passage cautions that nonlinear patch effects are not additive, so these figures do not prove attention carries the whole effect.
  • Cross-family replication: +0.0346 to +0.0505 is transported by resid.L39 in the second family across four donor pairings, surviving Benjamini–Hochberg over 42 components.Attention blocks at L39 and L48 and prespecified head attn.L39.H15 also survive, whereas mlp.L39 is negative in all four.

5 THE WINDOW HAS TWO MEASURED EDGES

The transport window has two measured edges: it becomes detectable after survival fails at shallow depth, then is destroyed when deeper patches replace the late computation. Its fractional position replicates across architectures, but interpretation requires a checkpoint with near-ceiling task accuracy.

  • Measured edges: At n = 150, the counterfactual is null through L33, becomes detectable at L36 (+0.113 [+0.057, +0.177], task accuracy 0.807), and plateaus over L39–L45 at +0.227 to +0.400.The interpretable cells are L36–L42 because accuracy falls from 0.920 to 0.19 by L48; neither edge is a step and no peak is named.
  • Measured edges: Above the interpretable window, accuracy collapses and a patch replaces the late computation wholesale, producing the upper destruction edge.Significance continues to L57, but the cells are no longer interpretable once accuracy has fallen substantially.
  • Measured edges: At L33, the transported value is entirely gone (+0.0048 [−0.0010, +0.0113]), while at L36 about half survives (+0.0453 [+0.0326, +0.0596]) against the +0.0888 asymptote.This fixed-later-band readout measures survival directly, supporting the lower edge as a survival failure.
  • Cross-architecture replication: At fractional depth 0.60 in 62-layer gemma-4-31B-it, L37 yields +0.170, +0.107 and +0.140 across three donor pairings at task accuracy 0.74 to 0.77.Qwen3.6-27B’s interpretable cells sit at 0.56–0.66, supporting fractional depth as the cross-architecture unit.
  • Scope condition: Llama-3.1-8B-Instruct is not interpretable under the criterion because its flexible arm solves at 0.610 and every non-null effect occurs at patched accuracy 0.34 or below.The design needs a checkpoint starting near ceiling; this is a scope condition rather than evidence against the window.

6 WHAT A SINGLE DONOR PAIRING IS WORTH

A single donor pairing is an unstable basis for component-level significance: different seeds produce substantially different discoveries, with only 14 of 42 components surviving all four pairings. Pairing also changes the estimator’s residual nuisance, while the control cannot distinguish transport from wholesale copying without task accuracy.

  • Pairing sensitivity: 21, 19, 15, and 23 of 42 components are significant across four pairings, but only 14 survive all four.A single pairing reports 7–64% more significant components than are stable, with the excess almost entirely individual heads.
  • Pairing sensitivity: +0.0043 under one pairing and −0.0154 under another show a ±0.015 nuisance from the draw rather than a directional instrument bias.The residual is exact when the realised candidate set is recorded per trial; the correct statement asserts no direction.
  • Estimator limits: The control cannot separate transport from wholesale copying; task accuracy, not the estimator, provides that guard.Under an adversarial bound, L39–L45 survive on both pairings while L36 does not.

7 WHAT THE BEHAVIOUR RUNS THROUGH

The behavior selectively runs through resid.L39: ablating it sharply harms the report and flexible arms while improving the control. A derived concept direction accounts for part of the effect, beyond norm loss or rival-concept projections.

  • Selective necessity: −0.600 [−0.740, −0.460] report loss and −0.120 [−0.220, −0.040] flexible loss contrast with +0.080 [+0.020, +0.160] control improvement after resid.L39 ablation.resid.L39 is the only component clearing zero in this selectivity comparison.
  • Derived direction: +0.113 to +0.193 at L39 and +0.080 to +0.147 at L42 are lost when the derived lens direction is projected out.These losses equal 19–46% of the answer rate on the same trials, with task accuracy recovering correspondingly.
  • Control comparisons: −0.013 to +0.007 for norm-matched random directions and −0.017 to +0.015 for rival-concept means are null, unlike the gold direction’s +0.113 to +0.193.The controls test whether the effect comes from projection or activation-norm removal rather than the candidate direction.
  • Control comparisons: +0.127 to +0.213 remains after orthogonalising the gold direction against the other 19, while the worst rival reaches only +0.014 to +0.042.Two null controls and two positive direction-specific tests make the effect difficult to attribute to projection rate alone.

8 A READOUT SHIFT IS NOT A MEASURE OF USE

A Jacobian-lens readout shift is not a calibrated measure of causal use: components can move the readout similarly while producing sharply different answer effects. A controlled attention-output patch demonstrates this mismatch and motivates causal testing before interpreting readout differences.

  • Readout shift versus causal use: +0.1333 [+0.0871, +0.1847] was the donor-language lens shift from mismatched attention-output patching, 4.9× the target-language shift.The same component was the only one whose effect reversed with substitution direction.
  • Readout shift versus causal use: +1.83, +1.68 and +1.63 were the three components’ donor-readout shifts, within 12%, but donor-minus-distractor effects differed 7.4×.The corresponding margin effects were +2.56, +0.44 and +0.35, with weak trial-level correlations of +0.27, +0.18 and +0.08.
  • Methodological implication: A controlled Jacobian-lens instance, paired with a matched positive control, shows why causal testing should precede interpreting readout differences.The positive control demonstrates that the concept-aligned component is not generally epiphenomenal.

9 RELATED WORK

Prior work distinguishes automatic from flexible representations and shows attention relaying workspace content, but leaves open what causes representations to enter the workspace. Related systems posit a gate, whereas this work argues that no gate decides over a variable before it reaches the readout position.

  • Prior workspace accounts: Gurnee et al. (2026) distinguish automatic from flexible representations and report attention heads relaying J-space content between positions, while leaving workspace entry unexplained.Their relay heads are selected from weights and ablated at every token position, so relay is not separated from same-position writing.
  • Gate-based accounts: Concurrent systems construct a gate against discovery, but this work argues that the gate is not architectural because no decision governs a variable before it reaches the readout position.The claim concerns a variable that is not yet at the readout position.
  • Attention-mediated transport: The proposed gather joins literature on attention moving content into the prediction position, including Geva et al. (2023), Wang et al. (2023), and Todd et al. (2024).The passage identifies the gather as part of an established literature on attention-mediated content movement.

10 LIMITATIONS

The study rules out an admission step that unmasks a variable already at the query position, but its evidence leaves an attention-route gate possible. Its mechanistic conclusions are limited in scope and localize gathering, not the computation that feeds it.

  • Interpretive limitation: The results refute admission at the query position, while a gate on the attention route remains consistent with the evidence.Passage-attention mass at the gathering layer is 1.90× as demand-sensitive as at 72 heads of three non-gathering layers by median comparison, but only 1.16× by maximum comparison, and is measurable only on full-attention layers.
  • Scope: Mechanistic evidence is narrower than the entry-effect evidence: the window is tested on two architectures, while route, head, mediation, and ablations are tested only on Qwen3.6-27B and its language family.The entry effect is measured on five checkpoints and two task families.
  • Localization limit: The study localizes the gather but not the computation feeding it, because no patched passage span transports the variable.The variable’s pre-gather form therefore remains unlocalised rather than distributed.
  • Mechanistic limitation: The gather is necessary but not sufficient for the later readable form.The passage introduces this qualification alongside the unlocalised computation feeding the gather.

11 CONCLUSION

The conclusion rejects a gate-based admission account: the variable is already decodable across arms, while attention moves it into the measured position at a shared fractional depth. This transport is demand-sensitive, but readout shifts do not measure a component’s causal effect on the answer.

  • Conclusion: The variable is decodable from every arm, so admission does not occur where the readout is taken.The measurement instead sits where the variable has to be moved to.
  • Conclusion: Attention performs the necessary but insufficient moving at one fractional depth across two architectures, with sevenfold sharper concentration under demand.
  • Conclusion: Readout-shift magnitude does not indicate what a component does to the answer.

A THE BENCHMARK, AND THE INVARIANTS IT ENFORCES · B THE GAUGE ARGUMENT, AND THE SPARSE DECOMPOSITION

The benchmark holds context and candidate labels constant across five arms while enforcing controls against label, tokenization, and open-vocabulary artifacts. A depth-varying gauge leaves model function unchanged but alters Jacobian diagnostics, while sparse pursuit shows lens coefficients are only an imperfect proxy for representation strength.

  • A THE BENCHMARK, AND THE INVARIANTS IT ENFORCES: The five arms share one passage and reference table, while instructions vary across automatic, report, flexible, control, and supplied conditions.The supplied condition names Swedish although the passage is Japanese, making its gold symbol V rather than K.
  • A THE BENCHMARK, AND THE INVARIANTS IT ENFORCES: The design enforces label symmetry, passage-external labels, forced choice, continuation-form scoring, and tokenizer-specific single-token filtering.All 20 language names pass the tokenizer filter, while 9 of 18 operator symbols pass and all survivors are letters.
  • A THE BENCHMARK, AND THE INVARIANTS IT ENFORCES: +0.1126 [+0.1015, +0.1240] versus +0.0891 shows that breaking label symmetry inflates the measured contrast.The shared-table design prevents the flexible-arm operator table from printing the gold label in only one arm.
  • B THE GAUGE ARGUMENT, AND THE SPARSE DECOMPOSITION: The depth-varying reparameterisation rescales residual streams while preserving the transformer’s computed function and logits through scale-free normalisation.A global rescale would leave Jℓ unchanged; the argument therefore requires depth-varying scales and does not claim realizability in the original architecture.
  • B THE GAUGE ARGUMENT, AND THE SPARSE DECOMPOSITION: 1.4×10−5 maximum logit change and 4.0 × 10−6 maximum Rz change verify numerical function preservation while the Jacobian diagnostic shifts as predicted.With aℓ = 1 below layer 20 and 2 at or above it, the mean diagonal of Jℓ moves by 2.0000 at layer 19 and 1.0000 at layer 20.
  • B THE GAUGE ARGUMENT, AND THE SPARSE DECOMPOSITION: The Jacobian-lens readout is gauge-invariant without refitting, whereas a learned affine translator requires refitting because its bias does not scale with the hidden state.The invariance follows from Jℓhℓ being homogeneous of degree one.
  • B THE GAUGE ARGUMENT, AND THE SPARSE DECOMPOSITION: Spearman 0.24 between normalized gold coefficient Sz and Rz shows sparse pursuit is an approximation rather than a certified or dependable dependent-variable measure.A concept can enter support at rank 13,332, while another can rank third and remain outside support; the optimization is combinatorial.

C THE BEHAVIOURAL WINDOW, IN DETAIL · D TRANSPORT AND THE ATTENTION ROUTE, IN DETAIL

The behavioural window is bounded by survival below and destruction above, with donor-value transport concentrated in mid-depth query representations rather than passage patches. Attention heads carry this transport, but head-level effects and mediated-share estimates are pairing- and metric-sensitive.

  • C THE BEHAVIOURAL WINDOW, IN DETAIL: L39–L45 survive the adversarial bound, while L36 survives measured residual and pairing controls but not the worst-pair bound.The deep cells fail for a substantive reason: residual patches can transplant near-final computation and emit the donor’s symbol without transport.
  • C THE BEHAVIOURAL WINDOW, IN DETAIL: +0.1250 [+0.0437, +0.2125] at four patched query positions is the cleanest span result, while wider spans increase flips but reduce accuracy from 0.900 to 0.550.At one position, the flip rate is null but the graded margin reaches +0.550 [+0.271, +0.831].
  • C THE BEHAVIOURAL WINDOW, IN DETAIL: +0.0453 [+0.0326, +0.0596] is the first significant survival effect at L36, rising to +0.0888 at L45, while target values are suppressed monotonically.The source-patch control transports only +0.0007 to +0.0025 on its own and adds nothing beyond a query patch.
  • C THE BEHAVIOURAL WINDOW, IN DETAIL: 45.8%, 33.3% and 43.9% at L39, versus 36.7%, 27.3% and 18.5% at L42, quantify mediated shares that vary across pairings and are not reported as a stable point.The ratio is pairing-sensitive because it divides gold-removal cost by full-arm donor-symbol rate; it is the least stable quantity measured.
  • D TRANSPORT AND THE ATTENTION ROUTE, IN DETAIL: The second family replicates transport through attn.L48 and attn.L39.H15, whereas mlp.L42 is null across all four pairings.Both attention cells clear zero under Rz, Lz and Mz; correction holds under Rz and Mz for every pairing and under Lz in three of four.
  • D TRANSPORT AND THE ATTENTION ROUTE, IN DETAIL: 21 / 19 / 15 / 23 of 42 survivors under Rz leave only 14 four-way intersections, showing strong head-level instability across donor pairings.Under Lz the counts are 23 / 18 / 16 / 22, and under Mz 26 / 25 / 21 / 29; ten of twelve pairing-sensitive cells are attn.L39.H* effects of ±0.001–0.005.
  • D TRANSPORT AND THE ATTENTION ROUTE, IN DETAIL: H13 (+0.0099) and H14 (+0.0093) each carry roughly half of H15’s language effect, while H16 opposes transport at −0.0421 [−0.0724, −0.0196].Thus H15 is sufficient for the block’s transport effect rather than uniquely necessary, and the layer contains both installing and opposing heads.
  • D TRANSPORT AND THE ATTENTION ROUTE, IN DETAIL: +0.1091 [+0.0965, +0.1217] versus +0.0891 [+0.0799, +0.0983] shows that the visibility effect survives regenerated equal-length instructions with an identical tail.The route comparison used one clean forward pass per record, without patching, comparing L39 against heads in L15, L27 and L51.

E CROSS-ARCHITECTURE SCREENING, AND THE CHECKPOINTS THAT FAILED IT … H ARTIFACTS AND REPRODUCIBILITY

Cross-checkpoint screening finds a positive operator-controlled contrast across four eligible dense models, while ceiling/floor failures and tokenizer, homogeneity, and accuracy limitations constrain comparability. Supporting analyses show a demand-specific behavioral window, caution that readout shifts are uncalibrated, and document reproducible, audited artifacts and fixed evaluation procedures.

  • E CROSS-ARCHITECTURE SCREENING, AND THE CHECKPOINTS THAT FAILED IT: Four eligible dense checkpoints pass the required tokenization and homogeneity screens, while Mistral fails tokenization and Gemma-3 fails homogeneity and lens numerical stability.The screens also require solving the flexible arm near ceiling for the depth sweep; exclusions are measurement-based rather than family-based.
  • E CROSS-ARCHITECTURE SCREENING, AND THE CHECKPOINTS THAT FAILED IT: L37 is the only Gemma-4 layer significant under all three donor pairings while accuracy still permits a reading; effects from L40 upward are larger but are destruction, not transport.At L37, the three pairing effects are +0.170 [+0.103, +0.237], +0.107 [+0.047, +0.170], and +0.140 [+0.073, +0.210].
  • F SUPPORTING FIGURES AND TABLES: A readout shift is not calibrated use: three components shift visibility within 12% of one another while their behavioral effects differ 7.4×.On Gemma-4 at L37, resid.L37 produces +0.1700 [+0.1033, +0.2367] at accuracy 0.760, whereas attn.L37 and mlp.L37 each produce +0.0200 with intervals including zero.
  • F SUPPORTING FIGURES AND TABLES: +0.0517 [+0.0336, +0.0715] is the visibility increase from writing flexible L39 attention output into the automatic run, but the exploratory bidirectional screen supports no significance claim.The reverse swap lowers visibility by −0.0270 [−0.0369, −0.0188], making attn.L39 the only component with opposite-sign directions excluding zero.
  • G A WORKED EXAMPLE: Rz can mislead: control reaches Rz = 0.9953 while the gold token ranks 1155th, whereas report reaches Rz = 1.0000 at ranks 5 and 1.The worked example shows the concept rising to the top specifically in the flexible and report arms, while other arms read task-specific content.
  • H ARTIFACTS AND REPRODUCIBILITY: Every behavioral number uses one forward pass and candidate-token argmax, with stochasticity limited to seeded splits, pairings, and bootstrap resampling.Patching and readout positions are separately pinned, and patch semantics are specified for residual, MLP, and attention components.
  • H ARTIFACTS AND REPRODUCIBILITY: The released artifacts include datasets, generators, fitted lenses, and paths for every number, with an audit command machine-checking document numerals against generated JSON or JSONL artifacts.The published 27B lens lacks a configuration file and convergence trace; reconstructed datasets are byte-identical at the specified revisions without redistributing source passages.
Loading 2608.15022v1…