PaperlayerView exact revision on arXiv

Source-linked AI summary

Residual Streams Read, Recurrent States Remember: The Global Workspace in Mamba Models

Wenlong Wang, Fergal Reid

arXiv:2609.32102v1cs.AI

TL;DR

The paper asks whether global-workspace representations extend from transformers to recurrent state-space language models. It extends Jacobian-lens readouts to Mamba residual streams and recurrent states, finding complementary recovery and persistence, while steering gains are less consistent for relational reasoning.

  • Problem

    It is unknown whether intermediate concepts in state-space language models enter a shared, broadly usable representational format, including whether recurrent state preserves concepts as residual visibility changes.

    Source: §1 · pp. 1–2

  • Method

    The paper extends Jacobian lenses to recurrent states and compares state, residual, and joint readouts, then tests lens-derived edits for verbal report and relational redirection.

    Source: §1 · p. 2, §9 · p. 15

  • Results

    Joint readouts improve over the residual J-lens on at least five of six task families in every tested Mamba checkpoint; Mamba-2 state alone exceeds residual and logit lenses on all six.

    Source: §9 · p. 15

  • Takeaways & Limitations

    Recurrent state provides a complementary carrier of intermediate concepts, and its recovery, persistence, and causal uses require separate measurements.

    Source: §9 · p. 15

    Checkpoint comparisons do not isolate architecture from training data, parameter count, tokeniser, precision, or competence, and some causal tests use only items the unedited model answers correctly.

    Source: §8.1 · p. 15

Abstract

from arXiv · show

Can the global-workspace account of transformer representations extend to state-space language models? We fit Jacobian lenses to the residual streams and recurrent states of Mamba-1, Mamba-2 and Mamba-3, using the original 1000-prompt recipe. Joint residual--state readouts improve recovery of known intermediate concepts over the residual lens on at least five of six task families in every tested Mamba checkpoint. On Mamba-2, state alone exceeds residual and logit lenses on all six families; a normalised joint readout improves on both components on five. Temporal maps and word-list experiments show earlier content remaining state-readable as residual visibility changes. We also propose sign-guarded steering, which improves target top-five success over coordinate exchange on matched verbal-report trials in five models. Recurrent state alone supports this verbal access. These gains do not extend consistently to relational answers: guarded edits often output the edited concept itself, and Mamba-3's joint edits can disrupt successful state-only redirection. Recurrent state thus provides a complementary carrier of workspace content, whose recovery, persistence and causal uses require separate measurements.

1 Introduction

The paper asks whether intermediate concepts in recurrent state-space language models become broadly available, extending the global-workspace question beyond transformers. It studies where concepts can be recovered, whether they persist, and whether readouts can redirect computation.

Motivation

  • The study asks whether intermediate concepts enter a shared, broadly usable representational format, as global-workspace theories propose for selected information.Prior work introduced the Jacobian lens and J-space for transformer representations; recurrent models motivate extending interpretability beyond standard transformers because latent updates are not exposed as separate generated-text steps.Source: §1 · pp. 1–2

Contributions

  • With all 63 source blocks, Mamba-2’s state lens beats residual and logit baselines on every evaluation family.The paper introduces full-path and readout-only state J-lens variants.Source: §1 · p. 2
  • Joint state–residual readouts improve over the residual J-lens on at least five of six families for every tested Mamba model.Temporal traces and controlled list prompts also show earlier content remaining state-readable, with counterexamples to a universal handover story.Source: §1 · p. 2
  • Sign-guarded steering improves verbal-report success over coordinate exchange on matched trials across all five models tested with both methods.The gains do not consistently extend to two-hop reasoning or flexible generalisation, where outcomes depend on model and intervention site.Source: §1 · p. 2

Scope

  • The authors report evidence for verbal recovery and persistence, while limiting causal access claims to the task, edit construction, and intervention site.They do not infer consciousness, and checkpoint comparisons do not isolate architecture from training, scale, or competence.Source: §1 · p. 2

2 Background

The background presents the Jacobian lens as a way to read residual activations through average downstream effects, then motivates recurrent state as a second observation point alongside the residual stream.

2.1 The Jacobian lens and functional workspace claims

  • The J-lens averages input–output Jacobian effects over prompts and source positions, with causal masking excluding future-to-past terms.The resulting token-labelled directions are used to reveal concepts a model is poised to verbalise.Source: §2.1 · p. 3
  • The J-lens differs from the logit lens by accounting for the model’s average downstream transformation, and from the tuned lens by deriving directions from derivatives rather than a supervised affine decoder.Residual activations are transported into final-residual coordinates before final normalisation and unembedding.Source: §2.1 · p. 3

2.2 Residual streams and recurrent states

  • In the schematic SSM update, the recurrent state is updated from the prior state and current input, while the output combines state and input contributions.Mamba variants differ in their selective, structured-duality, or richer complex/MIMO recurrence designs.Source: §2.2 · p. 3
  • The residual stream is token-local while recurrent state compresses the prefix, giving two simultaneous observation points rather than mutually exclusive memory systems.The paper’s narrower question is whether averaged Jacobians expose known intermediate concepts in both and whether their directions support functional tests.Source: §2.2 · p. 3

3 Methods

The methods compare residual, recurrent-state, and joint readouts across model checkpoints, then test intermediate recovery, persistence, and causal edits. Jacobian lenses are fitted with a shared recipe, while task and intervention protocols distinguish concept recovery from downstream answers.

3.1 Models and coverage

  • Checkpoint coverage varies, and Nemotron’s joint readout is incomplete because its measured interfaces omit the attention KV cache.The MIMO checkpoint has readout results but not the full state intervention suite.Source: §3.1 · p. 3, Table 1 · p. 4

3.2 State and joint Jacobian lenses

  • State Jacobian lenses map post-update recurrent states into final residual coordinates, with full-path and readout-only variants differing in future-state propagation.The joint representation pairs the residual activation and recurrent state; Mamba-3 uses matching rotary-frame coordinates, and a fixed permutation controls coordinate correspondence.Source: §3.2 · p. 4

3.3 Fitting recipe

  • The paper-comparable lenses use 1000 WikiText-103 prompts, with valid fitting positions excluding the first 16 and final token positions.Target contributions are summed without an extra division, and weighted reverse-recurrence calculations avoid materialising every state Jacobian.Source: §3.3 · pp. 4–5

3.4 Intermediate-recovery protocol

  • Six task families score recovery of known, single-token intermediate concepts using normalised pass@k AUC rather than task-answer accuracy.Any-block recovery uses every fitted block, while intervention tests use fixed middle-half bands and include a band-restricted recovery control.Source: §3.4 · p. 5

3.5 Persistence and causal tests

  • Persistence tests track intermediate vocabulary ranks and later recovery of words in 80-word lists; causal tests ablate selected lens directions or edit residuals and states.Interventions use fixed prompt bands and paired trials; category-report scores the edited concept, whereas the suite scores a relation or function applied to it. Local contrast improvement does not guarantee downstream task improvement.Source: §3.5 · pp. 5–6

4 Recurrent states expose information missed by residual readouts

Recurrent-state readouts recover known intermediate concepts better than residual readouts across tested Mamba checkpoints, and combining the interfaces often improves recovery further. Recovery also varies with depth: state readout peaks earlier on several task families, while the ordering depends on the task.

4.1 Residual and state readouts are complementary

  • State readouts beat residual J-lenses on five of six families for Mamba-1 and Mamba-3 SISO, all six for Mamba-2, and four for Mamba-3 MIMO; plain-sum joint readouts improve over residual on six families for Mamba-1 and Mamba-2 and five for both Mamba-3 variants.Table 3 reports directional per-family comparisons, not statistical significance for every difference.Source: Table 3 · p. 7, §4.1 · p. 7
  • Across all 63 Mamba-2 source blocks, the Jfull state lens beats residual, logit, and tuned lenses on all six families.The largest reported differences are order of operations (0.795 versus 0.460 residual) and typo correction (0.706 versus 0.303); poetry state AUC is 0.101.Source: §4.1 · p. 7, Table 4 · p. 7
  • The normalised joint readout exceeds both Mamba-2 components on five of six families; association is the exception, with 0.231 joint versus 0.237 state.For multihop, AUC rises from 0.475 for residual and 0.557 for state to 0.679 jointly; the plain sum beats state on only two families.Source: §4.1 · p. 7, Figure 1 · p. 8

4.2 Recovery across depth

  • For multihop, multilingual, and arithmetic tasks, recovery peaks earlier in state than in the residual stream across Mamba-1, Mamba-2, and Mamba-3.Typo correction often instead favours an earlier residual readout.Source: §4.2 · p. 8
  • Restricting both interfaces to the same middle-depth range preserves stronger state recovery on most task families in every tested Mamba checkpoint.The depth ordering varies by task, and the analysis then tracks readability across token positions as prompts unfold.Source: §4.2 · p. 8

5 Earlier content remains state-readable

Temporal traces and controlled word lists show that recurrent states can keep earlier content readable, while residual visibility varies and persistence differs by concept. The amount of earlier content recovered also depends substantially on source-block coverage.

5.1 Following concepts across token positions

  • At state block 28, Seattle stays top-ranked from the end of “Needle” through the final prompt token, while residual visibility drops and later recovers.May shows the opposite pattern in the emerald question: it becomes readable late in the residual stream but remains outside the state lens’s top 100; both prompts are answered correctly.Source: §5.1 · p. 8, Figure 2 · p. 9

5.2 Controlled evidence for earlier-token content

  • Expanding Mamba-1 coverage from a sparse subset to all 63 source blocks raises words held@5 from 0.3 to 7.5 for Jfull and 7.8 for Jread.For Mamba-2, expanding Jfull coverage from 23 to 63 blocks raises words held@5 from 3.1 to 6.7 and transported-state pursuit from 10.1 to 18.6 words.Source: §5.2 · p. 9, Table 5 · p. 10
  • At equal coverage, Mamba-1 and Mamba-2 have similar transported-state pursuit occupancy, 18.7 and 18.6, while rank-based occupancy is 7.5 and 6.7.These estimators measure lens-recoverable content, not total model memory; matched coverage removes an earlier apparent monotonic trend across Mamba generations.Source: §5.2 · pp. 9–10
  • Recovery and persistence establish recurrent state as an additional carrier of workspace content, but a readable direction does not by itself establish causal access or relational use.The following intervention analysis tests whether lens-derived edits elicit verbal reports and redirect two-hop answers.Source: §5.2 · p. 10

6 Workspace properties across residual streams and recurrent states

Across the measured workspace properties, recurrent states provide recoverable content and can support verbal access, but intervention outcomes depend on task and edit site. Reporting an edited concept does not ensure that the model uses it to answer a relation.

6.1 Workspace properties across models

  • Coordinate exchange reaches the requested two-hop answer on 5–15 of 90 trials and flexible-generalisation answers on 12–30 of 192 trials at the displayed interfaces.These rates characterize a particular intervention, and a thresholded null does not establish that another edit cannot access the representation.Source: §6.1 · p. 12
  • State-readable word-list capacity is 7.5, 6.7, 5.9, and 3.1 words for Mamba-1/2/3 and Nemotron, respectively, exceeding Qwen’s 0.7 and Pythia’s 2.1 residual-readable words.These counts measure lens-recoverable content under the stated coverage, not total model memory; the residual–state union is bounded separately.Source: §6.1 · pp. 10, 12, Table 6 · p. 11
  • State introspection is strong on Mamba-3 and Nemotron, reporting injected tokens at top-1 on 49/101 and 93/99 trials, but is less consistent across recurrent models.Mamba-1 reports 0/99 and Mamba-2 4/99; the prompted reporting prefix tests positive injection, not spontaneous detection or reliable reporting of unedited internal state.Source: §6.1 · p. 12
  • Historical signed state clamps also vary by model: Jread gives 19/90 two-hop redirects on Mamba-3 and 7/90 on Mamba-1, while Mamba-2’s full-band Jfull clamp gives 1/90.Pythia and Nemotron further separate recovery from intervention outcomes: their recovery comparisons do not predict whether coordinate exchange produces the requested answer.Source: §6.1 · p. 12

6.2 Steering residual and state representations towards verbal report

  • Sign-guarded steering increases target-report success on all five models tested with both methods, from 130/621 to 621/621 on Nemotron and 114/335 to 333/335 on Mamba-2.Success means the target enters the next-token top five; state-only edits also show the effect, but edit magnitudes differ, so equal-dose efficiency is not established.Source: §6.2 · pp. 12–13, Table 7 · p. 13

6.3 Tracing intervention outcomes in two-hop reasoning

  • Sign-guarded steering often outputs the edited concept itself rather than the requested relational answer, including on trials the model originally answered correctly.This separates stronger verbal report from successful use of the concept in a relation.Source: §6.3 · p. 13
  • On Mamba-3, state-only guarded steering produces the requested answer on 16 of 90 trials, while joint residual–state steering succeeds on none.The output patterns do not identify the underlying mechanism.Source: §6.3 · p. 13

7 Related work

The paper builds on Jacobian-lens work that identifies representations poised to affect verbalisation, and extends interpretability comparisons to recurrent state. It treats workspace properties as functional criteria rather than evidence about consciousness.

Related work

  • The Jacobian lens uses an average derivative to identify representations poised to affect future verbalisation, unlike logit and tuned lenses’ direct or learned residual decoding.The original J-space work relates these readouts to functional workspace properties; this paper focuses on complementary verbalisable content in recurrent state and tests intervention outcomes.Source: §7 · p. 13
  • Figure 3 displays saved first-token predictions, separating requested answers, edited concepts, and other outputs across two-hop trials.Its panels distinguish all trials from those initially answered correctly; these are not complete responses.Source: Figure 3 · p. 14
  • Prior interpretability work tested techniques on recurrent models, while this study fits the full future-path Jacobian of recurrent state and compares residual, state, and joint readouts.Mamba and related SSMs use selective recurrence rather than attention-based sequence mixing [Gu and Dao, 2023; Dao and Gu, 2024; Lahoti et al., 2026].Source: §7 · p. 14
  • Workspace properties are used as functional criteria—such as integration, broadcast, report, and flexible use—rather than as a claim about a particular decoder.The analogy draws on recurrent and long-range interactions in global neuronal workspace accounts and a proposed functional role for serial transformer layers.Source: §7 · p. 14

8 Discussion

The discussion bounds conclusions by checkpoint differences, single-token scoring, and intervention-specific interpretive limits. It proposes matched-task studies and broader checkpoints to test recovery and intervention more informatively.

8.1 Limitations

  • Checkpoint comparisons do not isolate architecture because models differ in training data, parameter count, tokeniser, precision, and competence.Nemotron also changes attention, MoE layers, scale, and training, and its KV cache is unobserved; evaluation covers single-token surfaces with incomplete coverage, especially for multilingual items.Source: §8.1 · p. 15
  • Some causal tests include only items answered correctly before editing, and native edit magnitudes are not matched across operators.Saved first-token predictions omit full continuations and probability changes; Mamba-3 verbal results retain a one-trial replay discrepancy near the top-five boundary.Source: §8.1 · p. 15

8.2 Future work

  • The experiments test selected functional properties associated with a workspace; they neither establish nor refute broad claims about consciousness.Future work proposes evaluating alternative Jacobian aggregators on the same recovery and intervention protocol and testing larger pretrained or posttrained Mamba checkpoints.Source: §8.1 · p. 15, §8.2 · p. 15
  • A matched-task follow-up could compare direct naming and attribute queries, test natural counterfactual competence, and freeze doses across activation-edit designs.Saving concept and answer probabilities alongside full continuations could help distinguish failure hypotheses; mediation would require its own causal test.Source: §8.2 · p. 15

9 Conclusion

Recurrent-state Jacobian lenses extend workspace-content recovery beyond residual streams, and joint readouts improve recovery across the tested Mamba checkpoints. The conclusion is qualified by model- and intervention-specific results, distinct protocols, and a recovery cost from one lens ablation.

9 Conclusion

  • Joint residual–state readouts improve on the residual J-lens on at least five of six families in every tested Mamba checkpoint; Mamba-2 state alone beats residual and logit lenses on all six.On Mamba-2, the normalised joint readout improves on both components on five families; temporal and word-list experiments show content remaining state-readable as residual visibility changes.Source: §9 · p. 15

Reproducibility statement

  • The code fixes the fitting recipe, task prompts, scoring, and causal constructions; the planned public artefact release will include checkpoint revisions, seeds, hardware details, and result manifests.The passage describes planned release contents, not an already available release.Source: Reproducibility statement · p. 16

A.1 Task families and token coverage

  • The six evaluation families contain 93 multihop, 107 multilingual, 55 order-of-operations, 98 poetry, 102 association, and 96 typo prompts.Mamba-2 single-token coverage varies by family, from 92/92 association intermediates to 394/428 multilingual intermediates.Source: §A.1 · p. 17

A.2 Any-block and band-restricted recovery

  • Mamba-2’s Jfull state lens exceeds both residual baselines on all six families, while its joint readout is not uniformly best within the intervention band.Table 8 reports both all-source-block and intervention-band comparisons.Source: §A.2 · p. 17, Table 9 · p. 18, Table 8 · p. 18

A.3 Additional recovery baselines and comparison models

  • Pythia’s conforming J-lens exceeds logit and tuned lenses on all six families; Nemotron’s residual–state sum improves on residual in four of six under the post-update convention.Nemotron’s joint readout omits the attention KV cache, and these checkpoint comparisons do not isolate attention’s contribution.Source: §A.3 · pp. 18–19, Table 10 · p. 18

A.4 Recovery cost of removing lens output row 2107

  • Zeroing Mamba-2 lens output row 2107 changes the readout map without changing model weights or unedited activations, with three family-level losses exceeding 0.02 AUC.The intervention is output-row removal, not removal of an input column of the Jacobian.Source: §A.4 · p. 19, Table 11 · p. 19

B Intervention conventions and denominators

  • Intervention counts use distinct protocols and denominators: Mamba-2’s historical signed-clamp hits were 1/1/1 out of 90, while its residual coordinate exchange was 10/80 overall.The paired workspace comparison uses 90 items and unweighted directions; legacy exchange uses a separate protocol and excludes items without single-token surfaces.Source: B Intervention conventions and… · p. 19, Table 12 · p. 20, Table 13 · p. 20

B.1 Mamba-2 residual interventions and geometry diagnostics

Mamba-2 residual interventions produce different outcomes under coordinate exchange and signed clamps, while a dominant residual lens coordinate links stronger ablation effects to recovery costs. Exchange is a reflection rather than guaranteed movement toward a target.

B.1 Mamba-2 residual interventions and geometry diagnostics

  • 10/80 Mamba-2 items yielded the requested answer under residual coordinate exchange, including 7/23 clean-correct items; this is a separate protocol from the workspace clamp.The geometric distinction may explain differing behavior but does not establish the cause of weak signed-clamp response.Source: §B.1 · p. 21, Figure 4 · p. 22
  • Coordinate 2107 is 128 times the median coordinate RMS; projecting it out removes 85% of residual squared norm on average and leaves 0/37 initially correct answers unchanged.The dominant-coordinate pattern is shared with Mamba-3 SISO but is not universal across tested recurrent and hybrid models.Source: §B.1 · p. 21
  • Zeroing lens row 2107 strengthens the ten-direction pursuit ablation, leaving 12/37 initially correct answers unchanged versus 22/37 for the original subspace, but reduces recovery AUC on three families.AUC falls by 0.0322 on multihop, 0.0267 on multilingual, and 0.0436 on order of operations, while rising by 0.0157 on poetry; Figure 4 compares ablations and controls.Source: §B.1 · p. 21, Figure 4 · p. 22

B.2 Exchange algebra, dose and replay checks

  • The 90-item construction-specific figure separates persistent state clamps, matched direction controls, residual coordinate exchange, and pursuit ablations; its denominators must not be pooled.The Mamba-3 replay qualification notes one verbal-trial change and says the paired contrasts do not all pass the historical replay gate.Source: Figure 4 · p. 22, §B.2 · p. 23
  • With independent unit columns, coordinate exchange preserves the sum component and negates the difference component, so its contrast change depends on the current activation rather than always favoring the target.Float64 inversion retains rank two for every paired source–target combination, but correcting float32 truncation still yields only 1/335 Mamba-2 state verbal successes; guarded edits have different trajectories and doses.Source: §B.2 · pp. 22–23

C Corrected paired verbal-report experiment

The corrected verbal-report experiment tests whether edits make a previously low-ranked category target enter the next-token top five. Sign-guarded edits outperform exchange in these trials, but verbal access does not establish downstream relational success, and evaluation scope and output diagnostics qualify interpretation.

C Corrected paired verbal-report experiment

  • A verbal-report trial starts from a clean category answer, selects a different target ranked outside the clean model’s top ten, and scores whether editing brings it into the next-token top five.Residual, state, and simultaneous edits use the same eligible source–target trial within each model, with matched-norm random directions as controls.Source: C Corrected paired verbal-report… · p. 23
  • In the worked Mamba-1 trial, guarded state editing ranks Brazil first, but joint guarding ranks it 11,560th and outputs the fragment “b”; unit exchange leaves France top-ranked.The historical residual clamp also ranks Brazil first, while the state clamp alone leaves it at rank 16.Source: C Corrected paired verbal-report… · p. 24, Table 16 · p. 24

C.1 Full operator comparison and controls

  • All 26 paired verbal G−X contrasts have positive simultaneous intervals, and state-only guarded edits reach top five on 290/446 Mamba-1, 258/335 Mamba-2, 662/728 Mamba-3, and 489/621 Nemotron trials.Three matched-norm random seeds yield far fewer state successes; the Mamba-3 contrasts retain a kernel-sensitivity qualification.Source: §C.1 · p. 25, Table 18 · p. 25, Table 19 · p. 26

D Workspace-suite comparison conventions

  • The corrected workspace comparison uses different block coverage across representations and models, while Nemotron’s high state introspection coexists with little added steering under historical simultaneous clamps.Nemotron’s state clamp alone yields zero two-hop swaps and zero flexible-generalisation redirects; its joint counts are 31/90 and 78/192, respectively.Source: D Workspace-suite comparison conventions · pp. 26–27

D.1 Worked introspection example

  • Persistent state injection places lightning first for Mamba-3 and Nemotron under a supplied affirmative reporting prefix, but this is a next-token prediction, not free generation or spontaneous detection.The corresponding top-1 counts are 49/101 and 93/99.Source: §D.1 · p. 28

D.2 Worked retained-word example

  • After lists of 20, 40, 60, and 80 words, the example’s state lens recovers 6, 6, 5, and 9 previously read words at some evaluated block.The ten-list final-quarter average is 5.92 words, rounded to a 5.9-word lower bound; this measures lens readability, not verbal recall.Source: §D.2 · p. 28

E Paired task outcomes and saved-output diagnostics

E.1 Output classification and original-answer competence

  • On originally correct two-hop trials, guarded edits output the target intermediate on 12/20, 10/25, 23/27, 39/49, and 13/19 trials across Mamba-1, Mamba-2, Mamba-3, Nemotron, and Pythia.Observed edited-concept outputs fail the original task score; original-answer competence does not establish competence on the substituted relation.Source: §E.1 · p. 31, Table 26 · p. 32, Table 27 · p. 33

E.2 Paired interface interactions and examples

Paired edits produce model- and intervention-dependent outcomes: state-only redirections can become edited-concept outputs under joint editing, while equal marginal totals can conceal different successful trials.

E.2 Paired interface interactions and examples

  • In Mamba-3, 15 of 16 state-only successes are new relative to clean, and 11 become intermediate outputs under joint editing.The example is a paired behavioural interaction, not evidence for a particular failure mechanism.Source: §E.2 · p. 35
  • Mamba-1 and Mamba-2 have 1 and 2 state-only successes, all becoming intermediate outputs under joint editing; their joint totals are 4/90 and 6/90.One success in each model already occurs under clean, leaving 0 and 1 new state successes; paired losses do not establish aggregate state-only superiority.Source: §E.2 · p. 35
  • Nemotron has 16/90 requested answers under guarded residual editing but 1/90 under guarded joint editing, despite equal 16-success totals for unit exchange and guarded residual edits.Those residual methods share only 11 successful trials; nine of 15 successful unit joint exchanges become intermediate outputs under guarded joint editing.Source: §E.2 · p. 35
  • In the selected Einstein example, guarded joint editing predicts Newton rather than requested Isaac across all four recurrent models, unlike the clean predictions.This illustration is not a prevalence estimate.Source: §E.2 · p. 35
  • Flexible-generalisation argument outputs are rare for guarded state edits (0/1/2/4 of 192) but common for guarded joint edits (106/51/131/98).Every observed argument output fails the task criterion; the output audit is descriptive and does not isolate a failure mechanism.Source: §E.2 · pp. 35, 37, Figure 5 · p. 36

F Context dependence of Mamba-2 residual Jacobians

Held-out Mamba-2 residual Jacobians vary strongly with context early and align more late, while their shared output structure is asymmetric and does not by itself explain intervention results.

F Context dependence of Mamba-2 residual Jacobians

  • The 200-position analysis covers Mamba-2 residual Jacobians, not recurrent-state Jacobians or comparisons across Mamba generations.Its empirical-mean norm ratio depends on sample size, so the older 32-sample ratio is not a model invariant.Source: F Context dependence of Mamba-2… · p. 37
  • At layer 0, mean pairwise cosine is 0.033 and shared top-16 output directions capture 0.663 of Jacobian energy, versus 0.074 for input directions and approximately 0.006 for random directions.Across layers, same-position maps become more aligned late, reaching mean pairwise cosine 0.537 at layer 56.Source: Figure 6 · p. 37, F Context dependence of Mamba-2… · p. 37
  • At layers 0–24, the fitted lens’s leading rank-one component accounts for 86% to 51% of squared Frobenius norm yet aligns only weakly with individual held-out Jacobians.Agreement with the held-out sample mean emerges only later, and the origin of the early component remains unresolved.Source: F Context dependence of Mamba-2… · p. 37
  • The held-out same-position derivatives and fitted lens average different quantities, and local next-token transport is not the intermediate-recovery metric.No estimator-rescue experiment shows that the geometric mismatch causes the clamp result; geometry is treated as a diagnosis, not a completed mechanism.Source: F Context dependence of Mamba-2… · pp. 37–38

G Shared output geometry and intermediate-concept recovery

A shared output projection captures common residual–state geometry but is an incomplete semantic readout; the section also specifies how state lenses are accumulated and what the numerical audits cover.

G Shared output geometry and intermediate-concept recovery

  • The shared projector pairs the 16 most-aligned directions within each lens’s leading 64 output directions, with rank-16 own-direction and random-projector controls.The construction is predeclared, applied at every source block, and uses no evaluation labels for selection.Source: G Shared output geometry and… · p. 38
  • Across measured cells, shared-16 captures 12–73% of local-map energy versus 0.6–0.8% for random projections, while same-context overlap exceeds cross-context overlap.Same-context top-16 overlap is 0.34–0.80 versus 0.10–0.31 across contexts; these quantities do not establish a context-invariant semantic format.Source: G Shared output geometry and… · p. 39, Table 29 · p. 39
  • The shared rank-16 projection has a median relative AUC loss of 66% across 18 model–family cells, although its difference from random is positive on 13 cells.Against own-top-16 joint readouts it is lower on five cells, higher on one, and has intervals overlapping zero on twelve.Source: G Shared output geometry and… · pp. 39–40, Table 30 · p. 39
  • Removing shared directions changes joint AUC by −0.063 to +0.141; Mamba-2 multihop falls from 0.679 to 0.643, while shared-only recovery is 0.233.At rank 64, shared projection has a median loss of approximately one third; the complement may itself contain shared information, so these results do not identify interface-private concepts.Source: Table 30 · p. 39, G Shared output geometry and… · p. 40
  • Projected edit directions usually perform close to random-projector directions; Mamba-1 flexible-generalisation joint success falls from 30/192 to 7/192 with shared-16.Mamba-3’s 14/192 guarded joint result is matched by own-top-16; the tested edits do not establish that the shared subspace is unused.Source: G Shared output geometry and… · p. 40

H.1 Constructing and accumulating the state lenses

  • The full-path state lens adds future-state propagation to the current-output derivative, whereas the readout-only control omits that recurrence term.The transition is evaluated at captured inputs, and both lens variants map into final residual coordinates for decoding.Source: §H.1 · p. 41
  • The implementation accumulates source-position contributions without storing per-token state Jacobians, using forward weights for decay products.Mamba-1 uses channel- and state-coordinate-specific decays; Mamba-2 uses a scalar decay per head.Source: §H.1 · p. 41

H.2 Numerical checks and verification coverage

  • Automated checks validate covered archived numerical artefacts, while three saved-output audits reproduce archived summaries; neither independently replicates all historical model runs.The audits cover 450 two-hop trial records and edited-concept outputs, but reproducibility is limited to available archived artefacts.Source: §H.2 · p. 42, Table 31 · p. 42

Related papers

Loading 2609.32102v1…