Looped Transformers under the Jacobian Lens: Does the Global Workspace Survive Recurrence?

Wenlong Wang; Fergal Reid

Source-linked AI summary

Looped Transformers under the Jacobian Lens: Does the Global Workspace Survive Recurrence?

Wenlong Wang, Fergal Reid

arXiv:2609.01924v1Sep 1, 2026cs.AI

TL;DR

The paper asks whether recurrence can produce the global-workspace functionality previously identified in feedforward transformers. It extends the Jacobian lens with virtual unrolling and evaluates looped Ouro and Huginn against untied Qwen. Both recurrent models form workspace-like representations, but their access interfaces differ with transport structure and supervision.

Problem
It was unknown whether transformer depth implemented through recurrence, rather than untied layers, supports the same workspace functionality.
Method
The authors apply a virtually unrolled Jacobian lens and the full workspace suite, including readouts and eleven causal experiment families, to Ouro, Huginn, and Qwen.
Results
Both looped models exhibit workspace functionality, but Ouro requires interventions across remaining loops while Huginn’s reads, writes, and ablations operate within an approximately two-recurrence horizon.
Takeaways & Limitations
Recurrence does not prevent a workspace from forming, but it reshapes how workspace content enters, persists, and exits the model.
Takeaways & Limitations
The cross-architecture comparison uses one pretrained checkpoint per family, confounding architecture with training recipe and scale.

Abstract

from arXiv · show

Recent work identifies a mid-depth band of verbalisable, causally potent representations in a standard feedforward transformer --- a functional analogue of a global workspace. Whether the same workspace functionality emerges when depth is implemented through recurrence rather than a stack of distinct layers remains unknown. Looped and depth-recurrent transformers provide a direct test of this question because they reuse the same weights across depth. We extend the Jacobian lens to iterated architectures using a virtual-unrolling adapter. We apply the full workspace suite --- lens fitting, readout, and eleven causal experiment families --- to Ouro-2.6B (48 layers looped 4 times, deeply supervised) and Huginn-0125 (a 4-layer core recurred 16 times, trained for latent reasoning), using Qwen3.6-27B (64 untied layers) as the standard baseline. We find that a workspace forms in the iterated part of each architecture, but that recurrence changes how it can be accessed. Ouro reconstructs workspace content in every loop, and linear transport cannot carry that content across loop boundaries; writes and ablations must therefore span every remaining loop. Huginn carries content forward across all sixteen recurrences, while reads, writes, and ablations act only within a sliding window of roughly two recurrences. Whether newly injected content can be verbalised tracks explicit per-iteration supervision; whether existing content can be steered does not.

1 Introduction

This paper asks whether recurrence can support the same global-workspace functionality found in standard feedforward transformers. It extends the Jacobian-lens workspace suite to looped models and finds a workspace in both, but recurrence changes how it is maintained and accessed.

Motivation: The study tests whether transformer depth implemented through recurrence can support a functional global workspace.The question arises because recurrent processing is central to biological workspace theories, whereas standard transformers use untied layers once.
Models: The experiments compare Ouro-2.6B, Huginn-0125, and Qwen3.6-27B under a shared workspace-analysis protocol.Ouro has 48 layers looped four times, Huginn recurs a 4-layer core 16 times, and Qwen has 64 untied layers.
Method: A virtual-unrolling adapter lets the original Jacobian-lens fitting and readout machinery operate on weight-tied computation graphs.The suite includes lens fitting, transport analysis, readout, and eleven causal experiment families.
Findings: Both looped models exhibit workspace signatures, but their causal interfaces follow iteration structure rather than absolute depth.The workspace content is verbalisable, causally steerable, and selectively engaged, while transport and intervention reach differ by architecture.
Findings: Ouro requires interventions across all remaining loops, whereas Huginn limits reads, writes, and ablations to an approximately two-recurrence transport horizon.Ouro’s loops reconstruct content, while Huginn’s recurrent transport preserves content but supports only local access.
Caveats: Jacobian-lens conclusions require validated horizons, distance-controlled comparisons, and estimator-aware causal controls.Beyond the validated horizon, readouts can fail; uneven lens spacing and single-direction ablations can produce misleading results.

2 Background and related work

The background frames the Jacobian lens as a causal, transport-based probe of workspace-like representations and situates looped transformers within recurrent and parameter-sharing architectures. It also distinguishes this functional analysis from claims about consciousness.

Jacobian lens: The Jacobian lens estimates mean input–output Jacobians between source and target layers, then unembeds transported states for readout.Its token directions support injection, ablation, and clamp-transfer interventions.
Iterated architectures: A looped transformer repeatedly applies a weight-tied block, with loop scope, iteration count, and intermediate supervision defining key design axes.Ouro loops its full stack, while Huginn loops only a 4-layer core between untied prelude and coda blocks.
Training regimes: Ouro trains every loop end with deep supervision, whereas Huginn supervises only the post-coda output and uses truncated backpropagation through recurrence.These training differences are relevant to how intermediate states become decodable.
Comparison: Huginn at r=16 provides a close virtual-depth and compute match to Qwen3.6-27B for comparing tied and untied computation.The virtual depths are 68 and 64 layers, respectively.
Scope: The paper uses workspace theory as a functional characterization and makes no claim about consciousness.Related probing work has primarily used fixed unembedding-based probes, motivating a Jacobian-lens analysis of recurrent models.

3 Method: virtual unrolling

The method virtually unrolls recurrent computation so each firing of a tied block becomes a layer-like node for Jacobian-lens fitting. Because long-range transport can fail, the study fits target-specific lens families and validates distance-controlled readouts and interventions.

Virtual unrolling: Virtual unrolling assigns each firing of a physical block its own virtual-layer position in the computation graph.The adapter exposes these virtual blocks to the original fitting machinery without changing the Jacobian-lens estimator.
Fitting protocol: The lens fits use a shared 1000-prompt protocol, while source–target geometry is the principal model-specific difference.The same corpus and estimator support cross-model comparisons under a common procedure.
Target selection: Using the final virtual state as a universal target fails because Ouro transport dies across loop boundaries and Huginn transport extends only roughly two recurrences.The failure makes a single long-range lens an unreliable readout instrument for both iterated models.
Target selection: The study therefore fits lenses to individual Ouro loop ends and Huginn recurrence ends, rather than one final target.These targets are chosen so each source can reach its lens target through the unrolled graph.
Controls: Readouts and cross-model comparisons are distance-controlled by matching source-to-target offsets, preventing transport attenuation from being mistaken for architectural effects.The causal experiments empirically validate the resulting lens family through interventions along token directions.

4 Transport structure: what the lens artefacts reveal

Transport structure separates bulk-state attenuation from semantic workspace persistence. Ouro’s checkpoints destroy semantic content across loop boundaries, whereas Huginn preserves content but limits access to a short recurrence horizon.

Distance and bulk transport: Aggregate transport weakens with source-to-target distance, so cross-architecture comparisons require matched distances rather than raw transport magnitude.At long range, bulk metrics can collapse even when a specific latent concept remains recoverable.
Distance and bulk transport: 2% top-1 agreement and ∥J∥/d = 0.46 at distance 39 show that Qwen’s aggregate transport is as attenuated as Ouro’s at matched distance.Despite this collapse, the same transport can still carry a specific latent concept.
Checkpointed versus continuous recurrence: Huginn preserves workspace content computed by recurrences 2–3 through all 16 recurrences because its raw latent state crosses boundaries without a supervised decode step.The evidence attributes Ouro’s destruction to the checkpoint interface, not looping itself.
Weight tying and transport symmetry: Matched-offset transports repeat across loops: Ouro reaches mean cosine 0.885, while Huginn’s independently fitted lenses agree at cosine 0.9991.Weight tying creates loop-relative symmetry even though absolute Jacobian similarity decays with virtual distance.
Weight tying and transport symmetry: Ouro permits independent per-loop fitting and composition, whereas continuous-recurrence Huginn requires end-to-end fitting for long-range transport.The practical difference follows from checkpoint factorisation in Ouro and entangled transport across Huginn’s recurrence boundaries.

5 What the iterations compute

Both models iteratively refine representations, but Ouro repeatedly reconstructs workspace content while Huginn carries it forward through recurrent state and re-injected input. Task difficulty determines when answers become readable, and Huginn’s readable region follows each lens target.

5.1 Ouro: iterative refinement, mostly done by loop 3: 92% of loop-3 checkpoint tokens already match the final top-1 output, and the early-exit gate assigns loop-level exit probabilities [0.03, 0.14, 0.34, 0.49].Lens readout, direct checkpoint unembedding, and the model’s internal stopping signal all support convergence mostly by loop 3.
5.1 Ouro: iterative refinement, mostly done by loop 3: Paris is top-1 at the end of Ouro loop 1 and every later loop, so later loops maintain this easy answer rather than compute it.The remaining loop ends preserve an already available representation.
5.1 Ouro: iterative refinement, mostly done by loop 3: Italy becomes readable in Ouro’s loop-2 band, while loops 3–4 maintain it; 47+38 settles only in loop 4.These examples link task difficulty to the number of loops required before the workspace band contains the answer.
5.2 Huginn: sliding-window access: Huginn’s readable region slides with the lens target: each lens resolves content only in the one-to-two recurrences before its target.For example, Tokyo is rank 1 at recurrence steps 2–3 through the r=4 lens and steps 14–15 through the primary lens.
5.2 Huginn: sliding-window access: Tokyo is readable from recurrence 2, French by recurrence 3, and the boot riddle arrives late and equivocally, showing that harder tasks settle later in both architectures.Huginn carries the early answer forward, whereas Ouro destroys and re-derives it at each checkpoint.
5.3 Uniform, weak decodability at every depth: 15–25% of Huginn’s recurrent states match the emitted next token across virtual layers 14–65, while its final coda reaches 63%.Unlike Qwen and mid-loop Ouro, Huginn remains weakly decodable throughout depth, consistent with implicit pressure from variable recurrence training.

6 Workspace geometry across architectures

Independent occupancy metrics recover the workspace’s architecture-specific geometry: a mid-depth band in Qwen, repeated bands in Ouro, and persistent but windowed content in Huginn. Correcting lens-distance artefacts is essential to distinguish model structure from readout effects.

Measurement: The sparse concept-inventory estimator reconstructs each state from non-negative J-lens atoms and counts occupied cells among the first 25 atoms above 0.05 ∥h∥.It requires no ground-truth answer and is cross-checked against the top-8 softmax readout.
Qwen: Qwen has 0 latent slots below L20, 42 latent versus 326 spoken slots in L24–48, and 37 latent versus 112 spoken slots at L55+.The mid-depth band contains the strongest latent occupancy alongside the spoken answer.
Ouro: Ouro’s in-band occupancy exceeds its out-of-band per-layer rate in all four loops, including 22.3 versus 16.6 slots per layer in loop 4.Per-layer rates are required because the out-of-band region contains nearly twice as many layers.
Task examples: The currency riddle shows architecture-specific concept geometry: Ouro shifts from euro toward Italy, while Huginn lacks Italy, Italian, lira, and lire in J-space across 67 layers.Huginn’s five total slots accompany an incorrect Euro answer, whereas Ouro’s Italy occupancy rises from 177 to 390.
Huginn: Huginn’s completed lens tiling makes occupancy nonzero at every virtual layer, confirming persistent concepts rather than accumulation across recurrences.The partial tiling’s rising curve is caused by unequal source-to-target lens distances.
Readability: Decoded next-token probability is strongest for Ouro checkpoints, weakly positive for Huginn, and near zero for Qwen’s mid-band.Qwen reaches 0.019 through the J-lens versus 0.003 through the logit lens at L36.

7 The causal suite: eleven intervention families

The causal suite shows that recurrence preserves workspace function but changes its intervention interfaces. Ouro requires interventions across remaining loops, whereas Huginn’s reads, writes, and ablations are bounded by a roughly two-recurrence horizon.

Writes: Ouro’s flexible-generalisation swap reaches 38% when clamped across loops 2–4, versus 21%, 29%, and 25% for loops 2, 3, and 4 alone.Later loops partially restore the original answer by re-deriving content from raw context.
Writes: Huginn’s last-window write matches all three windows, while the second-to-last window alone achieves 3%, revealing a roughly two-recurrence transport horizon.The intervention horizon bounds writes because lens directions remain valid only near their fitted targets.
Ablation: Concept-direction ablation preserves answers at 72% for Qwen, 71% for Ouro, and 61% for Huginn, while unrelated-direction controls preserve 100% for all three.This specificity supports the interpretation that the removed lens direction carries the targeted concept.
Ablation: Wide ten-direction ablation makes Huginn’s last window as destructive as all windows together, whereas Ouro’s loop-2 removal leaves only 6/42 answers.The wider intervention can remove a state component that later checkpoint reconstruction does not restore.
Verbal report: Huginn reroutes already computed content into its top-5 report at 98% (124/126), but injected content remains unreportable at 0 of 97 concepts.Ouro alone makes newly injected concepts top-1 reportable at 31%, matching its explicit per-iteration supervision.
Interpreting nulls: Null interventions require separating capability floors, readout bounds, and the Jacobian lens’s blind spot for content that is present but not influencing the next token.Huginn’s low no-intervention baselines are 28% and 44%, while capacity measurements are explicitly facts about instrument reach, not memory.

8 Conclusion

Both looped models exhibit a global-workspace signature, but recurrence makes workspace access depend on iteration boundaries and representation interfaces. The models differ in whether content is re-derived or carried forward, and explicit per-iteration supervision predicts verbalisation of injected content.

Both looped models show verbalisable, causally steerable, and selectively engaged workspace content.
Ouro re-derives workspace content at each supervised checkpoint, so writes and ablations must span every remaining loop.Its loop-to-loop process destroys the workspace component in transit and requires continued re-derivation.
Huginn preserves content across sixteen recurrences, but reads, writes, and ablations operate only within a roughly two-recurrence horizon.The last recurrence window is as effective as all windows for writes and ablations.
Only explicit per-iteration supervision predicted verbalisation of newly injected content: Ouro reports 31%, whereas Huginn reports 0/97.Existing-content steering is distinct from verbalising injected content, and untied scale cannot substitute for this supervision.
The comparison is limited by linear Jacobian lenses, single-checkpoint-per-family confounding, and the absence of a systematic sweep over Huginn’s recurrence count.The authors also note that neither released checkpoint establishes workspace storage without continued re-grounding.

A Verification of reported numbers

The verification pipeline re-derives reported quantitative claims from released artefacts and audits tables and inline statistics automatically. Condition-level aggregates remain the main exception because they cannot be independently reconstructed from per-trial records.

232 automated checks audit tables and inline statistics from per-trial records, run logs, and readout dumps.A companion suite adds 132 checks for Appendix E gradient-pursuit artefacts.
Most reported values are recomputed from per-trial records, including percentages and 95% Wilson intervals.
Condition-level aggregates are verified against stored aggregate values but cannot be independently re-derived without per-trial recording.Re-running those experiment families with per-trial records is planned.

B Full causal-suite tables

The full causal-suite tables provide exact counts under the paper’s intervention conventions and clarify several zero, selectivity, and recurrence-window results. Position interventions apply across token positions and listed virtual layers.

Tables 6–8 expand condensed results into exact counts, marking rows whose artefacts contain only aggregates.
Direction interventions apply at every token position and every listed virtual layer, with outcomes read from the final sequence position.The lens itself is position-marginalised; position enters during readout and intervention.
Summoning is a strict no-discrimination result: summoned concepts match foils within 0.01, and asking versus not asking changes nothing.
Only Ouro shows the gated line-count pattern, comparing band readability when the count is directly asked versus never asked.
Numbers never flip on either looped model, remaining 0/48 throughout; Huginn’s last-window ablation matches the all-windows result in every category.Ouro’s direction cosine for numbers is 0.64.

C Robustness to reasoning SFT: Ouro-2.6B-Thinking

Reasoning SFT lowers direct-answer task baselines but leaves Ouro’s workspace interface largely intact. Structural and causal signatures replicate, with language selectivity becoming always-on and ignition becoming sharper.

The SFT changes only the model identifier while preserving the architecture, corpus, hyperparameters, structural analyses, and causal-suite protocol.The Thinking tokenizer’s reasoning prefix is not emitted under the default direct-answer chat template.
Deep supervision and checkpoint-mediated transport persist after SFT, with loop-end top-1 agreement spanning 0.62–0.90 and mean composition cosine 0.962.The trained exit gate remains concentrated on loops 3/4 at 0.37/0.46.
Writes still span every remaining loop, while loop-2-only ablation remains nearly null at 97% kept; introspection remains top-1 31%.The loop-2-only clamp is half-undone by re-derivation at 17%.
SFT lowers raw baselines under direct-answer evaluation: flex-gen falls 66%→59%, verbal report 80%→69%, and probe-swap 43/90→35/90.Intervention effects move by roughly the same amount, preserving their ratio to baseline.
Language selectivity becomes always-on, while ignition sharpens as the in-band 10–90 transition width drops 0.55→0.20.The scrambled-control width also drops from 0.24→0.12.

D Demonstration-metric semantics

The demonstration metrics reconstruct occupancy and probability fields from released readout grids, with explicit depth-window and top-8 truncation semantics. A completed lens tiling removes a spurious monotonic trend and reveals the recurrent plateau structure.

Metric inputs: Each model receives eight chat prompts and a 128-token greedy completion, with J-lens and logit-lens top-8 tokens and probabilities recorded at every virtual-layer, position cell.These grids are the released raw artefact from which every §6 number is computed.
Occupancy: Occupancy counts token occurrences across positions, layers, and eight slots, excluding whitespace-only and non-word strings.Section §6.1 uses full depth rather than the demonstration’s default initial-depth skip.
Probability field: The probability field averages each readout’s probability for the token emitted at the next position across prompts and positions.Because only top-8 entries are stored, omitted tokens contribute zero.
Tiling correction: Completing the lens family with twelve additional lenses flattens the apparent monotonic Huginn occupancy rise into the recurrent plateau.The partial family placed later windows closer to their targets, creating a curve that could be misread as content accumulation.
Demonstration: The animal demonstration uses a prompt about the largest land animal and compares Qwen3.6-27B, Ouro-2.6B, and Huginn-0125 replies.The latent/spoken classification merges singular and plural variants: tree is latent for all models, while food, mammal, vegetation, fruit, and root vary by model.

E The gradient-pursuit concept inventory

The gradient-pursuit inventory reconstructs sparse concept components from Jacobian-lens atoms and supports occupancy, capacity, and ablation analyses. Its results expose estimator-dependent patterns, recurrent transport structure, and substantial but incomplete state coverage.

Inventory construction: The inventory applies greedy non-negative pursuit to Jacobian-lens atoms, refitting selected coefficients until 50 atoms or no positive score remains.Dumps preserve selections, coefficient snapshots at k ∈ {8, 16, 25, 50}, and residual fractions for offline occupancy analysis.
Occupancy geometry: Ouro shows four workspace bands, whereas Huginn shows a flat recurrent-stack plateau with a single step structure under the occupancy analysis.The same geometry appears under both estimators.
Estimator comparison: The estimator comparison shows that top-8 occupancy patterns can diverge from inventory patterns, including an animal-prompt latent-class thinning that reflects eight-slot competition rather than state absence.The sport prompt retains the layer-level commitment identified as real on Qwen.
Coverage and capacity: At k = 50, pursuit explains 4.3% of Qwen’s, 10.2% of Huginn’s, and 21.5% of Ouro’s state variance.At k = 25, the captured fraction is 91% for Qwen, 78% for Ouro, and 63% for Huginn, so the cap is not a saturation point for recurrent models.
Downstream analyses: The coefficient threshold distinguishes vanishing semantic coefficients from merely selected atoms, while capacity and subspace-ablation protocols test held concepts and causal removal.Capacity counts a word as held when a selected atom at any read layer matches it; ablation removes projection onto ten pursuit directions.

F Worked examples

Worked examples contrast readout and intervention behaviour across the three models, while adapter diagnostics identify recurrence-specific implementation hazards. The examples connect hidden concepts, verbalisation, and reliable virtual-depth accounting.

Readout cells: A hidden sport is sharp in Qwen’s J-lens at mid-band L42 with probability 0.54, while its logit-lens reading contains punctuation fragments at approximately 10^-3.This illustrates mid-stack content that requires transport to become token-decodable.
Intervention record: The introspection intervention injects a concept along its lens direction and tracks that concept’s answer-distribution rank across injection strengths.The example uses lightning across all three models.
Verbalisation split: Ouro’s injected concept improves monotonically into reportable range, Qwen’s improves substantially but remains two orders below that range, and Huginn’s responds initially before plateauing.In Huginn, the state moves measurably but the injected concept does not reach the output as a word.
Adapter hazards: Huginn Jacobians become identically zero if recurrence steps are passed through the no-grad phase, so the adapter requests (0, r) steps and checks non-zero Jacobians.This is a recurrence-specific failure mode that can silently corrupt lens fitting.
Determinism and accounting: Seeded recurrent-state initialization and disabled test-time noise provide bit-exact deterministic forwards, preventing lenses from averaging unmatched computations.Gradient checkpointing remains disabled because backward recomputation desynchronizes virtual-layer counters.
Runtime compatibility: The models require compatibility patches when remote code written for transformers 4.x is run under version 5.14.Ouro needs three load-time patches, whereas Huginn needs exactly one.