Shared Weights, Selected Computations: How Looped Transformers Route What Each Loop Does

1 INTRODUCTION

• Final-answer-only supervision does not require a particular intermediate sequence, so shared weights alone cannot specify each loop’s operation.The hidden state entering the shared parameters changes as computation proceeds.Source: §1 · pp. 1–2
• Prior work shows that shared parameters can support reusable algorithms, but leaves open whether an entering state can causally select a frozen loop’s next computation.Studies describe learned trajectories, including different progress rates, without establishing how entering-state control is implemented.Source: §1 · p. 2
• The paper tests this question in graph walks, where native readouts can advance by different numbers of steps or remain at a reached target.A learned trajectory therefore need not follow a fixed one-loop-one-step rule.Source: §1 · pp. 2–3
• A learned linear layer steers a frozen loop toward different transitions, and attention-pattern patching transfers routing effects while preserving the receiving run’s values.Across matched graph-model pairs, intermediate supervision changes which transitions the controller can induce.Source: Figure 1 · p. 2, §1 · p. 3

Recurrent computation

• Prior work shows looped Transformers can execute programs, learn iterative algorithms, improve length generalization, and use additional loops for reasoning, without specifying each loop’s operation after training.The cited work includes Giannou et al. (2023), Yang et al. (2024), Gatmiry et al. (2024), Fan et al. (2025), Saunshi et al. (2025), Geiping et al. (2025), and Zhu et al. (2025).Source: §2 · p. 3

Computation learned across loops

• Studies find that training budgets select different rates of progress, while cyclic trajectories and readout-dependent interpretations complicate direct observation of loop computation.The cited studies are Zhang et al. (2026), Blayney et al. (2026), and Lu et al. (2025).Source: §2 · p. 3

Supervision of recurrent states

• Training objectives shape recurrent trajectories and intermediate information, yet output supervision does not fully specify internal computation or control hidden-state scale.The cited studies are Fan et al. (2026), Popescu et al. (2026), and Sharma & Vu (2026).Source: §2 · p. 3

State steering and causal interventions

• The paper combines affine state steering with activation patching to test whether entering states select learned computation and whether attention routing implements that selection.Prior work uses steering to modify hidden states and patching to test component contributions; the cited studies are Turner et al. (2023), Singh et al. (2024), and Wang et al. (2023).Source: §2 · p. 3

3 PHENOMENON: TASK PROGRESS VARIES ACROSS LOOPS

• The experiment reads out predictions at intermediate loops to observe how computation progresses on a graph-walk task.The models repeat a two-layer Transformer block for six or eight loops on a requested eight-step path, with only the final answer supervised.Source: §3 · pp. 3–4
• Figure 2 plots prediction frequency by loop, marks the training loop budget with a red dashed line, and marks each loop’s unique most frequent prediction with a white dot.The visualizations extend readouts beyond the training loop budget.Source: Figure 2 · p. 4, §3 · p. 4
• Within a model, readouts can advance mainly one or two nodes per loop, move at irregular speeds, and eventually stabilize at the final answer.D8L8 trajectories differ, while D8L6 fits progress into six loops; this shows that loops in one shared-weight model can perform different algorithms.Source: §3 · p. 4

4.1 STATE STEERING WITH A FROZEN BACKBONE

• The controller applies a learned linear map J to each token’s state before the shared block F, while the backbone remains frozen.For graph experiments, J has a diagonal term, a rank-48 term, and a bias; it cannot aggregate graph information across token positions on its own.Source: §4.1 · pp. 4–5
• The fitted map is evaluated on out-of-distribution graphs unseen during both backbone and map training to test generalization and state-based redirection.This evaluates whether modifying information already represented in each token can redirect computation.Source: §4.1 · p. 5
• With one additional loop, steering raises one-hop accuracy from 48.2% to 99.6%, while two-hop accuracy reaches 98.2% with Jtwo on held-out graphs.Both maps select their intended targets while the shared backbone weights remain frozen.Source: §4.1 · p. 5

4.2 COMPOSING CONTROLLER J

• Two-loop reuse reaches the two-hop target in 93.6% of examples with the one-hop map applied twice; one-hop-then-two-hop and reverse order reach the three-hop target at 65.3% and 74.5%.Omitting the second map reduces accuracy to 48.3%, and reversing map order changes accuracy.Source: §4.2 · p. 6
• Longer reuse preserves some control but degrades: at the third call, repeated one-hop and two-hop maps reach 56.3% and 23.6% accuracy, and mixed control is 10.1% at the eighth.The independent evaluation uses 512 new held-out graphs and scores exact match to the cumulative target.Source: Figure 4 · p. 5, §4.2 · p. 6

5 MECHANISM: ATTENTION ROUTING MEDIATES STATE-DEPENDENT COMPUTATION

• The experiments test whether attention patterns route retrieved content, reproduce linear-map steering, and switch transitions between one-hop and two-hop runs.The mechanism tests include D8L6 and Ouro, with different activation sets and patching scopes.Source: §5 · p. 6

5.1 PATTERN AND OUTPUT PATCHES SELECT DIFFERENT ANSWERS

• Pattern patches select content using the receiving run’s values, while output patches transfer the donor run’s output.Pattern patches produce E in 77.8% of D8L6 cases and 71.1% of Ouro cases; output patches produce D in 80.9% and 93.8%, respectively.Source: §5.1 · pp. 6–7, Figure 5 · p. 7

5.2 ATTENTION PATTERNS CARRY STATE-STEERING EFFECTS

• Patching steered attention patterns into unsteered D8L6 runs raises one-hop accuracy from 39.2% to 61.6%; the reverse patch lowers it from 99.7% to 68.1%.The effect on unsteered runs varies across backbones; for Ouro, steered patterns yield 248/256 correct answers, close to 251/256 under full steering.Source: §5.2 · p. 7

5.3 ATTENTION ROUTING SWITCHES THE NEXT TARGET

• Second-layer attention-pattern patches switch the predicted target in both directions, and restoring selected head outputs repairs most eligible errors.The target switches appear in all three selected backbones; restoring comparison-head outputs repairs 0.5% of eligible D8L6 errors and none of the eligible Ouro errors.Source: §5.3 · p. 8, Figure 6 · p. 8

• One-hop accuracy is 100% for stepwise models and 66.5% for final-only models, while two-hop accuracy is 50.5% for final-only and 13.2% for stepwise models.Final-only models have higher two-hop accuracy in all five matched pairs; the Figure 7 caption identifies answer colors relative to u = f 8.Source: §6 · p. 8, Figure 7 · p. 9
• On Ouro, stepwise training predicts one deletion per loop, while final-only models answer 255/256 requests correctly by loop 3.Source: §6 · p. 9
• 98.0% held-out accuracy for final-only plus J versus 7.8% for stepwise plus J on Ouro deletion depths 5–8; both unsteered models score zero.The shared rank-128 residual affine map is fitted on deletions 1–8 and used with four loops.Source: §6 · p. 9, Figure 8 · p. 9
• The authors interpret the supervision-dependent steering results as evidence that fitted maps rely on algorithms learned by the backbone.Source: §6 · p. 9

7 CONCLUSION

• A shared block can be redirected toward different semantic transitions by modifying only its entering state, with attention routing mediating this control.Attention-pattern patches can recover steering effects and switch the target while preserving the receiving run’s values.Source: §7 · p. 9
• Which transitions are controllable depends on what the backbone has learned, supporting steering as redirection rather than an arbitrary new computation.Source: §7 · p. 9
• Controllers that work over one or two loops degrade under longer composition; successful long-horizon reasoning requires states compatible with repeated routing.Source: §7 · p. 10

AI USE STATEMENT

• The authors report using ChatGPT and Codex for code implementation and debugging, literature search, and manuscript refinement, with manual review.Source: AI USE STATEMENT · p. 10

ETHICS STATEMENT

• The experiments use synthetic graph, parity, and rule-based language tasks and do not evaluate the safety of deploying steered models.Source: ETHICS STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

• Reproduction materials are available in the repository linked below the abstract.Source: REPRODUCIBILITY STATEMENT · p. 10

A DEPTH EXTRAPOLATION

• 45.05% to 73.73% accuracy over lengths 101–500 with steering reverses to 47.55% versus 36.40% without steering at measured lengths 505–1000.The longer-range averages cover 100 sampled lengths strictly above 500.Source: A DEPTH EXTRAPOLATION · p. 12, §A.1.3 · p. 13
• On an independent held-out cycle subset, graph-continuation accuracy fell from 97.42% at loops 9–16 to 35.11% at loops 17–32, showing within-range accuracy does not ensure accuracy at greater depth.Source: A DEPTH EXTRAPOLATION · p. 12

A.1.2 MODEL AND STEERING CONFIGURATION

• Parity backbone training covers lengths 1–20, map training covers lengths 20–40, and the same map acts before loops 2 through n without intermediate partial-parity targets.Evaluation at longer lengths keeps the map fixed while increasing input length and requested loop count.Source: §A.1.2 · p. 13

A.1.4 READOUT TIMING ACROSS BACKBONES

• For seed 2, steering reduces absolute readout drift from 0.96 to 0.18 loops per 100 tokens over lengths 41–490, but the timing effect varies across seeds.Seed 1 gains slightly in mean accuracy while its drift and residual error both increase.Source: §A.1.4 · p. 14, Table 1 · p. 14

A.2.1 TASK DEFINITION AND INPUT FORMAT

• The graph-continuation task keeps the query depth fixed at 8 while scoring the node reached after t graph edges from the original start.The map is trained on successive nodes through loop 16 using steered-run states, then held fixed for evaluation beyond loop 16.Source: §A.2.1 · p. 14, §A.2.2 · p. 15

A.2.2 MODEL AND STEERING CONFIGURATION

• The graph continuation study uses twelve separate eight-node D8L8 backbones, and maps act before every loop, including evaluation loops beyond 16.The evaluation uses a broad sample of 8,192 graphs; a separate protocol reports results on graphs excluded from training.Source: §A.2.2 · p. 15, §A.2.3 · p. 15

A.2.4 VARIATION ACROSS BACKBONES AND EVALUATION SETS

• The broad evaluation sample was not checked against historical training exclusions, so its results are distinct from the separate training-excluded evaluation.Both populations show lower accuracy beyond fitted loops, though they average over different graph structures.Source: §A.2.4 · p. 15
• Across twelve backbones, steered accuracy ranges from 75.01% to 100.00% over loops 9–16 and from 34.73% to 54.56% over loops 17–32.Each value averages the two fixed maps, and every backbone has lower mean accuracy beyond the map-fitting range.Source: §A.2.4 · p. 15
• 97.42% accuracy over loops 9–16 on 78 held-out single-cycle graphs falls to 35.11% over loops 17–32; all twelve backbones lose accuracy beyond map training.Unsteered accuracy is 12.5% in both ranges; on an eight-node cycle, retaining one endpoint yields that score without continuing the walk.Source: §A.2.4 · p. 15

B.1 READOUT SUMMARY

• Native readout increments compare the unique most frequent decoded path positions across the full evaluation set, not transitions for individual examples.Ties are left unresolved; a zero increment (“Hold”) is zero decoded displacement modulo ten, not evidence that the hidden state performs no computation.Source: §B.1 · p. 19

C.1 TASK DEFINITION AND INPUT FORMAT

• The graph task supplies the graph, start node, and requested depth in the input, and predicts the node reached after the requested number of edges.For target selection, the request remains fixed at eight steps; maps are applied to the loop-6 state before one frozen loop.Source: §C.1 · p. 20, §C.2 · p. 20

C.2 MODEL AND STEERING CONFIGURATION

• The D8L6 target-selection maps are rank-48 diagonal-plus-low-rank affine maps trained while the backbone remains frozen.One map targets the successor of the eight-step answer and the other its second successor.Source: §C.1 · p. 20, §C.2 · p. 20

C.4.1 RESTORING HEAD OUTPUTS RECOVERS ANSWERS

• In seed 6, selected-head restoration repairs 60.1% of disrupted answers versus 0.9% for controls; in Ouro, it repairs 249 of 249 versus 0 for controls.Restoring all affected outputs repairs every disrupted answer in both models.Source: §C.4.1 · p. 22

C.4.2 PATCHING ONE-HOP AND TWO-HOP PATTERNS

• Attention-pattern patching compares one-hop and two-hop steering while retaining each run’s values, and same-run replacements preserve predictions.Pattern and output patching use the same eligible pairs within each fit; the supplied passages give no target-switching accuracy result.Source: §C.3 · p. 21, §C.4.2 · p. 23

C.4.3 COMPOSITION OF FROZEN TARGET-SPECIFIC MAPS

• Two-step composition depends on map order: one–one reaches 93.64% accuracy, while two–two reaches 68.12%.These evaluations reuse frozen maps without further training.Source: Table 5 · p. 23, §C.4.3 · p. 23

C.4.4 EIGHT-STEP CONTROLLER REUSE

• On the fixed 3,200-instance set, eighth-call exact match is approximately 10% across sequence families; independent evaluation finds 9.05%–11.57% for mixed sequences across five backbones.These repeated-control tests do not establish sixteen-hop extrapolation with distinct nodes at every step.Source: Table 6 · p. 24, §C.4.4 · p. 24, Table 8 · p. 24
• For seed 6 mixed sequences, omitting the second map reduces accuracy from 78.89% to 32.93%, while reading directly after that map yields 0.66%.At longer lengths, continuous control no longer maintains an advantage.Source: Table 9 · p. 25, §C.4.4 · p. 25

C.4.5 SELECTED BACKBONES AND AGGREGATION

• Across three selected backbones, restoring selected heads recovers 74.0% of disrupted answers, versus 0.5% for comparison heads; restoring all four heads recovers 100%.Recovery is measured only on answers correct before query replacement and incorrect afterward.Source: §C.4.5 · p. 26

D.1 TASK DEFINITION AND INPUT FORMAT

• Ouro’s letter-walk task encodes a ten-person cycle in shuffled text, asks for a specified walk depth, and expects only the final person’s name.Its backbone is fine-tuned on 1–4-step requests with four loops, then frozen while a dense affine map is fit on 1–8-step requests; mechanism evaluations request eight steps.Source: §D.1 · p. 26, §D.2 · p. 26

D.3 EVALUATION AND ATTENTION INTERVENTIONS

• Ouro’s intervention evaluation uses fixed backbone and map weights, with selected heads identified at loop 4 and separate sets for pattern-versus-output, restoration, and steering-pattern tests.Pattern-versus-output and restoration tests include layer-matched comparison heads; restoration evaluates recovery only for answers disrupted by query replacement.Source: §D.3 · p. 27
• Pairs distinguish run-2 answers, run-1 answers, and the run-1-starting-person answer on the second graph; reference runs define the latter without accuracy filtering or one-step-per-loop assumptions.Source: §D.3 · p. 27
• The independent steering-pattern evaluation used 256 new graph pairs (512 identities) on ten-person cycles, with the backbone, map, heads, and loop-4 intervention sites fixed.Source: §D.3 · p. 27
• Steered and unsteered pattern swaps were compared with value-only, unrelated-graph, loop-2, and layer-matched disjoint-head controls.Patches affected prompt processing only; generation retained each run's steering schedule, and first-token and full-name correctness were scored.Source: §D.3 · pp. 27–28
• 0/256 accuracy gain followed explicit same-run pattern reconstruction, which differed from BF16 attention by at most 0.125 logits; output replacement and restoration had zero maximum logit error.Source: §D.3 · p. 28
• Full-name correctness was compared with first-token correctness, truncated generations counted incorrect, and graph absence from backbone training could not be established.Ouro figures report Wilson confidence intervals.Source: §D.3 · p. 28

D.4 ADDITIONAL RESULTS

• Selected outputs recover 100.0% of eligible broken answers, with a Wilson 95% interval of 98.5–100.0%.Full-name and first-token correctness agree, no generation is truncated, and restoring all affected outputs reconstructs the clean logits.Source: §D.4 · p. 28, Table 10 · p. 28
• 182/256 pattern patches (71.1%) produce the rerouted answer, while output patches produce the run-1 answer in 240/256 cases.Pattern patching produces the run-1 answer in 0/256 cases; output patching produces the rerouted answer in 7/256, distinguishing rerouting from transferring retrieved output.Source: §D.4 · p. 28, Table 11 · p. 28
• 248/256 (96.9%) answers are correct under the fixed pattern patch on new pairs, versus 251/256 under full steering; reverse and comparison interventions score zero.The paired accuracy gap is 1.17 percentage points (95% bootstrap interval −0.39 to 2.73), which does not establish equivalence or replication across independently trained Ouro backbones.Source: §D.4 · p. 29, Table 12 · p. 29

E MATCHED SUPERVISION IN GRAPH WALK

• The matched-supervision experiment adds intermediate losses while holding the query and backbone training stream fixed, using ten-node D8L8 models and separate controls.Source: E MATCHED SUPERVISION IN GRAPH WALK · p. 29

E.1 PAIRED BACKBONE TRAINING

• Five paired eight-loop models share initialization and training-token streams; stepwise training adds intermediate losses without normalizing total loss or gradient scale.Both regimes use 20,000 AdamW updates, and gradients pass through all eight loops.Source: §E.1 · p. 29

E.2 MAPS AND EVALUATION

• On frozen backbones, separately fitted maps target the endpoint, one-hop, or two-hop transition; rank-48 maps are evaluated using disjoint graph pools.Test instances are shared across conditions, and results average two fits within each of five backbones.Source: §E.2 · pp. 29–30

E.3 RESULTS ACROSS PAIRED SEEDS

• Without a map, all five final-only backbones retain u, while all five stepwise backbones predict fG(u) on the scoring population.Each backbone’s stay and one-hop maps reached 100% accuracy.Source: §E.3 · p. 30
• 100% accuracy for every stay map and every stepwise one-hop map; two-hop accuracy was 50.5 ± 15.2% for final-only models versus 13.2 ± 6.8% for stepwise models.Final-only two-hop accuracy was higher in every pair; the tested map family and fitting procedure do not establish the limits of other state interventions.Source: §E.3 · pp. 30–31, Table 13 · p. 30, Figure 15 · p. 30

F.1 TASK DEFINITION AND INPUT FORMAT

• Ouro’s task deletes the leftmost adjacent antonym pair at each step; the answer is the pair removed at the requested deletion, in left-to-right order.Training and evaluation use 24-word inputs, with pair orientations and nesting varied.Source: §F.1 · p. 31

F.2 MODEL AND STEERING CONFIGURATION

• Final-only Ouro training supervises a requested deletion at loop 4, whereas stepwise training supervises each deletion at its corresponding loop across four loops.Each frozen backbone receives its own map shared across all four loops; map fitting supervises the requested pair at the fixed loop-4 exit.Source: §F.2 · pp. 31–32

F.3 EVALUATION PROTOCOL

• At deletion depths 5–8, final-only plus J answered 251/256 requests correctly, while stepwise plus J answered 20/256; both native models scored zero.The evaluation used 64 held-out sequence structures per depth across four conditions.Source: §F.3 · p. 32
• The deeper-deletion result compares one backbone and one map per training strategy, and low stepwise accuracy does not rule out other state interventions.Depths 5–8 were beyond backbone training but included in map training.Source: §F.3 · p. 32

H.1 CONTROL OF CONTINUED COMPUTATION IN QWEN3-8B

• The Qwen experiment tests five iterative task families using a frozen 36-layer stack and one shared affine map trained for requested counts k = 1–8.Evaluation branches share controlled computation through h4 before comparing continued J + F, F alone, J alone, or immediate readout.Source: §H.1 · p. 34
• 574/640 long requests were answered correctly by continued J + F, versus 183/640 for F alone, 200/640 for J alone, and 164/640 when stopping at h4.The full controlled execution reached 100% on successor, weekday, doubling, and Fibonacci tasks, and 48.44% on Collatz.Source: §H.1 · p. 34, Table 16 · p. 34

H.2 CONTROLLED RELATION COMPOSITION IN A SYNTHETIC KNOWLEDGE GRAPH

• The graph has 64 entities and 16 relations, each represented by a fixed permutation; the target is obtained by composing the relation transformations from the starting entity.The frozen backbone is a shared two-layer Transformer trained on lengths 1–3 with aligned intermediate-entity supervision.Source: §H.2 · p. 35
• A shared token-wise affine map is applied between frozen-backbone calls, while the no-map comparison uses the same input, readout, and number of backbone calls.The map is identity-initialized and reused at every boundary.Source: §H.2 · p. 35
• Across lengths 4–16, the affine map raises target accuracy from 2.53% to 99.92%.Accuracy with J is 99.61% at length 17, 97.75% at length 18, and 85.74% at length 19; the average is 42.87% over lengths 17–24 and 1.65% over lengths 25–32.Source: §H.2 · p. 35, Table 17 · p. 36
• The experiments provide additional evidence that shared affine interventions at loop boundaries can improve controlled execution with frozen backbones.The evidence spans the synthetic relation executor and a task-adapted language model.Source: §H.2 · p. 36