PaperlayerView exact revision on arXiv

Source-linked AI summary

Agreement rather than normalisation keeps depth reusable in looped language models

Ivan Viakhirev, Kirill Borodin, Maxim Abramov, Grach Mkrtchian

arXiv:2610.31943v1cs.LGcs.CL

TL;DR

Looped language models are useful only if depth remains reusable beyond the training budget, but whether this depends on normalising between loop steps or matching training and inference topologies is open. The paper tests both released-weight interventions and scratch-trained models with topology fixed before training. The results point to topology agreement, rather than boundary normalisation itself, as the condition associated with reusable depth.

  • Problem

    It is unclear which architectural feature keeps loop steps useful beyond the trained budget, a condition needed for looping to save parameters while providing additional inference-time depth.

    Source: §1 · p. 1

  • Method

    The study alters loop-boundary normalisation on released weights and compares scratch-trained models whose loop topology is set before training.

    Source: §1 · p. 1, §3.1 · p. 3

  • Results

    Scratch-trained models without boundary normalisation score 0.997, 0.981, and 0.932 at two, four, and eight times the budget, while the normalised configuration scores 0.772, 0.618, and 0.584 at those depths.

    Source: §4.3 · p. 7

  • Takeaways & Limitations

    The results support agreement between the trained and inference loop topologies, rather than boundary normalisation itself, as necessary for depth reuse.

    Source: §5 · p. 8, §7 · p. 9

    Causal evidence comes from 19–76M-parameter synthetic-task models; released models provide scale but do not permit varying their training topology.

    Source: §6 · p. 9

Abstract

from arXiv · show

Looped language models apply one block of layers repeatedly, and the design pays off only if depth is reusable: loop steps beyond the trained budget must remain usable at inference. We ask whether that rests on normalising the state between loop steps, as the released Ouro models do, or on agreement between the loop a model is trained with and the loop it is run with. On released weights this loop-boundary normalisation can be removed without adding parameters and with the readout path bit-identical, and the operation is an identity at one loop step, which serves as an implementation check. Removing it takes Ouro-1.4B and Ouro-2.6B to zero accuracy by the trained budget on a balanced binary task (chance 0.500). State-norm growth, the failure that prior accounts predict, is not the cause: the norm stays within a factor of 2.021 while the head commits to a single wrong token. Yet in 19M-parameter models trained from scratch on two synthetic tasks, 16 seeds per configuration, training without the normalisation is never worse than training with it and extrapolates better: the two are closest at the trained depth and separate on both sides of it (Cliff's delta up to 0.867 beyond the budget). Both results follow if agreement, and not the normalisation itself, is what keeps depth reusable. The damage sits in the readout together with the last layer of the block: in the models trained from scratch, refitting the head over the frozen modified loop recovers a third of the lost accuracy and plateaus, while refitting the last block layer with it recovers all of it. Finally, comparing looped with stacked depth at an equal inference loop count is confounded by unequal overshoot of the trained budget; at matched overshoot the advantage of looping vanishes on one task and shrinks on the other.

1 INTRODUCTION

The paper asks what makes loop depth reusable beyond the training budget, testing whether this depends on loop-boundary normalisation or agreement between training and inference topologies. Its two-part study finds that agreement matters, while boundary normalisation is not needed when the topology is fixed before training.

Released-weight intervention

  • On released weights, restricting Ouro’s model-level normalisation to the final loop step leaves the readout path bit-identical, changes only the returning state, adds no parameters, and is an identity at r = 1.The one-step identity is used as an implementation check.Source: §1 · pp. 1–2

Main findings

  • Without boundary normalisation, scratch-trained models are never worse and extrapolate better away from the trained depth, while removing it from released models breaks them within a few loop steps.The authors interpret the results as showing that topology agreement is necessary for reusable depth, whereas boundary normalisation is not beneficial when agreement holds.Source: §1 · p. 2

Study design

  • The study tests the released-weight intervention and trains models from scratch with topology fixed before training, using two task families and 16 seeds per configuration.The scratch-trained comparison uses training seed as the unit of analysis and rank tests over seeds.Source: §1 · p. 2

Mechanism and recovery

  • The readout and last block layer jointly account for the damage: refitting the head over a frozen loop recovers a third of the loss, while refitting the last layer with it recovers all of it.The reported measurements also show state scale barely moves while accuracy collapses, and that scratch-trained models without boundary normalisation perform better away from trained depth.Source: §1 · p. 2

Depth comparison

  • At matched budget overshoot K/r0, which equalises inference compute, looping’s advantage is undetectable on circuit and shrinks without closing on hops.Equal absolute loop count K is confounded by unequal overshoot of the trained budget.Source: §1 · p. 2

2 RELATED WORK

Prior work studies looped models, normalisation within recurrence, and loop-versus-stack trade-offs from several angles. This paper distinguishes its interventions by moving loop-boundary normalisation while holding the readout fixed, and by comparing inference-time depth at matched loop counts.

Looped language models

  • Recent language-model-scale looped systems include Huginn, Ouro, and retrofitted recurrence, alongside work on elastic depth, wrappers, memory-decoupled implementations, scaling, and looping as a substitute for chain of thought.Earlier work includes the Universal Transformer and results on simulating arbitrary programs with a looped transformer.Source: §2 · p. 2

Normalisation and extrapolation

  • Prior analyses study normalisation inside loops as gain control, a condition for reachable fixed points, placement effects, and convergence; other work examines out-of-distribution variance and readout or halting mechanisms.The cited work does not hold the readout fixed while moving loop-boundary normalisation on released weights.Source: §2 · pp. 2–3

Nearest neighbour

  • Sharma & Vu (2026) rescale Ouro-1.4B’s final hidden state while holding loop topology fixed; this paper instead changes loop-boundary normalisation while holding the readout fixed.The interventions cover orthogonal axes of the design space, and the cited account expects inter-loop normalisation to avoid failure by construction.Source: §2 · p. 3

Looped versus stacked depth

  • Prior scaling-law work compares recurrence and stack depth at fixed training compute or effective depth, whereas this study compares placements against the loop count used at inference.Schwethelm et al. (2026) report a recurrence-equivalence exponent of 0.46 in an iso-depth pretraining sweep.Source: §2 · p. 3

In-context lookup

3 SETUP

The setup separates released checkpoints, which provide scale, from smaller scratch-trained models, which permit topology to be varied before training. It defines the intervention, tasks, and evaluation rules used to compare loop-depth reuse.

3.1 MODELS

  • Released checkpoints provide scale, while scratch-trained models range from 19M parameters at six layers to 76M at twenty-four, enabling topology to be set before training.Sixteen runs per configuration were not feasible at 3B parameters, so the two model groups answer different questions; r and K denote inference loop counts, and r0 is the trained budget.Source: §3.1 · p. 3

3.2 THE AXIS: LOOP-BOUNDARY NORMALISATION

  • Loop-boundary normalisation acts on the state leaving one iteration and entering the next, distinct from normalisations inside the block.For Ouro the intervention restricts model-level normalisation to the final step; the scratch-trained version uses a dedicated between-iteration normalisation, skipped on the final step, and each operation is an identity at one loop step.Source: §3.2 · pp. 3–4
  • Instrumentation distinguishes modules called once per loop step from those called only on the readout path; “readout only” does not mean no normalisation acts there.That qualification matters only for the Huginn direction.Source: §3.2 · pp. 3–4

3.3 TASKS

  • The synthetic task families include Boolean circuits, pointer hops through an in-context map, and parity, with exact instance depth and independently controlled width.Binary families are balanced 50/50 in every stratum; hops has 26 possible answers.Source: §3.3 · p. 4

3.4 METRICS AND STATISTICS

  • Accuracy uses the full-vocabulary arg-max against the gold token and an explicit floor; the analysis uses paired-instance tests for released weights and seed-level tests for scratch-trained models.A pair enters analysis only if peak accuracy reaches 0.30 and exceeds its floor by more than 0.05; the seed-level normalisation comparison uses 16 seeds per configuration.Source: §3.4 · p. 4

4 RESULTS

Removing loop-boundary normalisation breaks released Ouro, but models trained from scratch without it maintain accuracy across a wider range of loop depths. At matched overshoot, the advantage of looping depends on the task.

4.1 THE REARRANGEMENT IS ASYMMETRIC

  • The identity check gives a 0.0000 accuracy and entropy difference at r = 1; the reverse intervention on Huginn is not informative because it fails the competence gate before manipulation.The identity check covers nine zero-shot arm-and-family pairs and four few-shot pairs.Source: §4.1 · pp. 4–5
  • At the trained budget on circuit, removing boundary normalisation takes Ouro-2.6B to zero accuracy, while the unmodified model peaks at 0.642.The rearrangement is an identity at r = 1; Ouro-1.4B also reaches zero by r = 3, and the removed-normalisation Ouro-1.4B returns a single token at r = 32 on hops.Source: §4.1 · pp. 4–5, Figure 1 · p. 5

4.2 THE READOUT COLLAPSES; THE STATE SCALE DOES NOT

  • The readout first becomes nearly uniform, then commits to a wrong token, while state-norm growth stays below a factor of 2.021.At the trained budget, a linear probe on the modified states is at chance, and applying normalisation every second step leaves the model at chance while sparser application takes accuracy to zero.Source: §4.2 · pp. 5–6, Figure 2 · p. 6

4.3 TRAINED FROM SCRATCH: THE NORMALISATION HURTS AWAY FROM THE TRAINED DEPTH

  • At four times the budget on circuit, the unnormalised arm scores 0.981 versus 0.618 with normalisation; on hops, the gap is no longer significant at eight times the budget.Across both tasks, normalisation hurts away from the trained depth, with the size of the beyond-budget effect depending on the task.Source: Figure 3 · p. 7, §4.3 · p. 7
  • Agreement between training and inference topologies is necessary but not sufficient: both scratch-trained configurations use their trained loop, yet the normalised configuration degrades away from its trained depth.The unnormalised configuration grows in state norm by a factor of 4.84, versus 1.45 with normalisation; the proposed damping explanation is an interpretation, not a measured mechanism.Source: §4.3 · pp. 7–8

4.4 THE DAMAGE SITS IN THE LAST BLOCK LAYER AND THE HEAD

  • Refitting the head over the frozen modified loop recovers about a third of lost accuracy and plateaus; also refitting the last block layer recovers all of it.The last recurrent-block layer is shared across loop steps; the authors therefore locate the damage in that layer together with the head.Source: §4.4 · p. 8

4.5 LOOPED AGAINST STACKED DEPTH AT MATCHED OVERSHOOT

  • At matched overshoot on circuit, looped and stacked configurations are within one standard deviation and no pairwise test rejects; on hops, a gap remains but shrinks.The hops comparisons are exploratory, and the one significant difference across the eight comparisons does not survive Holm correction.Source: §4.5 · p. 8

5 DISCUSSION

The discussion frames reusable depth as a contract between a loop and its readout, with extrapolation depending on both agreement and how the loop state evolves. It recommends matching overshoot when comparing looped and stacked depth.

5 DISCUSSION

  • Extra loop steps are usable only while the readout is calibrated for the loop used in training, and the loop without boundary normalisation extrapolates further when both configurations match their training loops.The discussion locates the contract in the last block layer and the head that reads it.Source: §5 · pp. 8–9
  • Compare extrapolation against budget overshoot K/r0, because equal absolute loop counts confound comparisons when trained budgets differ.A modification inside a released model’s loop should be treated as invalidating its readout until measured, even if state scale remains intact.Source: §5 · p. 9

6 LIMITATIONS

The causal evidence comes from smaller models trained from scratch, while released models provide scale but cannot vary training topology. The evaluation and safety claims are also bounded by specific data, task, and operation choices.

6 LIMITATIONS

  • Causal claims come from models of 19 to 76M parameters on two synthetic task families; released models carry scale but cannot vary training topology.The effect’s sign is established in both halves, but its magnitude only at the small scale.Source: §6 · p. 9
  • The causal half covers two of three task families, and nothing transfers to deeper task instances.The paper also reports that several released-model competence checks fail their predeclared threshold.Source: §6 · p. 9
  • The paired gap test uses tier A at r = 32 with 800 to 900 paired instances, rather than the preregistered second-tier primary cell; norm growth does not establish safety at any budget.The readout-normalisation axis was not varied.Source: §6 · p. 9

7 CONCLUSION

The conclusion treats normalisation placement as a loop–readout contract: agreement is necessary for reusable depth, while the normalisation is a cost under agreement. The paper also states its reproducibility and AI-use practices.

7 CONCLUSION

  • Removing boundary normalisation from released weights breaks the model despite limited state-norm growth, while scratch-trained models without it extrapolate better away from trained depth when each configuration matches its own loop.At matched overshoot, the loop advantage is not detectable at six seeds on one task and is reduced on the other.Source: §7 · p. 9

REPRODUCIBILITY STATEMENT

  • The authors say the full code and fixed protocol artifacts will be released publicly after review, and that reported numbers are generated from run ledgers by one script.The appendix lists configurations, training hyperparameters, accuracy grids, and per-depth tables.Source: REPRODUCIBILITY STATEMENT · pp. 9–10

AI USE STATEMENT

  • Generative AI was used for code review, bug finding, and text editing, but not for the listed research and experimental tasks; authors reviewed the assisted work and take responsibility for the final content.Experiments were run by the authors on their own hardware, and reported numbers come from logged runs.Source: AI USE STATEMENT · p. 10

A.1 MODELS

Table 1 distinguishes released checkpoints, which provide model scale, from scratch-trained models, which support causal claims.

A.1 MODELS

  • Released checkpoints carry scale, while models trained from scratch carry the causal claims.The caption points to Sec. 3.1 for why the two study halves answer different questions.Source: Table 1 · p. 12

A.2 TRAINING HYPERPARAMETERS

The training configurations share their data, tokeniser, schedule and seeds, differing only in the row that names each configuration.

A.2 TRAINING HYPERPARAMETERS

  • All configurations share the same data, tokeniser, schedule and seeds.Source: Table 2 · p. 12
  • The configurations differ only in the row that names them.Source: Table 2 · p. 12
  • Table 2 reports the training hyperparameters of the models trained from scratch.Source: Table 2 · p. 12

A.3 HUGINN IN THE ZERO-SHOT PASS

Table 3 compares released Huginn-3.5B with and without an added boundary normalisation, using zero-shot accuracy across three task families and inference-time loop counts; neither arm clears the competence gate.

A.3 HUGINN IN THE ZERO-SHOT PASS

  • Neither Huginn-3.5B arm clears the competence gate on any of the three task families.Source: Table 3 · p. 12
  • Table 3 compares Huginn-3.5B as released with the reverse operation of adding boundary normalisation.Source: Table 3 · p. 12
  • The table reports zero-shot accuracy by inference-time loop count across all three families.Source: Table 3 · p. 12

A.4 SCOPE OF THE CAUSAL HALF

The causal half uses a single hops cell because the grid is not learned as a mixture within the available budget. Across the supplied experiments, results also distinguish loop-depth extrapolation from transfer to unseen task depth and qualify comparisons by matched budget overshoot.

A.4 SCOPE OF THE CAUSAL HALF

  • The causal half uses one hops cell because an eight-cell mixture shows no transition within the budget, which cannot distinguish per-cell from cumulative learning thresholds.A single cell transitions around 2,750 steps; the matched-budget mixture has 128 thousand examples per cell.Source: §A.4 · p. 12
  • At unseen hops depths d⋆=2, 3, and 4, every configuration scores 0.001–0.030 against a 0.125 chance floor, and increasing K does not help.The passage distinguishes inference loop depth from task depth and reports no transfer between them.Source: §A.4 · p. 13
  • The control task using the same data path is learned to accuracy one, while paragg remains at its floor in every configuration.Source: §A.4 · p. 13

A.7 COLLAPSE METRICS

  • With boundary normalisation removed, Ouro-1.4B predicts one token across the hops split at r = 32, while Ouro-2.6B predicts two tokens with modal share 0.860.The unmodified arms predict 25 and 30 distinct tokens, respectively.Source: §A.7 · p. 13

A.8 TOP-1 MINUS TOP-2 LOGIT

  • Figure 4’s top-1 minus top-2 logit is noisy across r and interpretable only with modal share; the text cites it at the trained budget and r = 32.Source: Figure 4 · p. 14

A.9 LOOPED AGAINST STACKED DEPTH AT MATCHED OVERSHOOT

  • At matched budget overshoot, the loop advantage is absent on circuit and remains only for the loop versus stack at eight times budget on hops (p = 0.026, uncorrected).The hops comparison finds no advantage over 2 × 12 (p = 0.180).Source: Table 6 · p. 14, Figure 5 · p. 14, Table 7 · p. 15

A.10 PAIRED GAP TEST IN FULL

  • On circuit, the paired gap for Ouro-2.6B moved from −0.101 to +0.490 (∆=−0.591; McNemar p=1.0 × 10−140; n=800), while the two hops pairs did not pass the competence gate.Source: §A.10 · p. 15

A.14 REFIT CURVE: STEPS AND SCOPE

  • Refitting only the readout plateaus at a third of the lost accuracy on both tasks; unfreezing the last loop block recovers the loss in full.The table defines the last block as the final recurrent-block layer shared across loop steps.Source: Table 10 · p. 16, Table 12 · p. 17
Loading 2610.31943v1…