Training-Free Looped Transformers

Research question

• The paper asks whether a frozen, off-the-shelf checkpoint can be looped at inference without fine-tuning, continued training, auxiliary parameters, or architectural changes.Earlier recurrent methods tie recurrence to weights optimized for the looped structure, limiting use on most released checkpoints.Source: §1 · p. 1

Motivation

• Viewing each pre-norm transformer layer as a forward Euler step motivates treating looping as finer integration of the same ODE endpoint.The method uses smaller damped sub-steps rather than advancing the trajectory beyond the endpoint the unmodified network targets.Source: §1 · pp. 1–2

Approach

• The wrapper reapplies a contiguous mid-stack block at inference, leaving checkpoint weights unchanged and adding no parameters.It offers block- and layer-mode iteration; layer-mode is the safer default for MoE because it pins per-layer expert routing.Source: Figure 1 · p. 2, §1 · p. 2

Results

• +2.64 on MMLU-Pro and +2.01 on GPQA-Main for Qwen3-4B-Instruct, with +2.30 on ARC-Challenge for Qwen1.5-MoE-A2.7B-Chat.The evaluation covers seven model families and 45 model-benchmark cells.Source: §1 · p. 2

Contribution

• The reported improvements require no parameter updates, additional supervision, or benchmark-specific tuning.The paper presents repeated application of existing blocks as a way to expose latent inference-time computation.Source: §1 · p. 2

2.1 Loop wrapper

• A contiguous window defines an operator g, and the wrapper composes pre-loop layers, a K-step iteration g(K), and post-loop layers.If the window reaches either network boundary, the corresponding pre-loop or post-loop composition is the identity.Source: §2.1 · pp. 2–3

2.2 Block-mode versus layer-mode iteration

• For MoE layers, layer-mode reuses each layer’s expert mixture across iterations, while block-mode can become unstable as routing changes with perturbed hidden states.The two modes yield broadly similar quality on dense models; layer-mode is the correct default for MoE backbones.Source: §2.2 · p. 3

2.3 Loop strategies

• Naively applying g K times advances toward x(t=K), although post-loop layers are trained for the approximation of x(t=1).In a small regression model, damped sub-step endpoints stay in the trained low-loss valley while naive-loop endpoints drift into high-loss regions.Source: §2.3 · p. 4, Figure 2 · p. 4
• Damped Euler sub-steps divide the same integration interval [0, 1] into steps of size h=1/K to refine the endpoint approximation.The paper also evaluates higher-order Runge–Kutta integrators and other numerical integration strategies.Source: §2.3 · pp. 5–6
• A front-loaded Runge–Kutta quadrature controlled by β interpolates between the original output and the K-substep forward Euler endpoint.β=0 recovers the sub-step endpoint, while β=1 preserves the original output.Source: §2.3 · pp. 5–6

3.1 Experimental setup

• Across seven model families, evaluations use a fixed recipe without per-cell hyperparameter search, with block-mode for dense backbones and layer-mode for MoE backbones.The evaluated checkpoints span dense and MoE models, base and instruction-tuned variants, and MHA and MLA backbones.Source: §3 · p. 6, §3.1 · pp. 7–8

3.2 The depth fraction rule

• For models larger than 1.7B, the optimal loop-window center lies at fractional depth 0.43–0.71, while sub-1B models favor 0.25–0.56.The authors hypothesize that head specialization in late layers explains why smaller or heavily distilled models favor earlier windows.Source: §3.2 · p. 8

3.3 Layer-mode for MoE

• On MoE backbones, layer-mode iteration can flip most negative cells positive, yielding +0.5 to +1.7 pp on hard benchmarks by avoiding routing thrash.Block-mode re-evaluates expert gates on perturbed states, accumulating routing-induced changes.Source: Table 2 · p. 7, §3.3 · p. 8

3.4 Results on knowledge-based MC tasks

• The largest and most reliable gains occur on knowledge-heavy multiple-choice tasks, especially MMLU-Pro, GPQA-Main, and ARC-Challenge, with the largest gains on hard tasks for strong MHA backbones.MLA-based MoE models move in the same direction with 3–4× smaller magnitudes.Source: Table 3 · p. 8, §3.4 · p. 9

3.5 Cross-architecture summary

• Under the fixed recipe, 27 cells are positive and 12 neutral; the remaining negatives concentrate in sub-3B distilled checkpoints, while 87% of cells are positive or neutral.GPQA-Main still improves by +1.79 on Llama-3.2-1B at a very-early position, so the failure boundary is task-dependent.Source: §3.5 · p. 9

3.6 Ablations

• Ablations examine integration strategy, loop count, window width and position, iteration mode, cache and decode handling, and recipe robustness.The experiments also report robustness checks, per-subject decompositions, untuned transfer, and failed-configuration analyses.Source: §3 · p. 7, §3.6 · p. 9

3.7 Comparison with other looped transformer methods

• Compared with the naive loop, which collapses across tested cells, the proposed method produces the highest accuracy on nearly every cell.Figure 5 compares baseline, naive K=4 looping, and the proposed method on Llama-3.2-3B-Instruct and Moonlight-16B-A3B-Instruct.Source: §3.7 · p. 10, Figure 5 · p. 10

4 Related work

• Prior looped, recurrent-depth, adaptive-recursion, and implicit-depth transformers tie recurrence to training, whereas this wrapper targets released checkpoints without training or architectural changes.Examples include weight-tied stacks, looped pretraining, and implicit-depth fixed points.Source: §4 · p. 10
• Chain-of-thought, self-consistency, and reasoning-trained LMs spend test-time compute without changing the per-token forward, unlike this wrapper’s within-forward computation.Other latent-reasoning work uses continuous-space computation or adaptive halting.Source: §4 · p. 10
• Treating residual blocks as forward Euler steps motivates the wrapper’s integration strategies, but no tested fixed-point accelerator robustly beats K-stage Runge–Kutta.The paper also distinguishes its MoE layer-mode iteration from prior training-free layer skipping.Source: §4 · p. 10

5 Conclusion

• The wrapper refines a pre-norm block’s forward Euler approximation by iterating a contiguous loop window, using layer-mode on MoE backbones to stabilize routing.Across seven model families, 87% of cells are non-negative under a fixed recipe without per-cell tuning, with gains concentrated on knowledge-heavy multiple-choice benchmarks.Source: §5 · p. 11

A Notation and glossary

• Table 5 consolidates the symbols used throughout the paper and appendix.It lists each symbol’s type and where it is introduced.Source: A Notation and glossary · p. 20, Table 5 · p. 20

B.1 Proof of (9)

• The proof expands block-mode iteration into per-layer forward Euler steps and verifies the closed-form identity stated in Section 2.3.It defines the loop-window operator and recursively tracks the intra-window residual stream.Source: §B.1 · p. 20

B.2 Proof of (16)

• The proof of equation (16) substitutes the stated identities into its right-hand side and matches the left-hand side by expanding the definitions.The passage identifies k1 as Fg(x0) = g(x0) − x0.Source: §B.2 · p. 21

C. Decode loop algorithm

• During prefill, the loop body writes no cache entries and a single stash pass writes canonical KV; during decode, iterations read past KV but crop their writes before one canonical stash.The decode procedure then runs post-loop layers and returns the final output norm.Source: C Decode loop algorithm · p. 21, §C.1 · p. 21

C.2 Snapshot/restore

• Snapshot-and-crop lets each decode iteration attend to genuine past KV while preventing extra entries from becoming phantom prefix tokens.A stash pass writes the canonical KV entry after the loop iterations.Source: §C.2 · p. 21

C.3 decode_mode variants

• Decode mode can bypass looping, loop on every decode step, or loop only on the first n generated tokens; full is the default for reported generation results.The first-n setting is intended for CoT-prefix refinement but does not outperform full mode when looping helps.Source: §C.3 · p. 22

D Per-cell configurations

• The per-cell table records each cell’s baseline, best loop configuration, and percentage-point change; across families, winning windows generally fall at depth fraction 0.4–0.7.MoE cells tend to prefer layer-mode K=2 over block-mode K=3, while cache preference varies with task length and format.Source: D Per-cell configurations · p. 22, §D.1 · p. 23, Table 6 · p. 24

E.1 Anderson, heavy-ball, Aitken, α-schedules

• Across more than 40 configurations, no tested fixed-point accelerator robustly beats K=2-stage Runge–Kutta; Anderson degrades smoothly with K and catastrophically at K=8.The closest schedule ties damped Euler within −0.03 pp, and the robust default uses K=2-stage Runge–Kutta with n=4 and cache=first.Source: D Per-cell configurations · p. 23, E Strategy ablation tables · p. 23, §E.1 · pp. 23–24

E.2 Norm stabilization and polynomial blending

• Norm stabilization is catastrophic at K ≥3, indicating that the iterates diverge in direction as well as magnitude.The tested rescaling fixes each iterate to a fixed L2 norm.Source: §E.2 · p. 24

E.3 Layer-mode replicates the result

• Layer-mode K=2 damped Euler scores 56.347 on the same window and serves as the reference for the strategy sweep.The reported tables compare strategy deltas against layer-mode K=2 damped Euler.Source: §E.3 · p. 25, Table 11 · p. 26
• Higher-order solvers lose to damped Euler, and Anderson drops 16-task macro by -19.6 pp at K=4; across 40+ configurations, no accelerator robustly beats Runge–Kutta.Layer-mode losses for Heun, midpoint, and RK4 are -1.7 to -2.3 pp, versus -0.7 to -1.1 pp in block-mode.Source: §E.3 · p. 25

G.1 Catastrophic collapse on naive K=4 looping

• Naive K=4 looping on Qwen3-0.6B-Instruct raised lambada perplexity from ∼13 to 1054.12 and reduced 16-task macro by -10.21 pp.The test used a four-layer window without damping.Source: §G.1 · p. 27

G.3 Wide loop windows blow up

• On Qwen3-1.7B-Base, widening the loop window degraded performance, and looping the whole network drove lambada perplexity to 6.3 × 10^5.The authors characterize the useful contracting region as a contiguous mid-band of about four layers.Source: §G.3 · p. 27

G.4 Higher-order ODE solvers all degrade

• Heun, midpoint, and RK4 lose to damped Euler in both modes because the looped block does not meet the smooth-flow assumptions used by higher-order methods.Block-mode losses are -0.7 to -1.1 pp, and layer-mode losses are -1.7 to -2.3 pp.Source: §G.4 · p. 27

G.5 Position-search overfits to the 16-task aggregate

• Two position-search configurations gained +0.9 to +1.3 pp on the 16-task aggregate but regressed -1.0 to -1.4 pp on held-out MMLU 5-shot.The reported examples are Qwen3-0.6B-Base at +1.33 and -1.21, and Qwen3-1.7B-Base at +0.92 and -1.38, respectively.Source: §G.5 · p. 28
• The 16-task suite is a noisy target for sub-1pp claims because MMLU subjects may have only 100–300 examples, so reported wins were rechecked on ∼14k-example MMLU 5-shot.Configurations had to be non-negative on both the screen and held-out check to be retained.Source: §G.5 · p. 28

G.6 Layer-mode K=3 is uniformly catastrophic

• Layer-mode K=3 heavy-ball on Qwen3-4B-Base lost -5.0 pp on 16-task and -6.0 pp on MMLU 5-shot versus canonical K=3 Euler block-mode.The passage says layer-mode tolerates only mild K=2 Runge–Kutta; iterating individual layers more than twice produces strongly out-of-distribution per-layer states.Source: §G.6 · p. 28

G.7 Anderson at K=8

• Anderson m=3, β=1.0, K=8 on Qwen3-1.7B-Base dropped 16-task macro by -18.06 pp, the strongest evidence reported that the block is not contractive.The authors note Anderson is provably optimal on contractive maps.Source: §G.7 · p. 28

G.8 Sub-1B knowledge MC

• After per-cell tuning, Llama-3.2-1B remained negative on MMLU (-0.63) and MMLU-Pro (-1.36), but GPQA-Main flipped positive by +1.79 at a very-early window.The passage identifies failures in small checkpoints as task-dependent rather than absolute.Source: §G.8 · p. 28

H Concentration of benchmark gains

• Across MMLU, MMLU-Pro, and MMLU-Redux, gains concentrate on STEM and quantitative-reasoning subjects where the baseline is furthest from ceiling.The cited analysis covers MMLU’s 57 subjects, MMLU-Pro’s 14 categories, and MMLU-Redux’s 30 subjects.Source: H Concentration of benchmark gains · p. 28
• Qwen3-4B-Base under [17–19] K=2 Euler was essentially tied on MMLU macro, 73.01→72.90 (-0.11 pp), despite seven subjects gaining ≥+2 pp on hard STEM or quantitative content.Small offsetting losses spread across easier subjects; this is the closest available subject breakdown, not the [15−18] K=3 winner.Source: Table 13 · p. 28
• Qwen3-4B-Instruct gained +0.93 pp macro on the MMLU-Redux 30-subject sweep, with 14 subjects positive and 10 tied; the largest gains included professional_accounting (+8.00) and high_school_physics (+7.00).The passage also reports +5.00 for econometrics and +3.00/+4.00 for college_chemistry under K=1 Heun.Source: H Concentration of benchmark gains · pp. 28–29
• On Qwen3-4B-Instruct MMLU 0-shot, the 57-subject macro rose +0.99 pp (68.15 →69.14), with global_facts and us_foreign_policy additionally gaining +6 to +7 pp under K=1 Heun.These values are reported for K=2 Euler except the two subject gains specified under K=1 Heun.Source: Table 14 · p. 29

I Wall-clock cost of training-free looping

• 0% wall-clock cost for bypass; first_n ranges from -1.0% to +4.6%, while full decode-loop is approximately 22% slower.The profile used Qwen3-4B-Instruct on GSM8K@200 with K=3 Euler at [15–18].Source: I Wall-clock cost of training-free… · p. 29, Table 15 · p. 29

J.1 Held-out validation discipline

• Screen-winning configurations can regress on held-out MMLU 5-shot, so reported configurations were checked against both evaluations.Qwen3-0.6B-Base scored +1.33 on the 16-task screen and -1.21 on MMLU 5sh.; overfit configurations were excluded.Source: §J.1 · pp. 29–30

J.2 Per-config robustness on Qwen3-4B-Instruct

• Every tested loop configuration was positive on Qwen3-4B-Instruct’s 16-task aggregate at.Table 16 reports the aggregate deltas across configurations.Source: §J.2 · p. 30, Table 16 · p. 30

J.3 Multiple winning windows on small models

• Six Qwen3-0.6B-Base windows beat baseline on 16-task macro, with deltas from +0.36 to +1.33.The reported windows span [3–6], [6–9], [7–10], [8–11], [9–12], and [14–17].Source: §J.3 · p. 30

J.4 Language modeling perplexity is preserved

• The wrapper does not degrade language-modeling fluency and slightly improves perplexity on dense Qwen3 backbones.LAMBADA perplexity was measured on every reported loop configuration.Source: §J.4 · p. 30

J.5 Direction of effect is consistent across few-shot counts

• The loop’s direction of effect persists across few-shot counts and CoT regimes, including +2.01 on GPQA-Main 0-shot for Qwen3-4B-Inst.The paper characterizes this pattern as a few-shot-invariant frozen-context refiner.Source: §J.5 · p. 30

K Scaling to 30B: Qwen3-30B-A3B-Instruct broader sweep

• 11 of 12 tabulated Qwen3-30B-A3B-Instruct cells are non-negative under the better of K=2 Euler and K=1 Heun, without per-cell tuning.The sweep used the window across a 19-task held-out suite.Source: K Scaling to 30B:… · p. 30, Table 17 · p. 31

L Layer-mode wins beyond MoE backbones

• Layer-mode also yields validated wins on sub-1B dense backbones, where analogous block-mode iteration does not survive held-out validation.The Qwen3-0.6B-Base recipe was the only 0.6B configuration to beat canonical block-mode on both reported metrics.Source: L Layer-mode wins beyond MoE backbones · p. 31, Table 18 · p. 31

M Loss surface breadth across dense Qwen3 sizes

• On Qwen3-4B-Base, 13 positive windows span depth fractions 0.19–0.83, so useful placements do not require precise per-architecture tuning.The depth-fraction 0.45–0.60 rule captures the location of the maximum, while the positive-window basin is wide.Source: M Loss surface breadth across dense… · pp. 31–32, Table 19 · p. 31

N.1 KV cache handling

• Loop iterations avoid KV writes, then a single stash pass writes one canonical entry per loop layer and token.Cache=first favors long-prompt CoT and cache=last favors short structured generation; both were non-negative on every tested headline cell.Source: §N.1 · p. 32, Table 20 · p. 32

O Per-architecture implementation notes

• The wrapper patches each backbone’s top-level decoder; MoE layer-mode uses a per-layer iterator, while MLA backbones use a cache-compatibility shim.The shim restores three deprecated cache API entry points and is reversed when the patch is unloaded.Source: O Per-architecture implementation notes · pp. 32–33, Table 21 · p. 33

P Hyperparameter search protocol

• The structured search screened candidates on a 16-task aggregate, rechecked the top three on held-out MMLU 5-shot, and stopped at an 8-hour per-cell GPU budget.On benchmarks with at most 1,000 examples, run-to-run variance was at most 0.5 pp; deltas within ±0.3 pp were treated as neutral.Source: P Hyperparameter search protocol · p. 33

• Figure 6 plots the best loop-window depth-fraction range for each of nine model checkpoints.Source: Q Depth fraction rule across nine… · p. 34
• The shaded 0.45–0.60 depth-fraction band marks the empirical sweet spot.Source: Q Depth fraction rule across nine… · p. 34
• The figure places each checkpoint’s best-window range relative to the shared depth-fraction band.Source: Q Depth fraction rule across nine… · p. 34