Solve the Loop: Attractor Models for Language and Reasoning

1 Introduction

• Unlike chain-of-thought reasoning, which feeds written intermediate tokens back into context, latent refinement aims to improve internal representations without routing computation through discrete tokens.Prior work on looped architectures and latent reasoning explores iterative procedures, weight-shared depth, lower context-length costs, and downstream generalization.Source: §1 · p. 2
• Recurrence can be unstable and costly: training often requires backpropagation through time, memory grows with recurrent steps, and sequential compute can be substantial.Geiping et al. (2025a) reports raw FLOPs for recurrent-model training comparable to a feed-forward model ten times larger; specialized tiny recursive reasoners can degrade or collapse as they scale (Jolicoeur-Martineau, 2025).Source: §1 · p. 2

1.1 Contributions

• Attractor Models use a non-recurrent backbone to propose an output embedding, then a separate recurrent module refines it toward a fixed point before decoding.The architecture treats latent refinement as a fixed-point problem in output-embedding space, rather than unrolling the loop for a predefined number of steps.Source: §1.1 · p. 2
• Refinement steps are chosen adaptively by convergence, and training memory remains constant with iteration count rather than growing linearly as in standard looped language models.The two-stage structure—backbone proposal followed by attractor refinement—is presented as enabling stable, efficient training and strong performance.Source: §1.1 · p. 3
• The authors call it equilibrium internalization when next-token training moves the backbone’s initial prediction closer to the fixed point, making the solver increasingly unnecessary.The passage describes this as a form of automatic curriculum in which recurrence acts as a moving training target for the backbone.Source: §1.1 · p. 3
• A 770M Attractor Model outperforms a 1.3B Transformer trained on twice as many tokens, while 27M models reach 91.4% Sudoku-Extreme and 93.1% Maze-Hard.Across 140M, 370M, and 770M sizes, Attractor Models outperform standard Transformers and stable looped LMs; they use up to 31% less training compute than Parcae. On the reasoning tasks, standard Transformers and named frontier models score 0%, while specialized recursive architectures underperform and collapse when scaled.Source: §1.1 · p. 3

2 Background: Looped Architectures

• A looped model uses a prelude to represent the input, a weight-tied recurrent unit applied for T steps, and a coda to produce output probabilities.The recurrent state is commonly initialized to zero or Gaussian noise.Source: §2 · p. 3
• Recurrence depth T may be fixed, sampled during training, or controlled by a halting mechanism, and training typically backpropagates through depth.Because training cost and gradient memory scale with recurrent steps, changing T at inference can cause a train–test mismatch and degraded performance.Source: §2 · p. 3
• Unlike looped language models that decode after finite repeated steps, Attractor Models refine a backbone’s initial output embedding until its fixed-point residual is small.They then decode the resulting approximate equilibrium.Source: Figure 2 · p. 4

3 Solve the Loop with Attractor Models

• Standard looped models require a preset loop count, use memory linear in loop count, and can lose accuracy when inference runs extra loops; observed trajectory convergence motivates fixed-point modeling.Mechanistic analyses report that recurrent trajectories eventually converge to a fixed point for the vast majority of tokens (Blayney et al., 2026).Source: §3 · p. 4

3.1 Attractor Model: Backbone and Attractor Modules

• Attractor Models pair a meaningful output-embedding proposal from a backbone with a separate weight-tied attractor that refines it to equilibrium, making refinement depth a solver choice.The solver uses Anderson acceleration and stops at a residual tolerance or maximum step count; the initial proposal is injected at every step to prevent a proposal-independent fixed point. The informative warm start also stabilizes training compared with DEQ, whose iteration count can blow up later in training.Source: §3.1 · pp. 4–5

3.2 Training and Inference of Attractor Models

• Implicit differentiation with a one-step approximation avoids backpropagating through solver iterations, keeping attractor-block training memory independent of the number of forward iterations.The model applies standard next-token cross-entropy to the fixed-point output; at inference, solver tolerance and maximum iterations can be adjusted without retraining, with peak memory bounded by one attractor-module forward pass.Source: §3.2 · pp. 5–6

3.3 Equilibrium Internalization and Stability in Attractor Models

• Training often moves the backbone proposal close to equilibrium, a phenomenon called equilibrium internalization, while implicit gradients also bias the recurrent map toward convergent dynamics.Because the proposal and equilibrium share the output-embedding space, gradients move the proposal toward the solver’s equilibrium; the implicit gradient’s inverse factor becomes ill-conditioned near non-contractive regimes, creating a barrier against unstable fixed points.Source: §3.3 · pp. 6–7

4.1 Attractor Models Improve Large-Scale Language Modeling

• At 140M, 370M, and 770M parameters, Attractor Models achieve the best validation PPL, Lambada PPL, and CORE accuracy against matched Transformers and Parcae.The 770M model performs comparably to a 1.3B Transformer trained on twice as many tokens.Source: Table 1 · p. 7, §4.1 · pp. 7–8
• 25–31% fewer training FLOPs: Attractor Models often converge before Tmax, while one-step implicit differentiation keeps training memory constant across solver iterations.Standard looped language models’ memory grows linearly with loop count.Source: Figure 4 · p. 8, §4.1 · p. 8, Figure 5 · p. 8

4.2 Attractor Models Lead to Significant Gains in Hard Reasoning Tasks

• 91.4% on Sudoku-Extreme and 93.1% on Maze-Hard: the 27M Attractor Model outperforms specialized recursive baselines that collapse at this size.The tasks use approximately 1,000 examples each and require direct, non-autoregressive full-grid predictions; the reasoning setup uses phantom gradients because one-step gradients can be too crude for its sensitive, small-data solver dynamics.Source: §4.2 · pp. 8–9, Table 2 · p. 9

4.3 Equilibrium Internalization and Test-Time Behavior

• Attractor Models rapidly converge and reach peak performance at T = 1; their T = 0 backbone readout is at or near the converged value and matches or improves on 140M Parcae at T = 8.The authors attribute this equilibrium internalization to the backbone learning a proposal near the fixed point; Parcae quality improves with test-time iterations until roughly T = 8.Source: §4.3 · pp. 9–10, Figure 6 · p. 10, Figure 7 · p. 10

4.4 Ablations

• Val. PPL 42.18 → 38.74 → 34.05: tying the unembedding partly narrows the gap to DEQ, while placing the fixed point in tied output-embedding space and initializing from the backbone proposal yields the largest improvement.The backbone proposal gives the solver an input-dependent starting point and reaches the same residual tolerance in 1.7× fewer iterations.Source: Table 3 · p. 11, §4.4 · p. 11
• 12.4% of validation tokens converge when the proposal is used only for initialization; persistent additive injection yields the best results across all three metrics.Compared with additive injection, persistent concatenation takes 11.2 vs. 8.4 average iterations and has worse PPL, 36.81 vs. 34.05.Source: Table 4 · p. 11, §4.4 · p. 11
• 0.14 PPL improvement: full implicit differentiation improves PPL only slightly over the one-step approximation, while increasing peak training memory by 4.8× and step time by 2.7×.Phantom gradients lie between these alternatives, while one-step approximation maintains nearly all the full-gradient performance at lower cost.Source: Table 5 · p. 12, §4.4 · p. 12
• Backbone-proposal initialization yields lower perplexity, fewer iterations, and higher downstream accuracy than zero or Gaussian-noise initialization.An uninformative starting state substantially degrades both quality and convergence.Source: Table 6 · p. 12, §4.4 · p. 12

5 Conclusion and Future Work

• Attractor Models achieve strong results in large-scale language modeling and hard reasoning with tiny models by refining prediction embeddings through fixed-point solving.The architecture first produces meaningful prediction embeddings, then refines them through an attractor module.Source: §5 · p. 12
• Fixed-point refinement makes recurrence stable, memory-efficient, and adaptive while avoiding the cost of explicit unrolling.Source: §5 · p. 12
• Training induces equilibrium internalization, making the refinement procedure largely unnecessary at inference time; future work will study this phenomenon and differences from finite-loop recurrence.Source: §5 · p. 12

Looped and recurrent language models

• Prior looped and latent-reasoning models iterate hidden states or continuous representations, but fixed-step unrolling makes training memory depth-dependent and can hurt quality when test-time iterations increase (Zhu et al., 2025c).Mechanistic analysis finds that recurrent updates converge to a fixed point for the vast majority of tokens (Blayney et al., 2026), motivating the formulation.Source: A Related Work · p. 18

Implicit fixed-point models

• Unlike standard DEQs, Attractor Models place equilibrium in tied output-embedding space and warm-start it from a backbone prediction, a structure reported as essential for stable language-model-scale training.The paper also links this structure to equilibrium internalization (Section 4.3).Source: A Related Work · p. 18

Tiny recursive reasoners

• HRM and TRM achieve strong small-data Sudoku and maze accuracy with few parameters but show performance collapse as model size grows, whereas Attractor Models scale cleanly (Wang et al., 2025; Jolicoeur-Martineau, 2025).Attractor Models retain the iterative-refinement benefit of these tiny recursive architectures.Source: A Related Work · p. 18

Positioning

• Attractor Models decouple inference depth, training depth, and recurrent-block memory, with better perplexity than parameter-matched Transformers, 25–31% lower training FLOPs and constant memory versus looped LMs, and clean scaling on hard reasoning.Solver evaluations are chosen adaptively by tolerance ε, and the equilibrium is defined by a residual equation in output-embedding space.Source: A Related Work · p. 18

B.1 Well-posedness and the implicit gradient

• The analysis writes the attractor update as F(y) = fθ(y, E(x)) and its fixed-point equation as y⋆ = F(y⋆).The Jacobian JF is taken with respect to the state input.Source: §B.1 · pp. 18–19
• A local contraction requires F to map a closed ball into itself and be L-Lipschitz there for some L < 1.For continuously differentiable F, a sufficient Lipschitz condition is sup_y∈Br(ȳ) ∥JF(y)∥ ≤ L.Source: §B.1 · p. 19
• The Banach fixed-point theorem establishes that the contraction on the closed ball has a unique fixed point.The proof applies the theorem because the closed ball is complete.Source: §B.1 · p. 19
• Picard iteration from any y0 in the ball remains in the ball and converges linearly to the fixed point.The stated iteration is yk+1 = F(yk).Source: §B.1 · p. 19
• If F is continuously differentiable on the ball, I − JF(y⋆) is invertible and y⋆ depends continuously differentiably on θ nearby, validating the implicit gradient.The proof uses the implicit function theorem on G(y, θ) = fθ(y, E(x)) − y.Source: §B.1 · p. 19
• The backward-pass linear solve converges geometrically at rate L to its unique solution.The passage attributes convergence to the same contraction bound used for the fixed-point analysis.Source: §B.1 · p. 19

B.2 Looped language models are fixed-point iterators

• A depth-T LoopLM with shared weights is exactly T Picard iterations of the residual, warm-started from the input embedding.The equivalence identifies LoopLM as a truncated fixed-point computation.Source: §B.2 · p. 20
• Under Assumption 1 and an in-basin warm start, LoopLM converges to the fixed point as T grows, with discrepancy decaying geometrically.The fixed-point model is the infinite-depth limit of shared-parameter LoopLM, and finite-depth training approximates that equilibrium.Source: §B.2 · p. 20
• Anderson acceleration typically needs fewer backbone evaluations than unrolling LoopLM to a comparable residual tolerance, though the crossover depends on contraction and solver settings.Picard iteration converges linearly, whereas Anderson acceleration converges superlinearly near the fixed point under standard regularity conditions.Source: §B.2 · p. 20
• The convergence guarantee is local and not assured for arbitrary trained transformer blocks; experiments generally find the fixed-point Jacobian norm below one.The passage notes that Jacobian regularization can help enforce this condition.Source: §B.2 · p. 20

B.3.1 Attractor Models

• Attractor Models’ loss depends on the fixed-point output embedding, so implicit gradients optimize the backbone as a standalone next-token predictor decoded by E^T.The initialization and fixed point share the tied output-embedding space, and perturbing the initialization changes the loss through the fixed point.Source: §B.3.1 · p. 20

B.3 Where looped language models fall short

• Standard looped LMs start from a fixed, uninformative hidden state, and backpropagation through time does not train that initialization toward the loss-bearing final state in embedding distance.Because the initialization is not an output embedding, decoding it is not meaningful; it is fixed by the architecture rather than produced by a trained predictor.Source: §B.3.2 · p. 21

B.3.3 Implicit-Gradient Barrier

• As the dominant eigenvalue approaches +1 along a continuous gradient-descent path, the implicit-gradient factor (I − Jg)^−1 becomes unbounded.The inverse is undefined when 1 ∈ σ(Jg).Source: §B.3.3 · p. 21
• Unless v is orthogonal to the offending eigendirection, the gradient norm diverges and descent steps blow up before crossing, confining θ to {ρ(Jg) < 1}.Orthogonality is a codimension-one, hence non-generic, condition.Source: §B.3.3 · p. 21
• For exits through complex eigenvalues, the truncated Neumann series X diverges as ρ(Jg) approaches 1 under the one-step JFB approximation.Source: §B.3.3 · p. 21

B.3.4 Interpretation

• Fixed-loop training does not favor contractive iterations, so it can learn embeddings that become accurate only after exactly K steps and fail when extra loops are run.Such solutions perturb the embedding for K steps and happen to land accurately at step K.Source: §B.3.4 · p. 21
• Equilibrium training’s implicit gradient creates a barrier around non-contractive regions, confining optimization to convergent solutions where one-step gradients are descent directions and extra iterations remain stable.The solver-convergent regime is the one in which additional iterations remain stable.Source: §B.3.4 · p. 21
• The late emergence of fixed points in standard looped training (Blayney et al., 2026) is consistent with a contractive-solution basin that equilibrium training pressures the model toward from the beginning.The loss landscape contains a basin near contractive solutions, but standard looped training applies pressure toward it only late.Source: §B.3.4 · p. 21

• All experiments use NVIDIA H200 GPUs.Source: C Hyperparamter and Experimental… · p. 22
• The hyperparameter tables cover architecture by model family and scale, Attractor-only fixed-point solver settings, and Parcae recurrence settings.Parcae and Transformer do not use a solver.Source: Table 7 · p. 22, Table 8 · p. 22, Table 10 · p. 23
• At each scale, all three model families share the same optimizer, schedule, and data configuration, with differences noted where they occur.Source: Table 9 · p. 23
• Across 140M, 370M, and 770M scales, the settings include diagonal injection, like-init state initialization, mean recurrence 8, and mean backprop depth 4.The listed settings also include poisson-truncated-full sampling, per-sequence iteration, and prelude norm enabled.Source: C Hyperparamter and Experimental… · p. 23