PaperlayerView exact revision on arXiv

Source-linked AI summary

Solve the Loop: Attractor Models for Language and Reasoning

Jacob Fein-Ashley, Paria Rashidinejad

arXiv:2605.12466v1cs.LGcs.AIcs.CLcs.NE

TL;DR

Existing recurrent refinement methods face unresolved tradeoffs in stability, memory, and fixed computation depth. Attractor Models solve refinement as an adaptive fixed-point process, achieving strong language-modeling and reasoning results while exhibiting equilibrium internalization.

  • Problem

    Existing looped language models require fixed loop counts, increasing training memory with recurrence depth and potentially degrading when run longer at inference.

  • Method

    Attractor Models use a backbone to propose output embeddings and an attractor module to refine them to an adaptively solved fixed point.

  • Results

    Attractor Models achieve strong results across large-scale language modeling and hard reasoning with tiny models, while training remains stable, memory-efficient, and adaptive.

  • Takeaways & Limitations

    Fixed-point training can produce equilibrium internalization, making the refinement procedure largely unnecessary at inference time.

  • Takeaways & Limitations

    The supplied limitation passage is truncated and does not state a specific limitation.

Abstract

from arXiv · show

Looped Transformers offer a promising alternative to purely feed-forward computation by iteratively refining latent representations, improving language modeling and reasoning. Yet recurrent architectures remain unstable to train, costly to optimize and deploy, and constrained to small, fixed recurrence depths. We introduce Attractor Models, in which a backbone module first proposes output embeddings, then an attractor module refines them by solving for the fixed point, with gradients obtained through implicit differentiation. Thus, training memory remains constant in effective depth, and iterations are chosen adaptively by convergence. Empirically, Attractor Models outperform existing models across two regimes, large-scale language-model pretraining and reasoning with tiny models. In language modeling, Attractor Models deliver a Pareto improvement over standard Transformers and stable looped models across sizes, improving perplexity by up to 46.6% and downstream accuracy by up to 19.7% while reducing training cost. Notably, a 770M Attractor Model outperforms a 1.3B Transformer trained on twice as many tokens. On challenging reasoning tasks, we show that our model with only 27M parameters and approximately 1000 examples achieves 91.4% accuracy on Sudoku-Extreme and 93.1% on Maze-Hard, scaling favorably where frontier models like Claude and GPT o3, fail completely, and specialized recursive reasoners collapse at larger sizes. Lastly, we show that Attractor Models exhibit a novel phenomenon, which we call equilibrium internalization: fixed-point training places the model's initial output embedding near equilibrium, allowing the solver to be removed at inference time with little degradation. Together, these results suggest that Attractor Models make iterative refinement scalable by turning recurrence into a computation the model can learn to internalize.

1 Introduction

Attractor Models address the instability, memory growth, and sequential cost of recurrent refinement by solving latent predictions as adaptive fixed-point computations. The architecture achieves strong results in language modeling and hard reasoning while learning to internalize refinement into its initial output.

  • Motivation: Transformers use a fixed feed-forward computation, leaving unresolved whether token predictions should be refined before producing outputs.
  • Motivation: Recurrence can enable iterative procedures, shared-weight depth, reduced token-level reasoning costs, and improved downstream generalization, but recurrent models remain difficult to train and expensive to scale.Reported challenges include unstable training, memory requirements that grow with recurrent steps, and significant sequential computation.
  • Attractor Models: Attractor Models choose refinement steps adaptively by convergence, maintain constant training memory with respect to iterations, and avoid the cost of explicit unrolling.The stated design goals also include stable training and efficient inference.
  • Attractor Models: Attractor Models use a non-recurrent backbone to propose an output embedding, then a separate recurrent network refines it by solving a fixed-point problem.The recurrent refinement network is typically smaller than the backbone.
  • Equilibrium internalization: Equilibrium internalization moves the backbone’s initial prediction closer to the fixed point during next-token training, making the solver unnecessary with little inference degradation.The process is described as automatic curriculum or self-distillation of iterative refinement into the initial output embedding.
  • Empirical scope: Attractor Models outperform standard Transformers and stable looped language models across 140M, 370M, and 770M parameter sizes while also targeting hard reasoning with tiny models.The experiments cover large-scale language modeling and hard reasoning as two distinct regimes.

2 Background: Looped Architectures

Looped architectures compose a prelude, a weight-tied recurrent unit applied for T steps, and a coda that converts the final latent state into output probabilities. Existing models primarily differ in how they train, halt, or scale this loop, with recurrence depth T as a central design choice.

  • Architecture: Looped models use a prelude to produce input representations, a weight-tied recurrent unit to update latent states for T steps, and a coda to generate output probabilities.The recurrent update is h_t+1 = R(h_t, x̃), while decoding uses p = C(h_T).
  • Depth selection: Recurrence depth T may be fixed, sampled during training, or determined by an auxiliary halting mechanism.These alternatives characterize how looped architectures are trained, stopped, or scaled across models such as Parcae, Huggin, and Ouro.
  • Architecture comparison: Looped language models repeatedly apply a shared block for a finite number of steps before decoding the final state, whereas Attractor Models refine an initial output embedding toward an approximate equilibrium.Attractor Models stop when the fixed-point residual is small and decode the resulting approximate equilibrium.

3 Solve the Loop with Attractor Models

Attractor Models replace fixed-depth recurrent refinement with an equilibrium computation: a backbone proposes an output embedding, and an attractor module solves for its fixed point. Implicit differentiation keeps training memory independent of solver depth, while convergence—not a preset loop count or halting head—controls computation.

  • Attractor Model design: Attractor Models define predictions as the equilibrium of refinement rather than outputs after a prescribed number of recurrent steps.The backbone first proposes a meaningful output embedding, which the attractor module then refines.
  • Attractor Model design: The backbone initializes refinement from a coherent prediction embedding, improving training stability relative to DEQ’s later-training iteration blow-up.This warm start contrasts with recurrent initialization from zero, noise, or input-side representations.
  • Equilibrium solving: The solver uses Anderson acceleration and stops when the residual reaches ε or after Tmax steps, making computation adaptive to convergence.This replaces a learned halting head or preset loop count with a residual-tolerance condition.
  • Training: Implicit differentiation avoids backpropagating through every solver step, so memory in the attractor block does not grow with forward iteration count.The implementation uses a one-step approximation that reduces the backward pass to one vector–Jacobian product through the attractor equation.
  • Equilibrium internalization: Equilibrium internalization occurs when the trained backbone proposal lies close to the equilibrium, with the attractor acting as a moving teacher and automatic curriculum.This observation suggests the refinement computation can become increasingly represented in the backbone.
  • Equilibrium internalization: Attractor Models create a barrier against unstable fixed-point dynamics, unlike fixed-loop training that can fail when inference uses extra loops.The paper attributes this barrier to ill-conditioning near non-contractive regimes.

4 Experiments

Experiments show that Attractor Models scale effectively in language modeling and hard reasoning while reducing training-memory costs. They also converge rapidly and internalize equilibrium, allowing strong inference with little or no explicit refinement.

  • Parameter Scaling: At 140M, 370M, and 770M parameters, Attractor Models achieve the best validation PPL, Lambada PPL, and CORE accuracy against matched Transformers and Parcae.All models use the same data budget, optimizer, learning-rate schedule, and parameter counts; only the recurrent block differs.
  • Training efficiency: The one-step implicit backward pass keeps training memory constant in solver iterations, while realized training depth and FLOPs are reduced because the solver typically converges below ε in well under Tmax steps.Standard looped language models scale memory linearly with the number of loops.
  • Reasoning: With only 27M parameters and ∼1,000 training examples, Attractor Models reach 91.4% accuracy on Sudoku-Extreme and 93.1% on Maze-Hard, avoiding recursive reasoners’ larger-model collapse.TRM achieves 74.7% and 85.3% at 7M but collapses to 0% on both tasks at 27M.
  • Fixed-point convergence: Across 16 iterations, Attractor trajectories collapse onto a single attractor by iterations 8–16, while Parcae converges more slowly and remains noisier.The convergence behavior supports equilibrium internalization, in which optimization shifts work from iterative solving into the backbone proposal.
  • Equilibrium internalization: At every scale, peak performance occurs at T = 1, while T = 0 decoding of the backbone proposal remains strong because the proposal is already near equilibrium.This preserves iterative computation’s training benefits while reducing reliance on sequential test-time refinement.
  • Ablations and gradient efficiency: Additive proposal injection yields the best results across all three metrics, whereas concatenation converges in 11.2 vs. 8.4 average iterations and reaches 36.81 vs. 34.05 perplexity.Relative to a parameter-matched DEQ, tying the unembedding improves Val. PPL from 42.18 →38.74, while Attractor design reaches the same tolerance in 1.7× fewer iterations; full IFT improves PPL by only 0.14 but raises memory 4.8× and step time 2.7×.

5 Conclusion and Future Work

Attractor Models generate prediction embeddings and refine them by solving for a fixed point. This formulation makes recurrent refinement stable, memory-efficient, and adaptive while delivering strong results in language modeling and hard reasoning with tiny models.

  • Conclusion and Future Work: Attractor Models first produce meaningful prediction embeddings, then refine them through an attractor module by solving for a fixed point.This architecture defines recurrent refinement as fixed-point computation rather than explicit unrolling.
  • Conclusion and Future Work: The formulation makes recurrent refinement stable, memory-efficient, and adaptive while avoiding the cost of explicit unrolling.These properties address key limitations of recurrent architectures identified in the paper’s conclusion.
  • Conclusion and Future Work: Attractor Models achieve strong results across large-scale language modeling and hard reasoning with tiny models.The reported gains span both broad pretraining and challenging reasoning settings.

A Related Work

Attractor Models build on recurrent language models, implicit fixed-point models, and tiny recursive reasoners while decoupling inference depth, training depth, and training memory. Their equilibrium formulation combines adaptive solver depth with constant recurrent-block training memory.

  • Looped and recurrent language models: Looped and recurrent language models use weight-tied or shared blocks to iteratively update hidden states for latent reasoning.Universal Transformers share a single block across depth, while looped and recurrent models iterate recurrent updates.
  • Implicit fixed-point models: Deep Equilibrium Models replace finite unrolling with a fixed-point equation and use implicit differentiation, decoupling effective depth from training memory.Typical solvers include Anderson acceleration and gradient methods ranging from full implicit differentiation to cheaper surrogates.
  • Tiny recursive reasoners: HRM and TRM achieve strong accuracy on small-data Sudoku and maze benchmarks with few parameters, but performance collapses as model size grows.This behavior is described as “less is more”; Attractor Models aim to retain iterative refinement while scaling with parameter count.
  • Positioning: Attractor Models decouple inference depth, training depth, and training memory by defining an output-embedding equilibrium and adaptively selecting solver evaluations by tolerance ε.Training memory in the recurrent block remains constant in effective depth.

B Theory of Looped and Attractor Models · B.1 Well-posedness and the implicit gradient

The theory establishes well-posed fixed points under a local contraction assumption and justifies both Picard iteration and implicit differentiation. These guarantees support convergent backward linear solves for the attractor model.

  • B.1 Well-posedness and the implicit gradient: The analysis defines F(y) as fθ(y, E(x)), reducing the fixed-point equation to y⋆ = F(y⋆).The state Jacobian is J_F(y) = ∂F/∂y.
  • B.1 Well-posedness and the implicit gradient: The core assumption is local contraction: F maps a closed ball B_r(ȳ) into itself and is L-Lipschitz there for L ∈ [0, 1).The analysis uses a generic norm and its induced operator norm.
  • B.1 Well-posedness and the implicit gradient: When F is continuously differentiable, sup_y∈B_r(ȳ) ||J_F(y)|| ≤ L is sufficient for the Lipschitz condition.This provides a Jacobian-based criterion for local contraction.
  • B.1 Well-posedness and the implicit gradient: Under the contraction assumption, the theorem guarantees well-posedness and establishes the stated fixed-point results.The proof applies the Banach fixed-point theorem to F on the closed, complete ball B_r(ȳ).
  • B.1 Well-posedness and the implicit gradient: For any y_0 ∈ B_r(ȳ), Picard iterates y_{k+1} = F(y_k) remain in the ball and converge linearly.The convergence result is part of Theorem 1 under Assumption 1.
  • B.1 Well-posedness and the implicit gradient: If F is continuously differentiable, I − J_F(y⋆) is invertible and y⋆ depends continuously differentiably on θ near the current parameters.These properties follow from the implicit function theorem applied to G(y, θ) = fθ(y, E(x)) − y.
  • B.1 Well-posedness and the implicit gradient: The implicit gradient used in the backward pass is valid under the same contraction-based conditions.The theorem connects the differentiability of the fixed point to the gradient formula used in Section 3.
  • B.1 Well-posedness and the implicit gradient: The backward linear solve converges geometrically at rate L to a unique solution because the relevant Jacobian operator is contractive.The cited passages specify the solve through the transpose Jacobian and its fixed-point inverse.

B.2 Looped language models are fixed-point iterators … B.3.2 Standard looped LMs

The paper frames LoopLMs as finite Picard iterations toward the same fixed point targeted by Attractor Models, with geometric approximation error and potentially higher inference cost. Attractor Models instead train an output-embedding predictor coupled to a fixed-point attractor, unlike standard looped LMs whose initialization is an uninformative hidden state.

  • B.2 Looped language models are fixed-point iterators: A depth-T LoopLM with shared block F is exactly T Picard iterations of the residual gθ(·, x), warm-started from the input embedding.This identifies finite-depth looped computation with truncated fixed-point iteration.
  • B.2 Looped language models are fixed-point iterators: As T approaches infinity, LoopLM outputs converge to y⋆, and the discrepancy from the fixed-point model decays geometrically in T.The result assumes Assumption 1 and a warm start E(x) ∈ B_r(ȳ).
  • B.2 Looped language models are fixed-point iterators: The fixed-point model is the T →∞ limit of shared-parameter LoopLMs, while finite-depth training approximates the same equilibrium y⋆ with error controlled by (10).Thus, recurrence depth determines approximation quality rather than a different target equilibrium.
  • B.3 Where looped language models fall short: For a target residual tolerance ε, Anderson acceleration typically needs fewer fθ evaluations than Picard-unrolled depth T, although the crossover depends on L and solver hyperparameters.Picard iteration converges linearly at rate L, whereas Anderson acceleration converges superlinearly near y⋆ under standard regularity conditions.
  • B.3 Where looped language models fall short: The contraction assumption is local rather than guaranteed for arbitrarily trained transformer blocks, though experiments generally find ∥JF(y⋆)∥≤L<1.Jacobian regularization can help enforce the needed condition.
  • B.3.1 Attractor Models: Attractor Models place the initialization ỹ0 and fixed point ỹ⋆ in the same tied output-embedding space, so both can be decoded by E⊤ and the backbone is trained as a next-token predictor.The loss depends on ỹ⋆, while perturbations in the backbone output embedding affect the loss through the fixed point.
  • B.3.2 Standard looped LMs: Standard looped LMs begin from an uninformative hidden-state initialization, typically zero or noise, and train through backpropagation along the unrolled trajectory without driving h0 toward hT in embedding distance.Because h0 is not a separately trained predictor output, decoding h0E⊤ is not meaningful.

B.3.3 Implicit-Gradient Barrier

Implicit differentiation creates a barrier at the unit spectral boundary: as the dominant eigenvalue approaches +1, the inverse (I −Jg)−1 becomes singular and gradients diverge. Except for a non-generic orthogonality condition, gradient descent therefore remains confined to the contractive region {ρ(Jg) < 1}.

  • Implicit-gradient singularity: Along continuous gradient-descent paths, the dominant eigenvalue must approach 1 before leaving the unit disk through +1, forcing ∥(I −Jg)−1∥→∞.This is the canonical loss-of-contraction route for residual iteration maps.
  • Gradient-descent barrier: Unless v is orthogonal to the offending eigendirection, a non-generic codimension-one exception, the gradient norm diverges and the descent step blows up before the boundary is crossed.The resulting dynamics confine θ to {ρ(Jg) < 1}.
  • Gradient-descent barrier: For exits through complex eigenvalues, the same barrier holds under the one-step JFB approximation because the truncated Neumann series diverges as ρ(Jg) →1.This extends the barrier beyond the +1 eigenvalue route.

B.3.4 Interpretation

Equilibrium training avoids fixed-loop solutions that become unstable when extra iterations are run by steering optimization toward contractive, convergent solutions from the beginning. This explains why additional iterations remain stable under equilibrium training, whereas fixed-loop training lacks that guarantee.

  • Interpretation: Fixed-loop training can learn perturbations that reach accuracy only after K steps, causing failure when extra loops run at inference.Such solutions are valid under fixed-loop optimization because it has no inherent mechanism favoring contractive iterations.
  • Interpretation: Equilibrium training creates a diverging-gradient barrier around non-contractive regions, confining optimization to convergent and stable trajectories.Within this regime, the solver converges, the one-step gradient is a descent direction, and additional iterations remain stable.
  • Interpretation: Fixed points emerging only late in standard looped training is consistent with a basin near contractive solutions that equilibrium training targets from the beginning.The distinction is that only equilibrium training applies pressure toward this basin throughout training.

C Hyperparamter and Experimental Settings

The experiments use NVIDIA H200 GPUs and report architecture, fixed-point solver, training, and Parcae recurrence hyperparameters across model families and scales. The model families share optimizer, schedule, and data configurations at each scale, with specified implementation differences at 770M.

  • All experiments use NVIDIA H200 GPUs.
  • The appendix reports architecture hyperparameters for each model family and scale, alongside fixed-point solver settings for Attractor Models.Parcae and Transformer do not use a solver.
  • All three model families share the same optimizer, schedule, and data configuration at each scale, with differences noted where they occur.
  • At 770M, learning rates, gradient checkpointing, and torch.compile differ across Attractor Model, Transformer, and Parcae.Attractor Model uses AdamW LR 5 × 10−3 and Muon LR 6 × 10−3; Transformer/Parcae use 6 × 10−3 / 8 × 10−3. Gradient checkpointing is enabled for Attractor Model and Parcae, while torch.compile is enabled for Transformer and Parcae.
  • Parcae recurrence hyperparameters are reported separately.
Loading 2605.12466v1…