PaperlayerView exact revision on arXiv

Source-linked AI summary

Equilibrium Reasoners: Learning Attractors Enables Scalable Reasoning

Benhao Huang, Zhengyang Geng, Zico Kolter

arXiv:2605.21488v1cs.LG

TL;DR

Iterative latent reasoning lacks a clear mechanistic account of when additional test-time computation helps. The paper introduces Equilibrium Reasoners, which shape task-conditioned attractor landscapes and scale inference through depth and breadth, reaching 99.8% exact accuracy on Sudoku.

  • Problem

    The mechanisms governing when test-time scaling helps iterative latent reasoning models remain insufficiently understood.

  • Method

    Equilibrium Reasoners shape latent attractor landscapes using randomized initialization and path noise, then scale inference through iterative depth and stochastic breadth.

  • Results

    EqR substantially outperformed prior iterative reasoning models on controlled benchmarks, reaching 99.8% exact accuracy on Sudoku.

  • Takeaways & Limitations

    Learned attractor landscapes provide a mechanistic lens for scalable reasoning, with reliable scaling tied to correct attractors being stable, reachable, and selectable.

  • Takeaways & Limitations

    Convergence is not universally reliable: baseline residual reduction can reach spurious attractors, becoming useful as a task-error proxy only after landscape shaping.

Abstract

from arXiv · show

Scaling test-time compute by iteratively updating a latent state has emerged as a powerful paradigm for reasoning. Yet the internal mechanisms that enable these iterative models to generalize beyond memorized patterns remain unclear. We hypothesize that generalizable reasoning arises from learning task-conditioned attractors: latent dynamical systems whose stable fixed points correspond to valid solutions. We formalize this process through Equilibrium Reasoners (EqR), which enable test-time scaling without external verifiers or task-specific priors. EqR scales internal dynamics along two axes: depth, by running more iterations, and breadth, by aggregating stochastic trajectories from multiple initializations. Empirically, gains from test-time scaling are tightly coupled with stronger convergence toward solution-aligned attractors. This attractor perspective allows neural networks to adaptively allocate test-time compute based on task difficulty. While simple cases converge within 1 to 5 iteration steps, harder cases benefit from massive test-time scaling. By unrolling up to the equivalent of 40,000 layers, scalable latent reasoning boosts accuracy from 2.6% for feedforward models to over 99% on Sudoku-Extreme. These results suggest that learned attractor landscapes provide a useful mechanistic lens for understanding scalable reasoning in iterative latent models.

1. Introduction

The introduction frames scalable reasoning as convergence toward task-aligned attractors and presents EqR as a latent dynamical system that scales inference through depth and breadth. It argues that convergence diagnostics and attractor-landscape alignment explain when additional test-time compute improves generalization.

  • Motivation: 1,024 iterations at test time generalize beyond 16 training iterations, equivalent to unrolling over 40,000 effective layers.EqR evaluates exact accuracy against the fixed-point residual ∥fθ(z; x) − z∥, with lower residual indicating better convergence.
  • Motivation: More test-time compute can produce diminishing returns or worse performance, motivating the search for internal mechanisms of scalable, generalizable reasoning.The introduction identifies this as a central question for test-time scaling.
  • Attractor perspective: Test-time scaling is effective when the internal attractor landscape aligns with the task-metric landscape, so stronger convergence yields lower-error answers.EqR treats training as shaping a differentiable attractor surrogate while leaving adaptive computation to inference.
  • Diagnostics: Lower fixed-point residual tightly tracks lower prediction error across Sudoku-Extreme and Maze-Unique, providing a convergence-based diagnostic for generalizable reasoning.The residual is ∥fθ(z; x) − z∥.
  • Two-axis scaling: D ≳4, or equivalently 168 layers unrolled, marks the minimum depth beyond which breadth scaling becomes effective for exploring the landscape and finding attractors.EqR uses randomized initialization and noise injection, with depth D measuring steps per trajectory and breadth B measuring stochastic trajectories.

2. Background and Problem Formulation

Iterative reasoning models perform multi-step computation by repeatedly updating a latent state. Test-time scaling applies the same learned update rule for additional iterations, starting from an input-conditioned initialization and decoding the final state.

  • Iterative reasoning models maintain a latent state z_k ∈ R^n while performing multi-step computation on input x ∈ X.
  • A parameterized update operator repeatedly transforms the latent state across iteration index k using model parameters θ.
  • Starting from z_0 ∼ μ_0(· | x), the model runs K updates and decodes the final state, framing test-time scaling as additional applications of the same learned rule.

3. From Feedforward Predictors to Iterative Reasoners

The section defines a controlled path from feedforward predictors to iterative reasoners by progressively adding shared parameters, local gradient training, supervision scheduling, hierarchical updates, and adaptive computation. It isolates how these design choices alter optimization, latent-state updates, and compute allocation.

  • Weight-tied structure: Weight tying replaces distinct layers with repeated applications of the same update block across iterations.This isolates the effect of parameter sharing in the transition from feedforward depth to iterative computation.
  • Truncated gradients: Truncated gradients with detached carry preserve long forward trajectories while limiting backpropagation to local trajectory windows.This reduces memory and compute costs and avoids poorly conditioned backward dynamics from multiplying recurrent Jacobians.
  • Hierarchical iterations: Hierarchical iterations compare single-stream updates with two latent states updated at different frequencies to test the contribution of coupled fast/slow dynamics.This separates the effect of weight-tied iteration from the additional two-timescale structure used in HRM/TRM-style models.
  • Construction path: The construction path progressively adds five ingredients: weight-tied parameters, truncated gradients with detached carry, segmented online training, hierarchical iterations, and adaptive computation.Together, these components turn feedforward models into scalable iterative reasoners while shaping intermediate solver states and allocating compute by difficulty.
  • Supervision placement and optimization schedule: Segmented online training supervises segment endpoints and updates parameters immediately, unlike vanilla final-step training or trajectory supervision with one final update.The three schedules differ in where losses are placed and when optimizer steps occur along the K-step trajectory.
  • Adaptive computation time (ACT): Adaptive computation uses a learned halting score to allocate variable iteration budgets, distinguishing fixed-depth, oracle-halting, and learned-halting variants.The key distinction is whether the predicted halting signal actually controls compute allocation across examples.

4. Iterative Models as Attractor Dynamics

The section frames iterative inference as movement toward solution-aligned attractors rather than exact fixed points. It links attractor alignment and reachability to complementary depth and breadth scaling, predicting when additional test-time compute can help.

  • Attractor view: Attractor convergence preserves the useful convergence claim without requiring a single exact fixed point: favorable basins and lower-residual states can support useful test-time computation.A favorable basin is sufficient, even when stable outcomes include fixed points or small recurrent sets.
  • Landscape geometry: Attractor landscapes are characterized by task alignment and reachability: alignment concerns solution correctness, while reachability concerns which attractor trajectories reach across initializations or perturbations.Broad attractors are reachable from many initial states, whereas deep attractors are stable.
  • Scaling levers: Depth scaling adds iterations for within-basin refinement, breadth scaling aggregates independent restarts for basin coverage, and inference budget is summarized as NFE = D · B.The two levers target different aspects of trajectory reachability and refinement.
  • Landscape regimes: When correct and spurious attractors coexist, breadth is most useful for basin selection, while depth helps only after trajectories enter the correct basin.When the correct attractor is narrow or weakly attracted, breadth improves entry probability and depth can help trajectories settle, but gains remain limited by basin mass and stability.
  • Landscape regimes: In a well-aligned landscape, residual decay tracks task-error reduction, so depth reliably refines trajectories; scaling fails when no reachable attractor decodes to a good outcome.Overall, depth refines after reaching a useful basin, breadth increases basin coverage, and scaling succeeds when correct attractors are aligned and reachable.

5. Shaping Attractor Landscapes

This section presents attractor landscape shaping as the guiding training principle for making correct solutions stable and reachable from diverse initial states. It introduces randomized state initialization and noise injection as complementary, task-agnostic interventions that improve coverage and exploration without external verifiers or hand-crafted heuristics.

  • Guiding principle: Attractor landscape shaping trains iterative dynamics to admit correct solutions as stable attractors and make their basins reachable from diverse initial states.This supports reliable convergence as test-time compute increases.
  • Randomized state initialization: Randomized state initialization samples z0 ∼µ0(· | x) independently per trajectory, matching breadth scaling and probing different basins.Unlike fixed z0 training, this broadens shaped regions and encourages stable predictions across restarts.
  • Randomized state initialization: Randomizing z0 expands training-state coverage, increasing the likelihood that correct attractors are reachable and promoting path-independent predictions across restarts.The same (x, y) is observed under multiple initial states, penalizing divergent predictions.
  • Complementary interventions: RI broadens rollout starting regions, while NI perturbs each iteration to reduce train–test mismatch, avoid premature trapping, and broaden exploration.The interventions target complementary parts of each trajectory: initialization coverage and local dynamics.
  • Noise injection: λ ∈[0, 1) controls damping and β ≥0 controls isotropic Gaussian path-noise magnitude; λ = 0.05 and β = 0.01 perform best among the variants tested.Noise preserves the update architecture while enabling controlled local exploration around the deterministic trajectory.

6. Experiments

Experiments show that weight tying and training interventions transform feedforward models into scalable iterative reasoners. Shaped attractor landscapes improve accuracy, depth-and-breadth scaling, convergence-based selection, and compute efficiency.

  • Iterative construction: Weight tying, long unrolls, hierarchy, and ACT form a monotonic construction path from feedforward predictors to strong iterative models.Weight tying creates iterative capacity, while training strategies are needed to keep late-iteration states task-aligned under finite stability and memory constraints.
  • Landscape shaping: RI+NI training improves Sudoku from 84.8 to 86.4 and Maze from 44.9 to 82.2 at the base compute budget.RI alone raises Maze accuracy to 68.6, and the gains hold on both training and evaluation splits.
  • Test-time scaling: At B=1, increasing depth from D=16 to D=64 raises EqR from 86.4 to 93.0 on Sudoku and from 82.2 to 88.9 on Maze.Combining D=64 with B=128 further increases accuracy to 99.8 on Sudoku and 93.0 on Maze.
  • Adaptive computation: After landscape shaping, EqR+ACT reaches a matched 92.99% accuracy target with 11.34× fewer NFEs than the baseline.At D=1024, Avg. NFE drops from 1024.0 to 58.7 while accuracy changes from 96.1 to 95.3.
  • Convergence selection: Convergence-based Top-1 selection becomes compute-efficient after landscape shaping but can underperform majority voting for baseline TRM spurious attractors.Its reliability depends on residual convergence tracking task error rather than serving as a task-agnostic certificate.

7. Related Work

Prior work has developed iterative weight-tied and equilibrium models, including fixed-point formulations and path-independence diagnostics. Recent studies apply iterative latent computation to reasoning and explore both increased recurrent depth and adaptive test-time compute allocation.

  • Iterative weight-tied models: Iterative weight-tied models repeatedly apply an update operator to refine latent states, while Deep Equilibrium Models define representations as fixed points.Representative examples include the Universal Transformer and related variants.
  • Iterative weight-tied models: Path-independent equilibrium models use convergence to the same fixed point across trajectories or initializations as a diagnostic for reliable test-time refinement.The paper relaxes global uniqueness by studying whether finite rollouts and restarts concentrate around solution-aligned attractors.
  • Iterative weight-tied models: Recent weight-tied models make iterative latent computation an active scaling direction for language and visual reasoning, while HRM works report strong performance on complex structured tasks.The cited literature spans multiple recent weight-tied models and the HRM series.
  • Compute allocation in weight-tied models: Test-time compute allocation in weight-tied models concerns how many iterations to use, where to allocate them, and whether budgets should be uniform or input-adaptive.Related work studies recurrent-depth scaling and adaptive mechanisms such as token-wise depth.

8. Conclusion … A.1. Attractor Formulation and Residual Diagnostics

The paper frames scalable test-time reasoning through task-conditioned attractors, whose reachability and stability determine whether additional depth and breadth improve reliability. Its appendix formalizes fixed-point residual diagnostics, clarifying both when residual indicates correctness and how ill-conditioned dynamics can destabilize implicit gradients.

  • 8. Conclusion: Attractor alignment with the task metric and reachability from diverse initial states explain when depth and breadth improve iterative reasoning.Randomized initialization and path noise reshape the latent landscape, increasing coverage and stability of correct attractors on Sudoku and Maze.
  • Appendix: The appendix organizes additional material on qualitative studies, dataset and task definitions, and method and experimental details.The listed materials include benchmark specifications, attractor-landscape analyses, architecture and hyperparameters, feedforward learning-rate control, and evaluation metrics.
  • A. Additional Results, Analyses, and Findings: The appendix extends the main findings with residual diagnostics, training-dynamics ablations, stochasticity interventions, generalization experiments, and seed-stability analyses.These analyses span feedforward and weight-tied iterative models and examine initialization and path stochasticity.
  • A.1. Attractor Formulation and Residual Diagnostics: The attractor formulation treats the supervised latent state as a fixed point of an input-conditioned solver, with problem data available at every iteration.This differs from Universal Transformer recurrence, where token embeddings initialize the recurrent state and a shared block iterates over it.
  • A.1. Attractor Formulation and Residual Diagnostics: Inside a locally stable basin with L < 1, small fixed-point residual implies closeness to the local attractor.The bound is ∥z − z⋆∥ ≤ ∥Rθ(z; x)∥/(1 − L).
  • A.1. Attractor Formulation and Residual Diagnostics: Residual indicates correctness only with local stability, a correct attractor, and positive output margin; otherwise, low residual certifies convergence but not correctness.A spurious or low-margin attractor can therefore produce a converged yet incorrect state.
  • A.1. Attractor Formulation and Residual Diagnostics: Exact implicit gradients through long attractor solvers can be unstable when the resolvent (I − Jz)^−1 is poorly conditioned.Small parameter changes may then cause large attractor shifts, while lower-level solve errors can be amplified in the implicit gradient.

A.2. From Feedforward Models to Iterative Models: Training-Dynamics Ablations and Diagnostics

The ablations show that iterative reasoning benefits from shared parameters, greater depth, and training procedures that preserve alignment with the current latent trajectory. Segmented online training and learned halting substantially improve performance, while stale or oracle supervision can degrade it.

  • Weight-tied iterative models: 2.6%→32.6%: weight-tied iteration substantially improves Sudoku over the vanilla feedforward baseline under matched layer-evaluation budgets.The weight-tied model also achieves better training and evaluation error than the feedforward depth baseline at its best-evaluation checkpoint.
  • Depth scaling: 32.6%→51.3%: doubling iteration depth improves Sudoku, but longer trajectories make full-gradient training memory-prohibitive.Deeper iterative models trade slightly lower training accuracy for substantially higher evaluation accuracy under similar layer-evaluation budgets.
  • Trajectory supervision: 51.8%→47.1%: offline trajectory supervision hurts when anchors include early transient states, whereas late anchors can improve performance.Early states may conflict with the rollout, while late states are collected after residual changes have largely stabilized.
  • Segmented online training: 47.1%→74.7%: segmented online training improves Sudoku over 16× trajectory supervision by interleaving parameter and latent-state updates.Later trajectory segments are therefore generated by the current operator rather than stale parameters.
  • Hierarchy and halting: 84.8%: learned ACT raises hierarchical Sudoku performance above the corresponding single-latent z variant at 73.9%, while oracle halting collapses it to 13.6%.Halting signals thus shape training dynamics rather than serving only as an efficiency mechanism.

A.3. Additional Experiments on Randomized State Initialization

Ablations support simple randomized state initialization over the tested learnable initializer, while showing that noise scale materially affects TRM accuracy. The results are limited by single-run evaluations and incomplete exploration of alternative initialization designs.

  • Learnable initial state: The tested learnable initializer reaches 83.99% exact accuracy at 50k steps, below TRM + RI at 86.03% and near the TRM baseline at 84.06%.The initializer is an input-conditioned 2-layer MLP trained jointly with the model.
  • Learnable initial state: Across checkpoints through 50k steps, the learnable initializer never exceeds TRM + RI and its best checkpoint remains below the TRM baseline’s 50k result.The authors therefore avoid stronger claims about learnable initialization without broader ablations.
  • Randomness scale: 86.29% exact accuracy results from randomizing only zH, versus 84.06% with deterministic initialization; randomizing only zL reaches 86.25%.These settings use σH = 1, σL = 0 and σH = 0, σL = 1, respectively.
  • Randomness scale: 87.30% exact accuracy is achieved at σH = 1 and σL = 8, compared with 86.03%, 86.83%, and 86.85% at σL = 1, 4, and 16.The joint sweep demonstrates that the choice of noise scale matters.
  • Scope and limitations: Single-run evaluations provide no variance estimates, and denser noise-scale sweeps and broader studies of initialization variability remain future work.The section also does not analyze alternative initialization parameterizations, architectures, objectives, or regularization strategies.

A.4. Additional Experiments on Path Stochasticity … C. Dataset Details and Task Definitions

The appendix finds that fixed Gaussian path noise supports scalable stochastic reasoning, while EqR generalizes across tasks, backbones, seeds, and sampling budgets. Qualitative trajectories revise and sometimes overwrite earlier Sudoku predictions, and the appendix defines the dataset conventions underlying these experiments.

  • A.4. Additional Experiments on Path Stochasticity: Step-wise Gaussian noise injects path stochasticity into the iterative update, with ε_k ∼ N(0, I) by default.The update is z_k+1 = z_k + (1 − λ) r_θ(z_k; x) + β ε_k.
  • A.4. Additional Experiments on Path Stochasticity: Increasing stochastic restarts S improves performance in both noise settings, while fixed β=0.01 slightly overtakes learned noise at the largest sampling budgets.Learned noise helps at smaller budgets, but the authors retain fixed Gaussian noise as the default because its gains are more consistent as S increases.
  • A.5. Generalization Beyond the Main Setting: 55.28% exact accuracy on Mini-ARC exceeds HRM (44.85%) and TRM (48.35%), showing EqR improves over both baselines beyond the main settings.The experiments also test transfer from the Sudoku MLP-token-mixer backbone to a self-attention Transformer backbone.
  • A.5. Generalization Beyond the Main Setting: 99.8% inference-scaled accuracy follows training-intervention gains from 84.1% to 86.4% on the MLP-token-mixer backbone.For the Transformer backbone, the corresponding values are 72.0%, 74.7%, and 95.9%.
  • A.6. Seed Stability Diagnostics: 86.18 ± 0.44% exact accuracy exceeds the baseline’s 84.33 ± 0.59% at 50k training steps across five seeds.The reported 95% CIs are [85.63, 86.72] for EqR and [83.59, 85.07] for the baseline, with slightly lower seed-to-seed variation for EqR.
  • B. Qualitative Study: A TRM Sudoku-Extreme trajectory revises both correct and incorrect cells, with decoded values alternating across candidates before settling into a consistent solution.The trajectory has length 32, and one representative cell oscillates between 2 and 6, with 3 appearing once at step 8.
  • B. Qualitative Study: The qualitative trajectory exhibits “erase then retry” behavior rather than strictly sequential algorithmic solving.Partial revisions overwrite earlier choices as the latent state evolves.
  • C. Dataset Details and Task Definitions: The dataset appendix distinguishes the original Maze-1k benchmark from the uniquely solvable Maze-Unique setting and defines the task conventions used throughout.These definitions specify the dataset variants underlying the experiments.

C.1. Dataset and Benchmark Specifications · C.2. Datasets and Task Definitions Shape Attractor Landscapes · D. Method and Experimental Details

The benchmarks distinguish ambiguous from uniquely solvable tasks, revealing that label alignment is necessary for stable attractor dynamics and meaningful test-time scaling. The section also specifies the datasets and experimental details used to reproduce and interpret the results.

  • C.1. Dataset and Benchmark Specifications: Sudoku-Extreme is a challenging 9 × 9 benchmark designed to stress long-horizon constraint satisfaction.It remains difficult for strong reasoning models, including DeepSeek-R1 and Claude 3.7 8k.
  • C.1. Dataset and Benchmark Specifications: 2048 cases sampled uniformly from Sudoku-Extreme form the Sudoku-Lite evaluation subset for more efficient evaluation.Across model variants, Sudoku-Lite accuracy is slightly worse, so the smaller subset is not an easier benchmark.
  • C.1. Dataset and Benchmark Specifications: Maze-1k evaluates shortest-path prediction on 30 × 30 grids, but many instances have multiple shortest paths despite providing one labeled path.Maze-Unique instead retains perfect-maze instances with a unique shortest path and contains 1,000 training instances.
  • C.1. Dataset and Benchmark Specifications: Solver-based Sudoku-Extreme difficulty ratings do not fully predict model success, because models solve some extremely difficult instances while failing on easier puzzles.The main-text shorthand uses Sudoku for Sudoku-Extreme and Maze for Maze-Unique.
  • C.2. Datasets and Task Definitions Shape Attractor Landscapes: Ill-defined training targets prevent reliable attractor learning: iterative models on Maze-1k lack stable test-time scaling and remain unstable despite extensive tuning.The root cause is label ambiguity rather than model capacity or optimization.
  • C.2. Datasets and Task Definitions Shape Attractor Landscapes: Single-trajectory token-level cross-entropy supervision misaligns Maze-1k training with its one-to-many task by treating one arbitrary attractor as the sole target.The task admits multiple correct outputs, while the loss penalizes alternatives.
  • C.2. Datasets and Task Definitions Shape Attractor Landscapes: Maze-Unique aligns supervision with task structure, recovering stable attractor dynamics and meaningful test-time scaling as NFE increases.Maze-1k-trained models stay flat or slightly degrade when depth is scaled at test time.
  • D. Method and Experimental Details: The implementation, hyperparameter, baseline-tuning, and evaluation details support reproducing and interpreting the experimental results.These details are collected in this section.

D.1. Architecture and Hyperparameters … D.4. Compute Accounting for Iterative Inference

The appendix specifies task-specific architectures, optimizer settings, feedforward learning-rate controls, evaluation metrics, and the accounting used to quantify iterative inference depth and breadth. It also explains how convergence, restart stability, and equivalent-layer budgets are measured across trajectories.

  • D.1. Architecture and Hyperparameters: Sudoku-Extreme follows the TRM architecture and training procedure, while Maze-Unique reduces layers from 2 to 1 and hidden dimension from 512 to 128 to expose iterative refinement.Larger Maze-Unique models approach near-perfect performance, making test-time-scaling effects harder to observe.
  • D.1. Architecture and Hyperparameters: Training uses Adam-atan2 with learning rate 10^-4, weight decay 1.0, EMA ratio 0.999, 2k-step warmup, and (β1, β2) = (0.9, 0.95).The reported result for each model and dataset uses the best checkpoint observed during training.
  • D.2. Learning-Rate Control for Feedforward Baselines: A sweep over learning rates {5 × 10^-4, 5 × 10^-5, 1 × 10^-4} does not eliminate the feedforward generalization gap, despite training-set fitting.The 4-layer MLP changes from underfitting at the smallest learning rate, but evaluation accuracy remains extremely low across the sweep.
  • D.3. Evaluation Metrics: Averaged exact accuracy measures performance over B independent restarts, reducing to standard single-run accuracy when B = 1.This metric formalizes breadth-scaled evaluation across independent trajectories.
  • D.3. Evaluation Metrics: Top-1 convergence accuracy selects the restart with strongest final-state convergence using mean residual over the last L iterations, then checks prediction correctness.Unless otherwise stated, the convergence window is L = 3 iterations.
  • D.3. Evaluation Metrics: Majority vote accuracy aggregates predictions across B independent restarts, while path independence measures sensitivity to restart randomness; smaller ∆PI(B) indicates greater stability.The path-independence measure uses mean exact accuracy over B independent restarts for each input.
  • D.4. Compute Accounting for Iterative Inference: Inference budgets count outer-loop iterations, with D denoting trajectory depth, B denoting independent restarts, and NFE = D · B function evaluations.Each function evaluation corresponds to one outer-loop iteration.
  • D.4. Compute Accounting for Iterative Inference: 42 equivalent layers per Sudoku-Extreme outer iteration yield 43,008 layers for D = 1024; two-axis scaling with D = 64 and B = 128 yields 344,064 total layers without 344k-depth trajectories.The latter setting uses 8192 function evaluations, while each individual trajectory remains depth D = 64.

E. Extended Related Work and Discussion

This section situates Equilibrium Reasoners among implicit equilibrium models and latent reasoning approaches while distinguishing their focus on reliable, solution-aligned attractors. It also connects the framework to analyses of iterative computation and flat-minima robustness by translating stability from parameter space to latent-state space.

  • Deep Equilibrium Models: DEQs define representations as fixed points, whereas Equilibrium Reasoners study when latent dynamics converge reliably to correct solutions rather than spurious or unstable attractors.The key distinction is between using equilibrium convergence as a representation-learning or training device and shaping a landscape with large, stable, solution-aligned basins.
  • Latent reasoning models: Latent reasoning models organize extra latent computation vertically through repeated weight-tied updates and along a second axis described in the related work.The passage identifies repeated latent-state updates as the vertical axis and frames latent reasoning as computation before externally visible outputs.
  • Theoretical and mechanistic analyses of weight-tied iteration: Theoretical work on weight-tied iteration progresses from computational capacity, to the value of additional iterations and preserved state, to mechanisms governing iterative success or failure.The cited progression encompasses programmable computation, resource accounting, and hidden-state structure.
  • Theoretical and mechanistic analyses of weight-tied iteration: Mechanistic studies report systematic generalization, depth extrapolation, overthinking, cyclic trajectories approaching distinct fixed points, and stabilizing attention as iterations proceed.These findings motivate analyzing latent-state geometry rather than only capacity, scaling, memory, or stopping rules.
  • Flat minima: The framework transfers flat-minima robustness from parameter space to state space, seeking attractors stable to latent-state perturbations rather than weights θ.Training path stochasticity plays a role analogous to SAM’s perturbation step by encouraging a smooth, robust attractor landscape.
Loading 2605.21488v1…