Source-linked AI summary
Reasoning on the Simplex: Geometric Fixed-Point Models
Talgat Daulbaev, Ilya Glazkov, Maxim Rakhuba, Ivan Oseledets
TL;DR
Looped reasoners spend test-time compute by repeatedly applying a weight-tied map, but a small residual in unconstrained latent space need not mean the state is a fixed point. GFPR iterates categorical beliefs as predictions and can impose task structure through convex constraints. At similar scale it exceeds published single-trajectory results on several reasoning tasks, and the same update is evaluated as a language model.
Problem
In looped reasoning, a small residual in an unconstrained latent state does not establish that the iterate is a reachable fixed point.
Source: §1 · p. 1Method
GFPR iterates a field of categorical beliefs whose argmax is the answer, with task structure imposed through convex readouts or recurrent-state constraints.
Source: §1 · pp. 1–2, §3.4 · p. 5Results
Takeaways & Limitations
A compact convex belief state makes fixed-point existence a consequence of the update design, while training determines which equilibrium is reached.
Source: §6 · p. 9Baselines were quoted rather than rerun with matched training and inference budgets, and the language-model comparison is not compute-matched and evaluates likelihoods rather than generated text.
Source: §6 · p. 9
Abstract
from arXiv · showhide
Looped reasoners spend test-time compute by iterating a weight-tied map, but a small residual does not mean the state is a fixed point when that map lives in unconstrained latent space. We propose Geometric Fixed-Point Reasoning (GFPR), in which the iterated state is the prediction itself: a field of categorical beliefs on a product of simplices, whose argmax is the answer at every step. Because the state is a belief, task structure can be imposed through compact convex relaxations, either as structured readouts or directly in the recurrent state; in the latter case the update remains a continuous self-map, so a fixed point exists for any parameters. At about 7M parameters, GFPR reaches 95.1% exact match on Sudoku-Extreme, 92.0% on Maze-Hard, and 100% sequence accuracy on S_5 length 128, above the published FPRM numbers at the same scale. The same update also trains a 201M language model on FineWeb-Edu in which each site is a distribution over the vocabulary; with 24 Picard steps it is above GPT-2 small on four zero-shot multiple-choice tasks and above GPT-2 medium on ARC-Easy.
1 Introduction
Looped reasoning adds test-time computation by repeatedly applying a weight-tied map, but a small residual in unconstrained latent space does not by itself certify a fixed point. GFPR instead iterates categorical beliefs that are themselves predictions, allowing task structure to be imposed on the state or its readout.
1 Introduction
- The motivation is that chain-of-thought computation is tied to autoregressive text, adds latency with each token, and provides no grid-shaped state for local constraints.Harder benchmark instances often require more internal computation to solve reliably.Source: §1 · p. 1
- A small residual certifies an answer only if the looped map has a reachable fixed point, yet the cited FPRM checkpoint halts at a state that is not one.The update is also typically not locally contractive long after halting.Source: §1 · p. 1
- GFPR replaces the latent iterate with a field of categorical beliefs, whose argmax gives the answer at each step.Each site represents a local output such as a grid digit or token, with auxiliary registers for internal state.Source: §1 · pp. 1–2, Figure 1 · p. 2
- Task-specific convex structure can be imposed through structured readouts or directly in the recurrent state, where a continuous self-map on a compact convex set guarantees a fixed point.Which fixed point the iteration reaches is determined by training.Source: §1 · p. 2
- 95.1% exact match on Sudoku-Extreme and 92.0% on Maze-Hard are reported for GFPR at about 7M parameters, versus 94.2% and 87.0% for FPRM.GFPR also exceeds published single-trajectory numbers at similar scale on Sudoku-Extreme, Maze-Hard, and S5.Source: §1 · p. 2
2 Background: Looped Reasoning as Fixed-Point Iteration
Looped reasoning can be viewed as fixed-point iteration, but existence, convergence, and a small halt residual are distinct properties. The paper contrasts fixed-point guarantees with empirical behavior of FPRM and GFPR, including a single-puzzle case study.
2 Background: Looped Reasoning as Fixed-Point Iteration
- Damped iteration updates u by Tβ(u) = (1 − β)u + βFθ(u; x), while fixed points satisfy u⋆ = Fθ(u⋆; x).FPRM halts on a small relative iterate residual with decaying step size; TRM and HRM use ACT or a step cap, while DEQs solve implicitly.Source: §2 · p. 2
- A contraction guarantees a unique fixed point and global convergence, whereas a continuous self-map of a compact convex set guarantees existence but not uniqueness or convergence.For trained networks, local attraction depends on the damped Jacobian eigenvalues; non-normality can make residuals non-monotone.Source: §2 · pp. 2–3
- On one puzzle solved by both models, GFPR reaches float64 residual precision, while FPRM remains near 10^-2 at its k = 36 halt and has a largest Ritz modulus of about 1.04.For that puzzle, GFPR’s largest Ritz modulus is approximately 1.3 × 10^-3.Source: Figure 2 · p. 3, §2 · p. 3
- A small relative residual does not ensure FPRM has reached a fixed point: on 1,000 held-out Sudoku-Extreme puzzles, 89% are solved, but no halted state has residual below 10^-3.The median relative RMS residual is 3.1 × 10^-2.Source: §2 · p. 3
- On 75% of 128 held-out puzzles, the 8000-step FPRM probe reports local amplification above 1, a diagnostic rather than a population claim about the Jacobian spectral radius.The authors have not cross-checked that fraction with Arnoldi.Source: §2 · p. 3
3 Method
GFPR iterates categorical beliefs on a product of simplices, so each state is itself a prediction and known input values remain pinned. Training shapes the trajectory and task structure through convex constraints, with different procedures for token generation and structured reasoning tasks.
3.1 Belief state and update
- A continuous update on the pinned product of simplices has a fixed point for every θ, x, and β, while each belief’s argmax is already the current prediction.The residual therefore measures prediction movement, and known outputs are represented as fixed one-hot sites.Source: §3.1 · pp. 3–4
3.2 Training and inference
- Training differentiates a rollout tail from a state reached by a no-gradient rollout, supervising the end state and aligning the learned map with inference’s uniform-start trajectory.Sudoku also uses a residual loss and randomized rollout caps; Maze sets the residual weight to zero, and some test-time behavior beyond training rollouts remains extrapolation.Source: §3.2 · p. 4
3.3 Language models: wavefront decode
- For causal language modeling, wavefront decode commits tokens left to right while allowing a soft-ahead window of later beliefs to update before their prefix is fixed.Readiness uses a top-two probability gap, a persistently small total-variation residual, or a per-token step cap; teacher-forced training does not train on expected embeddings.Source: §3.3 · pp. 4–5
3.4 Task-specific convex constraints
- Compact convex relaxations impose task structure either as terminal structured readouts for Sudoku and Maze-Hard or as the recurrent state for S5.The S5 recurrent update remains in the convex set; Sudoku constraints encode row, column, and box digit mass, while Maze flow constraints alone do not guarantee a simple or shortest path.Source: §3.4 · pp. 5–6
4 Experiments
The experiments evaluate GFPR on three structured reasoning benchmarks and a language-modeling task, comparing it with looped reasoners and GPT-2 models. GFPR exceeds published single-trajectory baselines on the structured benchmarks and, after iteration, reaches GPT-2 small or medium performance on several language-model tasks.
4.1 Tasks
- The benchmarks measure exact-match answers on Sudoku-Extreme, Maze-Hard, and S5 state tracking, alongside zero-shot language-model multiple choice.Sudoku uses a public 422,786-puzzle test file; Maze-Hard uses 1,000 test mazes, and S5 tests generalization to sequences of up to 128 updates.Source: §4.1 · p. 6
4.3 Results
- 95.1% of Sudoku-Extreme puzzles and 92.0% of Maze-Hard mazes are solved by GFPR, versus 94.2% and 87.0% for FPRM; GFPR reaches 100.0% sequence accuracy on S5 length 128.The S5 model has 5.6M parameters; its training sequences contain at most 32 updates, while evaluation reaches length 128.Source: Table 1 · p. 7, §4.3 · p. 7
4.2 Baselines
- At a similar parameter scale, comparisons use quoted published baseline accuracies rather than reruns; fair comparison also requires matching inference budgets, and training recipes differ, so Table 1 alone does not isolate state-space effects.For example, GFPR uses dihedral augmentation on Maze-Hard, whereas FPRM does not.Source: §4.2 · p. 7
4.3 Results
- On S5, stopping when the largest per-site total-variation step falls below 5 · 10^-3 takes 81 steps on average, and every test sequence remains correct.The step cap is 256.Source: §4.3 · p. 7
4.4 Language-model results
- GFPR-LM scores rise from one to 24 Picard steps on ARC-Easy, SciQ, PIQA, and HellaSwag, with most improvement arriving between K=8 and K=16.The model is a 201M-parameter FineWeb-Edu run; its comparison is a proof of concept rather than a competitive language-model claim.Source: §4.4 · pp. 7–8
- At 24 Picard steps, GFPR-LM exceeds GPT-2 small on all four tasks and GPT-2 medium on ARC-Easy, but remains below GPT-2 medium on SciQ, PIQA, and HellaSwag.The comparison is not compute-matched; scores are based on answer-choice likelihoods rather than generated text.Source: Table 2 · p. 8, §4.4 · p. 8
5 Related Work
The paper situates GFPR among recurrent neural networks and weight-tied reasoners. Its distinction is that the iterated categorical-belief state is itself the output and can carry task structure.
Recurrent and looped models
- Unlike looped Transformers, HRM, and TRM, which iterate free latents and decode symbols with a separate head, GFPR iterates beliefs that are directly readable as predictions.The Universal Transformer also ties weights across depth, while looped Transformers reinject the input and raise the loop count at test time.Source: §5 · p. 8
Fixed-point reasoning
- FPRM and EqR use fixed-point iteration as a reasoning procedure, but their state remains an unconstrained vector for which the architecture does not guarantee a fixed point.FPRM damps Picard steps and halts on a small latent residual; EqR can spend extra compute on restarts.Source: §5 · p. 8
Recurrent and looped models
- Hopfield networks iterate an output state, while Almeida–Pineda networks inject input at each step; GFPR combines depth recurrence with a learned categorical-belief field as its output.Classic graph neural nets constrain transitions to contractions and still use a separate decoder.Source: §5 · p. 8
Language models
- On language tasks, GFPR iterates a belief over the vocabulary at each site, unlike chain-of-thought methods that spend compute by writing tokens.A standard Transformer visits each block once, while latent looped language models iterate a hidden state.Source: §5 · p. 8
6 Conclusion
GFPR iterates predictions represented as categorical beliefs, allowing task structure to be imposed as a convex set and guaranteeing fixed-point existence for continuous updates on that set. The conclusion reports gains on structured benchmarks and a small language-model proof of concept, while noting important comparison limits.
6 Conclusion
- The iterated state is the prediction, and task structure can be imposed on it as a convex set; a fixed point exists for any parameters, though training determines which one is reached.The conclusion describes fixed-point existence as a consequence of the design, not its purpose.Source: §6 · p. 9
- GFPR exceeds published single-trajectory results on Sudoku-Extreme, Maze-Hard, and S5 at similar scale; its language-model experiment shows the same update can work with vocabulary distributions.The unit-flow readout raises Maze-Hard from 92.0% to 96.0%, while the Sudoku consistency readout matches rather than improves on the product-of-simplices baseline.Source: §6 · p. 9
- The evidence does not isolate gains due to the state space because baseline training recipes and inference budgets are unmatched; the language-model comparison is also not compute-matched and uses likelihood scores.A few Sudoku puzzles settle to stable but incorrect grids; the authors suggest constraint-violation-based restarts as a possible next step.Source: §6 · p. 9
A FPRM and GFPR side by side
- FPRM decodes a grid from a head on a latent state, whereas GFPR reads the grid from its iterated belief after pinning clues; the figure’s bottom row describes a loop property, not another update.FPRM’s halted state need not be a fixed point, while GFPR’s compact belief state guarantees fixed-point existence.Source: Figure 3 · p. 12, A FPRM and GFPR side by side · p. 12
B Model and training settings
The released GFPR runs use a shared Transformer trunk with self-conditioning passes and a two-phase rollout. Settings vary by task, with S5 adding a causal depthwise convolution before operation embeddings.
B Model and training settings
- Both released runs use the same Transformer trunk over all sites, with each self-conditioning pass taking the previous pass’s softmax output as extra input.The final pass produces the output of Fθ, and logits are soft-capped as 15 tanh(ℓ/15) before each softmax.Source: B Model and training settings · p. 12
- Training uses a no-gradient prefix followed by a differentiated tail, with the loss applied to the final state.The prefix stops early after two consecutive steps below the per-site TV tolerance; Sudoku draws its depth randomly, while Maze uses a fixed depth.Source: B Model and training settings · p. 12
- Among released runs, S5 alone applies a causal depthwise convolution of kernel size 4 to the state before adding operation embeddings; the other runs do not.Table 3 defines K as the number of output symbols and A as the number of auxiliary coordinates. Rollout depth is the no-gradient prefix; D̄=32 is the log-normal Poisson draw in the main text.Source: Table 3 · p. 13
C Evaluation protocol
Evaluation uses task-specific test sets and iteration budgets, with Sudoku assessed on a separate public test release from checkpoint selection. Maze evaluation also checks whether settled parent fields trace connected routes.
C Evaluation protocol
- 95.11% exact match and 96.93% cell accuracy are reached on the 422,786-puzzle Sudoku-Extreme test file.The public test file is separate from checkpoint selection; evaluation uses a single trajectory and at most 35,000 damped Picard steps.Source: C Evaluation protocol · p. 13
- 92.0% exact match and 99.67% cell accuracy are reached on all 1,000 Maze-Hard test mazes; 99.7% of settled parent fields trace a connected route.Evaluation uses exactly 1,024 damped Picard steps.Source: C Evaluation protocol · p. 13
- 100% sequence accuracy and 100% final-state accuracy are reached on all 1,000 S5 test sequences of length 128, with a mean of 81 steps to tolerance.Inference allows at most 256 damped Picard steps from the uniform state.Source: C Evaluation protocol · p. 13
- Maze parent fields are evaluated against the dataset generator’s recovered neighbor-expansion order, matching the labeled route when that order reproduces it.Most Maze-Hard instances have several shortest routes, so the route comparison uses the same route scored by the published numbers.Source: C Evaluation protocol · p. 13
D Task-specific convex sets
The task-specific convex sets encode Sudoku constraints and Maze flow constraints, while the structured map can be used recurrently or as a terminal readout. These relaxations can contain fractional feasible points, so feasibility alone does not guarantee a discrete solution.
Task-specific convex sets
- Sudoku’s convex set enforces probability vectors per cell, one occurrence of each digit per row, column, and box, and fixed clue coordinates.Every valid completed grid is an integral feasible point, but fractional feasible points need not be valid discrete completions.Source: D Task-specific convex sets · p. 14
- The Sudoku equality constraints are solved through a numerical convex dual, with the primal point recovered from the dual solution.Numerical feasibility is measured by the equality residual ∥AF qF(λ) − ebx∥.Source: D Task-specific convex sets · p. 14
- The choice of Entmax order α modestly shifts the iteration transition, while final-accuracy curves nearly coincide after sufficient iteration.Figure 4 plots S5 final-state accuracy against recurrent iterations for different α values; Table 4 reports Sudoku-Extreme exact-grid accuracy by α and iteration budget.Source: D Task-specific convex sets · p. 14, Figure 4 · p. 15
- The Maze flow polytope imposes conservation away from the goal and start, with net outflow one at the goal and net inflow one at the start.Feasible flows may be fractional or contain circulations, so feasibility alone does not imply a simple, shortest, or unique path.Source: D Task-specific convex sets · p. 14
- The structured map can be applied at every recurrent step or only to the terminal score vector, with different gradient paths through the solver.In the terminal-readout construction, gradients do not differentiate through numerical iterations used to compute the structured prediction.Source: D Task-specific convex sets · p. 14
E Simplex language-model decode
The language-model implementation uses tied-weight Picard steps and wavefront decoding for open-ended generation. Its reported FineWeb-Edu multiple-choice scores instead use fixed-K teacher-forced unrolling.
E Simplex language-model decode
- Table 4 reports Sudoku-Extreme exact-grid accuracy across Entmax orders α and recurrent iteration budgets.The caption identifies the table’s metrics and comparison dimensions but gives no cell values.Source: Table 4 · p. 15
- The byte-pair TinyStories model uses a causal Transformer with weights tied across Picard steps and trains by a relax-plus-tail unroll.Next-token cross-entropy is applied to the output block.Source: E Simplex language-model decode · p. 15
- Open-ended generation uses wavefront decoding with soft-ahead width W=8 and damping β=0.5, without a key–value cache.Each Picard step is a full forward pass over the active length, so iteration counts describe this decoder rather than a cached autoregressive Transformer.Source: E Simplex language-model decode · p. 15
F FineWeb-Edu language-model evaluation
The FineWeb-Edu evaluation varies Picard depth for a 201M GFPR language model and specifies its training and zero-shot scoring setup. Most of the reported depth-sweep gain occurs between K=8 and K=24.
F FineWeb-Edu language-model evaluation
- The evaluated GFPR-LM has 201M parameters and trains on 6,000,148,480 FineWeb-Edu tokens for 36,622 steps.It uses 16 no-gradient steps followed by an 8-step differentiated tail; reported scores use EMA weights.Source: F FineWeb-Edu language-model evaluation · p. 15
- Zero-shot scoring uses next-token log-likelihood over answer choices, with a fixed iteration count K and a uniform initial state.Metrics are accuracy on ARC-Easy, SciQ, and PIQA, and length-normalized accuracy on HellaSwag.Source: F FineWeb-Edu language-model evaluation · p. 15
- Most of the depth-sweep gain appears between K=8 and K=24, while K=24→32 does not improve all four tasks.The authors use K=24, the training depth; these scores do not show that the simplex state has reached a fixed point.Source: F FineWeb-Edu language-model evaluation · p. 15, Table 5 · p. 15
- GPT-2 large is 774M parameters and is omitted from the main comparison because it is several times larger than the 201M GFPR-LM.Its scores on the same four tasks are 53.0 / 79.7 / 70.1 / 45.3.Source: F FineWeb-Edu language-model evaluation · p. 16
G FPRM diagnostic protocol
The diagnostic compares residuals and local amplification along FPRM and GFPR trajectories, then audits held-out puzzles. FPRM’s halt leaves a non-small residual and amplification above one in the reported probe, while the population probe is explicitly not a certified spectral-radius test.
G FPRM diagnostic protocol
- The single-puzzle trace uses FPRM’s public Sudoku checkpoint and evaluation loop, while GFPR starts from the uniform state with β=0.7.The FPRM run uses float64 for 80 steps, step-size decay 0.997, and patience 10.Source: G FPRM diagnostic protocol · p. 16
- At FPRM’s halt, the residual is 3.2·10^-2 and the local amplification probe is about 1.03; GFPR’s residual falls below 10^-12 by step 27.For this one puzzle, GFPR’s final probe value is about 1.4 × 10^-3, while FPRM’s residual remains between 1.3·10^-2 and 3.1·10^-2 after halting.Source: G FPRM diagnostic protocol · p. 16
- The probe uses Jacobian–vector products, but finite probe depth can reflect transient amplification for a non-normal Jacobian and need not certify its spectral radius.The figure’s right-hand trace reports bα(uk) with Kpi=20.Source: G FPRM diagnostic protocol · p. 16
- In the 128-puzzle population audit, 75% of final states have bα(u)>1, but Arnoldi was not run on those states.The authors therefore do not treat this fraction as certified evidence that ρ(JF)>1 on 75% of puzzles.Source: G FPRM diagnostic protocol · p. 16