Less is More: Recursive Reasoning with Tiny Networks

1. Introduction

• TRM recursively updates a latent feature and then revises its answer, using a tiny network to progressively improve predictions.The depicted process starts from embedded question x, answer y, and latent z, and allows up to Nsup = 16 improvement steps.Source: Figure 1 · p. 1, §1 · p. 2
• LLMs using Chain-of-thought and test-time compute still fall short on ARC-AGI, and Gemini 2.5 Pro reaches 4.9% test accuracy on ARC-AGI-2.Chain-of-thought can improve accuracy but is expensive, requires high-quality reasoning data, and can be brittle when generated reasoning is wrong.Source: §1 · pp. 1–2
• An independent ARC-AGI analysis found deep supervision doubled accuracy from 19% to 39%, while hierarchical recursion raised it from 35.7% to 39.0%.The analysis suggests repeated supervision matters more than recursion within each supervision step.Source: §1 · p. 2
• TRM raises test accuracy on Sudoku-Extreme from 55% to 87%, Maze-Hard from 75% to 85%, ARC-AGI-1 from 40% to 45%, and ARC-AGI-2 from 5% to 8%.The approach uses a much smaller network with only two layers.Source: §1 · p. 2

2. Background

• The background section points readers to Algorithm 2 for HRM’s procedure.The paper discusses the algorithm further in the following sections.Source: §2 · p. 2

2.1. Structure and goal

• HRM maps an input to an output of matching [B, L] shape using input embeddings, two recurrent networks, and an output head.Each recurrent network is a four-layer Transformer architecture.Source: §2.1 · p. 2

2.2. Recursion at two different frequencies

• A forward pass alternates two low-frequency and one high-frequency recurrent updates, with early evaluations detached and the final latent features used to predict the output.The approach allows arbitrary numbers of low-level steps and recursions.Source: §2.2 · pp. 2–3

2.3. Fixed-point recursion with 1-step gradient approximation

• HRM assumes its latent features reach a fixed point, then uses the Implicit Function Theorem and a one-step approximation to backpropagate only through the last two of six function evaluations.This reduces memory demands.Source: §2.3 · p. 3

2.4. Deep supervision

• Deep supervision reuses latent features across up to 16 forward passes, letting the model iteratively improve them toward a solution.Each minibatch is used for all 16 supervision steps before moving to the next minibatch.Source: §2.4 · p. 3

2.5. Adaptive computational time (ACT)

• Adaptive computational time (ACT) learns when to halt during training, reducing average Sudoku-Extreme supervision steps to fewer than 2 instead of the full 16.At test time, the full 16 supervision steps are run to maximize downstream performance.Source: §2.5 · p. 3

2.6. Deep supervision and 1-step gradient approximations replaces BPTT

• Deep supervision and one-step gradient approximations provide an alternative to backpropagating through time, with an example configuration yielding 384 effective layers of depth.The reported calculation is 4 × (2 + 1) × 2 × 16 = 384.Source: §2.6 · p. 3

2.7. Summary of HRM

• HRM combines two recurrent networks operating at different frequencies with deep supervision and adaptive computation, but relies on complex biological arguments and fixed-point assumptions that may not apply.The model is described as obtaining strong performance on hard question-answer tasks that regular supervised models struggle with.Source: §2.7 · p. 4

3. Target for improvements in Hierarchical Reasoning Models

• The section frames these HRM design issues as targets for improvement in Tiny Recursion Models.Source: §3 · p. 4

3.1. Implicit Function Theorem (IFT) with 1-step gradient approximation

• HRM's fixed-point premise is doubtful: reported residuals remain above zero, and zL is far from converged where the one-step gradient approximation is applied.The authors note that HRM performs only one fL evaluation before assuming a fixed point for both latent variables.Source: §3.1 · p. 4

3.2. Twice the forward passes with Adaptive computational time (ACT)

• ACT's Q-learning continue loss requires an extra HRM forward pass, so ACT uses two forward passes per optimization step.The extra pass includes all six function evaluations.Source: §3.2 · p. 4

3.3. Hierarchical interpretation based on complex biological arguments

• The authors argue HRM's biological rationale and lack of ablations make it hard to determine which design choices help, including why it uses two latent features.They propose replacing that account with a simpler explanation that does not rely on biological arguments, fixed-point theory, hierarchy, or two networks.Source: §3.3 · p. 5

4. Tiny Recursive Models

• TRM uses one tiny network and one forward pass for ACT, rather than HRM's two networks and two passes.The authors describe the method as requiring no complex theorem, hierarchy, or biological argument.Source: §4 · p. 5

4.1. No fixed-point theorem required

• A full recursion process uses n fL evaluations and one fH evaluation, with gradients propagated through the full process.The authors propose T−1 recursion processes without gradients before one process with backpropagation.Source: §4.1 · p. 5
• 56.5% to 87.4% test accuracy on Sudoku-Extreme follows from using full recursion with multiple gradient-free recursion processes.The approach removes the fixed-point assumption and one-step gradient approximation while retaining recursion before a backpropagated process.Source: §4.1 · p. 6

4.2. Simpler reinterpretation of zH and zL

• The authors reinterpret zH as the current solution and zL as latent reasoning, so each feature retains a distinct role across iterations.The previous reasoning helps carry forward how the model reached its solution; the previous solution keeps that answer separate from the reasoning state.Source: §4.2 · p. 6
• Using only y and z gives better Sudoku-Extreme test accuracy than carrying a single feature or splitting reasoning into more features.The authors report performance drops for both the single-feature and more-than-two-feature variants.Source: §4.2 · p. 6, Table 2 · p. 6

4.3. Single network

• 82.4% to 87.4% test accuracy on Sudoku-Extreme accompanies replacing two networks with one while halving the parameter count.The authors attribute the two networks' distinct roles to whether the input x is included in the update.Source: §4.3 · p. 6

4.4. Less is more

• 79.5% to 87.4% test accuracy on Sudoku-Extreme follows reducing the network from four layers to two and halving parameters again.The authors report that adding layers decreased generalization due to overfitting.Source: §4.4 · p. 7

4.5. attention-free architecture for tasks with small fixed context length

• 74.7% to 87.4% test accuracy on Sudoku-Extreme follows replacing self-attention with an MLP over sequence length.The authors say this works for Sudoku's small fixed context but is suboptimal on Maze-Hard and ARC-AGI, which use 30x30 grids.Source: §4.5 · p. 7

4.6. No additional forward pass needed with ACT

• 86.1% to 87.4% test accuracy on Sudoku-Extreme shows no significant generalization difference after removing ACT's continue loss and second forward pass.The replacement learns a halting probability using binary cross-entropy.Source: §4.6 · p. 7

4.7. Exponential Moving Average (EMA)

• 79.9% to 87.4% test accuracy on Sudoku-Extreme accompanies adding exponential moving average of the weights.The authors report that EMA prevents sharp collapse and improves generalization.Source: §4.7 · p. 7

4.8. Optimal the number of recursions

• T = 3, n = 6 in TRM (42 recursions) and T = 3, n = 3 in HRM (48 recursions) give optimal Sudoku-Extreme generalization in the tested settings.More recursions may help harder problems, but were not tested; increasing n too much in TRM can cause out-of-memory errors.Source: §4.8 · p. 7

5. Results

• The evaluation covers Sudoku-Extreme, Maze-Hard, ARC-AGI-1, and ARC-AGI-2; Sudoku uses 1,000 training samples and 423,000 test samples.Maze-Hard uses 1,000 training and 1,000 test mazes; ARC-AGI-1 and ARC-AGI-2 have 800 and 1,120 tasks, respectively.Source: §5 · pp. 7–8
• Augmentation uses 1,000 shuffling variants per Sudoku example, eight dihedral transformations per maze, and 1,000 ARC variants per example.ARC transformations include color permutations, dihedral-group transformations, and translations.Source: §5 · p. 8
• 87.4% test accuracy on Sudoku-Extreme is the best reported generalization for TRM without self-attention.TRM with self-attention generalizes better on the other tasks.Source: §5 · p. 8, Table 4 · p. 8
• 85.3% on Maze-Hard, 44.6% on ARC-AGI-1, and 7.8% on ARC-AGI-2: 7M-parameter TRM exceeds 27M-parameter HRM's 74.5%, 40.3%, and 5.0%, respectively.These results are reported for TRM with self-attention; Table 4 covers puzzle benchmarks and Table 5 covers ARC-AGI benchmarks.Source: Table 4 · p. 8, Table 5 · p. 8, §5 · p. 8

6. Conclusion

• TRM uses a single tiny network to recurse on its latent reasoning feature and progressively improve its final answer.Compared with HRM, it halves the number of layers, replaces two networks with one, and removes the extra forward pass for halting.Source: §6 · p. 8
• TRM is much simpler than HRM while achieving better generalization; its reported model sizes are 15M on Sudoku and 19M on Maze.Source: §6 · pp. 8–9
• Architecture choices are not guaranteed to work optimally across datasets, and the reason recursion helps remains unexplained.The authors say they suspect overfitting but have no theory to support that explanation; TRM is supervised and returns one deterministic answer.Source: §6 · p. 9

Hyper-parameters and setup

• All models use AdamW, batch size 768, hidden size 512, 16 maximum supervision steps, and stable-max loss.The learning-rate warmup is 2K iterations; TRM uses EMA 0.999, while HRM uses two 4-layer networks and TRM one 2-layer network.Source: Hyper-parameters and setup · p. 11
• Puzzle tasks train for 60K epochs and ARC-AGI for 100K epochs, all at a 1e-4 base learning rate.Weight decay is 1.0 for Sudoku-Extreme and Maze-Hard and 0.1 for ARC-AGI; ARC embeddings use a 1e-2 learning rate.Source: Hyper-parameters and setup · p. 11
• Sudoku experiments ran on one L40S, Maze experiments on four L40S, and ARC-AGI experiments on four H100 GPUs.Reported runtimes are generally under 36 hours for Sudoku, under 24 hours for Maze, and around three days for ARC-AGI.Source: Hyper-parameters and setup · p. 11

Ideas that failed

• Replacing SwiGLU MLPs with mixture-of-experts caused generalization to decrease massively.The authors attribute this to unnecessary added capacity, as with increasing the number of layers.Source: Ideas that failed · p. 11
• Backpropagating through only the last k recursions did not help generalization and made the approach more complicated.The authors report that backpropagating through the full n + 1 recursions makes the most sense and works best.Source: Ideas that failed · p. 11
• Removing ACT and stopping when the solution is reached caused a significant drop in generalization.The authors suggest the model may then spend too much time on the same data samples rather than learning across a wider range.Source: Ideas that failed · p. 11
• Tying the input embedding to the output head was too constraining and caused a massive generalization drop.Source: Ideas that failed · p. 11
• Replacing recursion with fixed-point iteration slowed training and worsened generalization, indicating convergence to a fixed point is not essential.The experiment used TorchDEQ to replace recursion steps with fixed-point iteration.Source: Ideas that failed · p. 11

• The single-z TRM variant is presented as PyTorch pseudocode with deep-supervision training.Source: Figure 4 · p. 12
• Both algorithm versions specify deep-supervision training in PyTorch.Source: Figure 4 · p. 12, Figure 5 · p. 12
• The multi-scale-z TRM variant is presented as PyTorch pseudocode with deep-supervision training.Source: Figure 5 · p. 12

• zH corresponds to the predicted response, whereas zL is a latent feature that cannot be decoded into a sensible output without transformation by fH.The figure shows tokenized zH and zL after reversing the embedding and applying argmax.Source: Figure 6 · p. 12