PaperlayerView exact revision on arXiv

Source-linked AI summary

Random Recursive Models

Jama Hussein Mohamud, Mirco Ravanelli

arXiv:2610.00541v1cs.LGcs.AI

TL;DR

Most recursive models repeatedly apply one transformation or a prescribed sequence, limiting them to fixed layer order. RRM samples reusable layers at each step; across five tasks it matches or exceeds corresponding baselines, and test-time inference methods further improve performance without retraining.

  • Problem

    Most recursive models repeat one learned transformation or a prescribed sequence, restricting computation to a fixed layer order.

    Source: §1 · p. 1

  • Method

    RRM samples reusable learned layers independently with replacement at each recursive step, separating the number of layers from the number of applications.

    Source: §1 · p. 1

  • Results

    Across all five tasks, the strongest RRM configuration matches or exceeds corresponding baselines, with roughly 50–75% fewer parameters in controlled image classification and autoregressive logic puzzles.

    Source: §1 · p. 1

  • Takeaways & Limitations

    Test-time depth can be varied without changing model size, and Monte Carlo inference and probabilistic test-time scaling improve performance without retraining.

    Source: §1 · p. 2, §3.2 · p. 4

    Why random layer reuse is effective remains unclear, and its behavior in larger models and tasks remains for future study.

    Source: §7 · p. 10

Abstract

from arXiv · show

Recursive models create computational depth through parameter reuse, offering a parameter-efficient alternative to increasing model size. However, most recursive models repeatedly apply one learned transformation or a prescribed sequence of transformations, restricting computation to a fixed layer order. We introduce the Random Recursive Model (RRM), which maintains a pool of $L$ learned layers and performs $T$ recursive steps by sampling one layer independently with replacement for each example and step. This enables flexible layer reuse while retaining the parameter efficiency of recurrence. We evaluate RRM on challenging reasoning tasks, where it matches or exceeds the baselines, often with 50-75 % fewer parameters. RRM can vary its depth at inference, including beyond that seen during training, without retraining or adding parameters, improving tasks that benefit from deeper iterative computation. RRM also supports Monte Carlo inference and probabilistic test-time scaling, both of which improve performance without retraining. These insights may open new directions in neural network architecture design.

1 INTRODUCTION

Most recursive models follow a fixed layer order, while the Random Recursive Model (RRM) samples reusable learned layers at each step. Across five tasks, RRM matches or exceeds corresponding baselines, and additional inference methods can improve performance without retraining.

1 INTRODUCTION

  • Recursive models reuse parameters across depth, separating model size from computation and allowing many computational steps without a distinct learned layer at each step.This includes computation allocated through test-time scaling.Source: §1 · p. 1
  • RRM replaces fixed-order recurrence with a pool of learned layers that can be reused at any point in the computation.At each recursive step, it samples one layer uniformly and independently with replacement for each example; L determines model size while T determines computational depth.Source: §1 · p. 1
  • Across all five tasks, the strongest RRM configuration matches or exceeds its corresponding baseline, with roughly 50–75% fewer parameters in controlled image classification and autoregressive logic puzzles.On more difficult reasoning tasks, RRM matches or exceeds existing methods without structured, hierarchical, or prescribed structure.Source: §1 · p. 1
  • RRM samples from the complete set of learned layers at every step, allowing layer reordering and reuse.Figure 1 compares four applications under fixed-order depth, fixed-recursive depth, and RRM.Source: §1 · p. 1, Figure 1 · p. 2
  • Monte Carlo inference and probabilistic test-time scaling both improve performance without retraining; probabilistic test-time scaling achieves the best result among the three evaluated models.Additional applications can improve tasks that benefit from longer iterative computation, while gains plateau or performance can degrade when test-time depth extends far beyond training depth.Source: §1 · p. 2

2 BACKGROUND

Fixed-order and fixed-recursive models both prescribe transformation order, though they differ in parameter sharing and how depth is constructed.

2 BACKGROUND

  • Conventional feed-forward and modern recursive networks both prescribe the transformation sequence before seeing an example, despite differing in parameter sharing.The background frames RRM against this shared fixed-order property.Source: §2 · p. 2

2.1 FIXED-ORDER DEPTH

  • A fixed-order network maps an encoded input through an ordered composition in which every example applies each layer once in the same order.Each layer has its own parameters and predetermined position; increasing depth ordinarily adds computation and parameters.Source: §2.1 · p. 2

2.2 FIXED-RECURSIVE DEPTH

  • A fixed-recursive model creates depth by repeatedly applying a learned transformation or prescribed sequence of transformations.Parameter sharing decouples computational depth T from model size, but the recursive computation still follows a prescribed order.Source: §2.2 · pp. 2–3

3 RANDOM RECURSIVE MODELS

RRM recursively applies randomly selected layers from a reusable backbone pool. It trains over sampled layer sequences and separates the number of learned layers from the number of applications, enabling computation scaling without changing model size.

3 RANDOM RECURSIVE MODELS

  • RRM retains the encoder, learned backbone layers, and task-specific head, but samples one layer uniformly at each of T steps instead of applying layers in a prescribed order.Sampling is independent across examples and steps and is with replacement, so a layer may be reused or omitted; fixed-order and fixed-recursive depth are special cases of RRM.Source: §3 · p. 3
  • At recursive depth T, RRM admits LT possible layer sequences, preserving parameter-efficient depth while giving each step access to multiple learned transformations.This supplies diverse computational trajectories without an additional learned mechanism.Source: §3 · p. 3

3.1 TRAINING

  • RRM minimizes task loss averaged over dataset examples and uniformly sampled length-T layer-index sequences.The objective is estimated with one independently sampled sequence per example per forward pass, and gradients update only the function applied at each step.Source: §3.1 · p. 3

3.2 TRAINING- AND TEST-TIME SCALING

  • The number of learned layers L determines backbone size, while T determines effective computational depth and can be varied during training or inference without adding layer parameters.Increasing test-time depth allocates more inference computation with the same parameters; RRM also supports Monte Carlo inference and probabilistic test-time scaling.Source: §3.2 · pp. 3–4

4 EXPERIMENTAL SETUP

The experiments replace established fixed-order backbones with RRM while retaining task formulations and core components. They evaluate controlled image classification, logic puzzles, and more difficult reasoning tasks under established protocols.

4 EXPERIMENTAL SETUP

  • Across tasks, experiments retain established task formulations, data splits, model components, and evaluation procedures while replacing the fixed-order backbone with RRM.Layer indices are sampled independently per example and application; reported RRM results generally average three evaluation seeds for a fixed checkpoint.Source: §4 · p. 4
  • CIFAR-10 compares an eight-layer fixed-order Vision Transformer with RRM using two reusable Transformer layers, varying training and test applications and evaluating Monte Carlo inference.The study measures test-set top-1 accuracy and uses k-NN classification and t-SNE to examine representations across test depths; the figure reports increasingly class-specific representations as applications increase.Source: §4 · p. 4, Figure 2 · p. 5
  • Sudoku and Zebra use the strongest solver-ordered formulation, with an eight-layer causal Transformer baseline and RRM changing only the backbone composition.RRM uses Ttrain = 16 and comparisons share a 320,000-step training budget; Sudoku evaluates L = 2 and 4, while Zebra uses L = 2 and also tests inference-depth scaling.Source: §4 · p. 4
  • Sudoku-Extreme, Maze-Hard, and ARC-AGI-1 follow HRM and TRM experimental setups to test whether simple random layer reuse supports useful recursive computation.Maze-Hard uses two non-causal Transformer layers; Sudoku-Extreme and ARC-AGI-1 add deep supervision, with test-time depth varied for Sudoku-Extreme and applications varied for ARC-AGI-1.Source: §4 · pp. 4–5

5 RESULTS

Across the evaluated tasks, RRM matches or exceeds baselines, often with substantially fewer parameters. Its test-time scaling benefits depend on the task and depth, while probabilistic scaling also improves results without retraining.

5.1 CONTROLLED TOY IMAGE CLASSIFICATION

  • On CIFAR-10, two-layer RRM stays close to the eight-layer dense baseline at roughly one quarter its parameters; M = 16 Monte Carlo inference closes the gap, and Ttrain = 8 slightly surpasses it.Increasing Ttrain to 16 provides no consistent benefit.Source: Table 1 · p. 5, §5.1 · p. 5

5.2 LOGIC PUZZLES AS LANGUAGE MODELING

  • RRM matches or improves on fixed-order dense baselines for Sudoku and Zebra; L = 2 uses roughly one quarter as many parameters on both, and improves Zebra accuracy.On Sudoku, L = 4 exceeds the baseline using about half as many parameters.Source: §5.2 · p. 5, Table 2 · p. 6

5.3 EXTREME REASONING TASKS

  • RRM exceeds TRM-Att on Maze-Hard at Ttest = 24 and closely matches TRM at 5M parameters and 672 layer applications on Sudoku-Extreme.TRM+RRM improves on the published TRM result and the authors’ dense reproduction; increasing basic RRM to Ttest = 32 produces the strongest Sudoku-Extreme result without retraining.Source: Table 3 · p. 6, §5.3 · p. 6, Table 4 · p. 6
  • RRM outperforms the reported TRM baseline on ARC-AGI-1 and remains competitive with 16× fewer supervision steps at inference; deep supervision was important on Sudoku-Extreme and ARC-AGI-1.Without deep supervision, RRM memorized Sudoku-Extreme training data and generalized poorly.Source: §5.3 · p. 6, Table 5

5.4 PARAMETER AND COMPUTE EFFICIENCY

  • RRM matches or outperforms baselines with substantially fewer parameters; it is competitive or better in inference compute on three of four reasoning tasks, with largest gains on Maze-Hard and ARC-AGI-1.Zebra gains accuracy at higher compute, so parameter efficiency does not necessarily imply compute efficiency.Source: §5.4 · pp. 6–7, Figure 3 · p. 7

5.5 TEST-TIME SCALING

  • L = 1 remains effective on CIFAR-10 and Sudoku-Extreme, but L = 2 improves both tasks; this comparison does not isolate random ordering from added model capacity.Figure 4 reports that L = 2 matches or exceeds the baselines on both tasks.Source: Figure 4 · p. 7, §5.4 · p. 7
  • Depth changes have task-dependent effects: Maze-Hard improves beyond training depth before plateauing, while CIFAR-10 can degrade when shallow-trained models are extended far beyond it.Sudoku-Extreme improves monotonically over tested depths; on Zebra, accuracy rises toward training depth but does not improve monotonically with further computation.Source: §5.5 · pp. 7–8, Figure 5 · p. 8
  • Probabilistic test-time scaling improves accuracy across three architectures without retraining, with RRM achieving the highest result on a fixed 1,000-puzzle subset.The evaluation used 100 trajectories per puzzle.Source: Table 6 · p. 8, §5.5 · p. 9

6 RELATED WORK

The related work distinguishes approaches that scale parameter capacity from those that scale computation through parameter sharing. It situates RRM among recursive and randomized methods while emphasizing its stepwise random layer reuse.

6.1 SCALING CAPACITY WITHOUT PROPORTIONAL INCREASE IN COMPUTATION

  • Capacity-scaling methods activate subsets of a large parameter pool, whereas RRM changes reuse order at fixed computational depth and can pair with mixture-of-experts layers.This contrasts increasing available parameters with reordering a fixed set of learned layers.Source: §6.1 · p. 9

6.2 SCALING COMPUTATION WITH SHARED PARAMETERS

  • Shared-parameter recursive methods repeatedly apply one transformation or a fixed sequence; RRM instead samples each application from a pool of learned layers.Both approaches share parameters across depth, but differ in how the sequence of transformations is chosen.Source: §6.2 · p. 9

6.3 RECURSIVE REASONING AND TEST-TIME SCALING

  • Recursive reasoning can use additional inference steps to tackle harder instances, but prior work also reports overthinking when recurrence extends too far beyond training.RRM studies random layer reuse as a simpler alternative to prescribed recurrent structures.Source: §6.3 · p. 9

6.4 LAYER ORDER AND RANDOMIZED NETWORKS

  • Unlike layer omission, permutation, or graph sampling, RRM samples a layer at each step, allowing arbitrary order and reuse without learning or searching for a route.Its sampling with replacement per example and step also allows inference depth to differ from the number of learned layers.Source: §6.4 · p. 10

7 CONCLUSION AND FUTURE WORK

RRM replaces fixed-order recurrence with random reuse of learned layers, matching or exceeding baselines while often reducing parameter count. The paper identifies open questions about why this works and how layer sampling and model scale affect it.

7 CONCLUSION AND FUTURE WORK

  • RRM matches or exceeds corresponding baselines, often with substantially fewer parameters.The conclusion attributes this pattern to random layer reuse acting as effective regularization that improves generalization while mitigating overparameterization.Source: §7 · p. 10
  • Separating learned-layer count from application count lets RRM scale computation without increasing model size.Additional inference applications improve Maze-Hard and Sudoku-Extreme, but gains plateau or eventually decline on tasks requiring less computation.Source: §7 · p. 10
  • Monte Carlo inference and probabilistic test-time scaling provide further improvements without retraining.Source: §7 · p. 10
  • The reasons random layer reuse is effective remain unclear, and its theoretical foundations are left for future work.Future work also includes studying sampling distributions, sequence diversity, larger models and tasks, and learned ordering.Source: §7 · p. 10

A.1 DEEP SUPERVISION

Deep supervision helps Sudoku-Extreme and ARC-AGI-1 training optimization and appears to drive much of their performance. Its carried state, stopped gradients between supervision steps, and early-exit mechanism also support probabilistic test-time scaling.

A.1 DEEP SUPERVISION

  • Deep supervision helps Sudoku-Extreme and ARC-AGI-1 optimization and seems to be a main performance driver.RRM appears to benefit from deep supervision while retaining simpler, more flexible layer reuse.Source: §A.1 · p. 15
  • A single hidden state is carried across supervision steps, with a task loss applied after each step and gradients stopped between steps.The next supervision step continues from the preceding state, but gradients do not cross between supervision steps.Source: §A.1 · p. 15
  • Early exiting uses the Q head to select the best candidate and is very helpful for probabilistic test-time scaling (Sghaier et al., 2026).Source: §A.1 · p. 15

A.2 EXPERIMENTAL DETAILS

The appendix specifies task datasets, model configurations, training protocols, and inference settings for the experiments, including baseline application counts and gradient truncation.

CIFAR-10

  • CIFAR-10 compares an eight-block dense ViT with RRM using one or two reusable layers, trained from scratch for 800 epochs and evaluated by test top-1 accuracy.The ViT uses 32 × 32 images split into 4 × 4 patches, width 256, eight attention heads, and no dropout; training uses batch size 512, Adam, and data augmentation.Source: §A.2 · p. 15

Sudoku and Zebra

  • Sudoku and Zebra use released solver-ordered datasets and an eight-layer causal Transformer; models train for 320,000 optimizer steps.Sudoku uses the complete released training and test sets; Zebra uses the complete training set and a fixed 15,000-puzzle test subset.Source: §A.2 · p. 15

Maze-Hard and Sudoku-Extreme

  • Maze-Hard and Sudoku-Extreme follow the HRM and TRM setups, and their test splits also serve for periodic checkpoint evaluation because no validation split is provided.Each has 1,000 training and 1,000 test examples or puzzles; both backbones have width 512, with task-specific Transformer block designs.Source: §A.2 · p. 15

Baseline application counts

  • Baseline application counts are T = 24 for HRM and T = 30 on Maze-Hard and ARC-AGI-1 and T = 42 on Sudoku-Extreme for TRM.The counts include applications across HRM's recurrent modules and TRM's low-level and answer updates.Source: §A.2 · p. 15

Gradient truncation

  • Truncating gradients through the first 7 of 21 training applications improves Sudoku-Extreme optimization while reducing training computation.Source: §A.2 · p. 16

ARC-AGI-1

  • ARC-AGI-1 RRM uses L = 2, trains with Ttrain ∈ {8, 15} and up to 32 supervision steps, and is evaluated at Ttest ∈ {8, 15, 21}.The Ttrain = 15 setup also tests Tdetach ∈ {0, 5}; one inference evaluation uses two supervision steps instead of the 32 used during training.Source: §A.2 · p. 16

A.3 LEARNED POLICIES FOR ADAPTIVE ORDERING

The appendix probes learned routing and test-time inference and scaling, then reports task-dependent depth behavior and ARC-AGI-1 ablations.

A.3 LEARNED POLICIES FOR ADAPTIVE ORDERING

  • The learned router samples layers from hidden-state-conditioned πϕ using REINFORCE, and an exit logit can shorten computation, but its tested policy had lower accuracy and dense-model-level cost.Concentrating routes on a subset of layers could make unused layers candidates for removal.Source: §A.3 · p. 16

A.4.1 MONTE CARLO INFERENCE

  • Monte Carlo inference estimates predictions across independent sampled layer sequences; more samples reduce variation but require additional forward passes without changing model parameters.A single sample uses one random computation.Source: §A.4.1 · p. 16

A.5 ADDITIONAL CIFAR-10 RESULTS

  • CIFAR-10 accuracy improves from four to eight training applications, then plateaus as training depth increases.The comparison trains separate L = 2 RRMs and evaluates each at its training depth using M = 16 sampled sequences.Source: Figure 6 · p. 17, §A.5 · p. 17
  • The stable test-time range depends on training depth: the Ttrain = 8 model deteriorates beyond Ttest = 12, while the Ttrain = 16 model is stable from Ttest = 16 to 24.Neither model improves consistently simply by running longer.Source: Figure 6 · p. 17, §A.5 · p. 18
  • Monte Carlo inference with M = 16 raises CIFAR-10 accuracy at each tested depth but does not prevent degradation far beyond training depth.Averaging multiple layer sequences therefore does not make additional test-time applications consistently beneficial.Source: Figure 6 · p. 17, §A.5 · p. 18

A.4.2 PTRM TEST-TIME SCALING

  • PTRM preserves complete candidate solutions and selects one using its Q-head score rather than averaging predictive distributions; the method requires no retraining.The evaluation uses a fixed subset of 1,000 test puzzles because the method is expensive.Source: §A.4.2 · p. 17

A.6 ADDITIONAL ARC-AGI-1 RESULTS

  • 16× fewer total applications let RRM still outperform the reported TRM baseline when Nsup falls from 32 to 2.At equal inference cost, the ARC-AGI-1 RRM differentiating through all applications performs best among the ablations.Source: Table 8 · p. 18, §A.6 · p. 18

A.5 ADDITIONAL CIFAR-10 RESULTS

  • The CIFAR-10 depth findings do not generalize uniformly: Maze-Hard and Sudoku-Extreme improve beyond training depth, while Zebra and ARC-AGI-1 largely plateau.The useful test-time range depends on task and training setup, and additional recurrence can eventually be unhelpful or detrimental in some settings.Source: §A.5 · p. 18
Loading 2610.00541v1…