PaperlayerView exact revision on arXiv

Source-linked AI summary

Recursive Inference Scaling: A Winning Path to Scalable Inference in Language and Multimodal Systems

Ibrahim Alabdulmohsin, Xiaohua Zhai

arXiv:2502.07503v4cs.AIcs.LG

TL;DR

Prior recursive-language-model comparisons did not explicitly account for recursion's added training compute, leaving the source of reported gains unclear. RINS recursively applies an early network block and, in compute- and parameter-matched comparisons, improves language and multimodal performance; lightweight adapters with stochastic recursion also support gains when recursion is off at inference.

  • Problem

    Prior recursive architectures were compared at similar token counts without explicitly accounting for the added training compute, leaving unclear whether their gains came from recursion or greater compute.

    Source: §1 · p. 2

  • Method

    RINS scales inference by recursively applying an early portion of an existing network to refine its output, with stochastic recursion and lightweight linear adapters as an extension.

    Source: §1 · p. 2

  • Results

    RINS outperforms +55 other methods in compute- and parameter-matched comparisons and improves multimodal performance, including SigLIP-RINS-B/16 0-shot ImageNet accuracy from 77.3% to 79.6%.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    Stochastic RINS with linear adapters comprising less than 1% of parameters improves language modeling even when recursive depth is not applied at inference.

    Source: §1 · p. 2

    Monotonic improvement with more inference calls is not guaranteed; repeated sampling may favor the most probable rather than optimal answer, especially when correctness is below chance.

    Source: §7 · p. 9

Abstract

from arXiv · show

Inspired by recent findings on the fractal geometry of language, we introduce Recursive INference Scaling (RINS) as a complementary, plug-in recipe for scaling inference time in language and multimodal systems. RINS is a particular form of recursive depth that significantly outperforms +55 other variants, including the recent "repeat-all-over" (RAO) strategy in Mobile LLM (Liu et al., 2024) and latent recurrent thinking (Geiping et al., 2025). Unlike prior works, we carry out our comparisons on a compute-matched regime, and demonstrate that for a fixed model size and training compute budget, RINS substantially improves language modeling performance. It also generalizes beyond pure language tasks, delivering gains in multimodal systems, including a +2% improvement in 0-shot ImageNet accuracy for SigLIP-B/16. Additionally, by deriving data scaling laws, we show that RINS improves both the asymptotic performance limits and the scaling exponents. More importantly, with light-weight (linear) adapters (comprising <1% of model parameters) and stochastic dropout, RINS offers a no-regret strategy, meaning that RINS-enabled pretraining improves performance in language modeling even when recursive depth is not applied at inference time. This corresponds to improving performance on a training compute-, parameter-, and inference-matched regime, suggesting its potential as a viable component of LLM pretraining!

1 Introduction

The introduction presents RINS as recursive inference-time scaling inspired by language self-similarity, and argues that its gains persist under compute- and parameter-matched comparisons. It also highlights multimodal improvements and a stochastic, lightweight-adapter approach that can improve language modeling even without recursion at inference.

1 Introduction

  • RINS outperforms +55 other methods, including RAO and latent recurrent thinking (Geiping et al., 2025), in compute- and parameter-matched comparisons.The authors note that for models over 1 billion parameters, RAO's gains can be matched by training the baseline longer at equivalent compute.Source: §1 · pp. 2–3
  • RINS recursively applies an early network block to its own output before passing that output to the next block, using language's self-similar structure as motivation.The paper describes this as a complementary approach to scaling inference time.Source: §1 · p. 2, Figure 1 · p. 2
  • With stochastic recursion and linear adapters comprising less than 1% of parameters, RINS-enabled pretraining improves language modeling even when recursion is omitted at inference.The authors call this a no-regret strategy and report minimal performance degradation when forgoing the extra inference computation.Source: §1 · pp. 2–3
  • RINS improves accuracy without increasing parameter count at the same training compute, and retains an advantage with KV-cache sharing despite some diminished gain.The paper emphasizes this setting for compact models intended for memory-constrained deployment.Source: §1 · p. 2
  • SigLIP-RINS-B/16 raises 0-shot ImageNet accuracy from 77.3% to 79.6% and CIFAR100 accuracy from 70.3% to 80.7%.The introduction presents these as multimodal gains beyond language modeling.Source: §1 · p. 3
  • The derived data scaling laws show improved asymptotic performance limits and scaling exponents for RINS.The introduction also reports minimal gains from a similar analysis in vision, supporting its proposed language-specific rationale.Source: §1 · p. 3

2 Recursive Inference Scaling

The section frames recursive architectures as a fixed-compute search over parameter-sharing patterns and identifies RINS as the best-performing pattern. RINS recursively applies an initial block before the remainder, motivated by language’s self-similar structure.

Taxonomy

  • For a fixed block set and compute budget, one recursive architecture is “better” when its training cost stays within budget and its expected loss is no higher.The comparison considers forward-pass FLOPs and treats lower performance-metric values as better.Source: §2 · p. 3

Motivation

  • Recursive depth increases inference computation by iteratively applying layers to their outputs without changing the underlying architecture, complementing approaches such as chain-of-thought prompting and repeated sampling.The approach is motivated by language’s self-similar patterns across levels of granularity.Source: §2 · p. 3
  • Figure 2 reports that RINS’s performance advantage grows with longer training, while the 1,536-token long-sequence baseline performs worse because it processes fewer examples at matched FLOPs.The figure describes language models trained on 200B tokens and measures training cost in layer × step units.Source: Figure 2 · p. 4

Taxonomy

  • RINS, defined as degree 1 with signature ArB for r > 1, emerges as the clear winner among the explored recursive architectures under matched training FLOPs.The authors hypothesize that its performance reflects language’s self-similar geometry.Source: §2 · p. 4
  • RINS partitions model depth into equally sized blocks A and B, applies A recursively to its own output r times, then applies B.The general architecture search varies parameter-sharing signatures and recursion degree; signature notation records block reuse, while degree indicates nested recursion.Source: §2 · p. 4

3 Experimental Results

In compute-matched language-model experiments, RINS outperforms the other tested approaches, with its advantage increasing as training continues. The evaluation compares models trained under matched FLOPs and also examines longer training runs.

Setup

  • The experiments train 300M- and 600M-parameter models on C4 and SlimPajama, matching recursive models to the baseline’s compute by training them on fewer tokens.The baseline trains for 200K steps, about 200B tokens; the study reports training 59 models, with RAO and latent recurrent thinking among the architectures tested.Source: §3 · pp. 4–5

Results

  • RINS outperforms the other approaches at matched training compute and parameter count, with recursive models overtaking the baseline after its initial advantage saturates.RINS also outperforms scaling inference time by increasing context length alone.Source: §3 · p. 5

Longer Training Duration

  • At 500B training tokens, the performance gap continues to widen in RINS’s favor over the non-recursive baseline and RAO, and the improvement persists downstream.This longer-duration comparison uses 600M-parameter models.Source: §3 · p. 5

4 Stochastic Recursive Inference Scaling

Stochastic RINS varies recursion depth during training, allowing inference-time depth to be chosen later. Increasing the training skip probability reduces the penalty when recursion is not used at inference, but also makes inference scaling less effective.

Method

  • During training, stochastic RINS samples recursion rounds as 1 + η, where η is binomial with r − 1 trials and success probability 1 − ps; block A always runs at least once.At inference, r ≥ 1 can be chosen to set compute; the reported comparison uses A3B against the non-recursive baseline in 1B-parameter models.Source: §4 · p. 5

Results

  • For ps = 0, RINS@1x has perplexity scores > 3; as ps →1, its performance converges to the baseline.Figure 4 varies inference cost for 1B-parameter language models and plots against training compute cost.Source: Figure 4 · p. 6
  • As skip probability increases, stochastic RINS reduces performance degradation when inference scaling is omitted while retaining gains from scaling, though those gains become smaller.Similar conclusions hold in the asymptotic regime under a power-law loss assumption.Source: §4 · p. 6

5 Multimodal Systems

The multimodal experiments compare recursive SigLIP models with same-size baselines under matched training compute. RINS outperforms the baselines in zero-shot and retrieval evaluations, including in overtraining experiments.

Setup

  • The initial vision-language comparison matches total training compute, using SigLIP-B/16 against RAO and RINS with degree 1.SigLIP-B/16 is a contrastive model trained on English-language image–text pairs; recursive models use fewer training examples to match the baseline's compute.Source: §5 · p. 6

Results

  • RINS with signature A2B outperforms the non-recursive baseline and RAO in zero-shot and retrieval evaluations, including about +0.7% on ImageNet 0-shot classification.The comparison includes long-sequence evaluation; the compared SigLIP-B/16 models have identical size and training compute FLOPs.Source: §5 · p. 6, Table 2 · p. 7

Overtraining

  • The overtraining comparison uses 40B examples from the same English-and-multilingual data mixture for identical-size recursive and baseline SigLIP-B/16 models.The recursive model uses signature A3B; the baseline is retrained on the same data mixture for an appropriate comparison.Source: §5 · p. 7
  • Across all benchmarks, SigLIP-RINS-B/16 significantly outperforms standard SigLIP-B/16, and stochastic RINS with skip probability ps = 1/4 improves results even further.Source: §5 · p. 7

Data Scaling Laws

  • For models with signature ArB, RINS (r > 1) improves scaling exponent c and asymptotic log-perplexity ε∞ relative to baseline r = 1; overtraining the baseline cannot match its performance gain.Source: §5 · p. 8

6 Further Analysis

RINS improves scaling behavior and benefits compact models, while supervised image classification shows no advantage for recursion. Linear adapters enable a no-regret training option, and KV cache sharing limits memory growth with depth while diminishing gains.

Vision

  • Recursive architectures, including RINS, provide no advantage in supervised image classification on ImageNet, consistent with the paper's hypothesis linking RINS gains to language's self-similarity.The vision evaluation uses encoder-only ViTs on ImageNet-ILSVRC2012.Source: §6 · p. 8

Optimal Number of Recursion Rounds

  • The optimal recursion count increases monotonically with training compute, and smaller language models benefit more from RINS.The comparison plots optimal r against training compute for 300M-, 600M-, and 1B-parameter models.Source: §6 · p. 8

Adding Linear Adapters

  • Linear adapters add fewer than 1% of parameters and make stochastic RINS a no-regret strategy, improving performance even when recursion is disabled at test time.For signature ArB, the adapter associated with the number of recursion rounds is applied before the projection head.Source: §6 · pp. 8–9

Adding KV Cache Sharing

  • KV cache sharing prevents memory footprint from growing with recursive depth, although it diminishes RINS's gain; RINS with sharing still beats the baseline.During each use of block A, queries attend to keys and values cached from its first call.Source: §6 · p. 9

7 Discussion and Related Works

The discussion situates RINS among inference-scaling methods and prior recursive architectures, highlighting its reported scaling gains. The pseudocode describes stochastic depth by varying the recursion rounds while guaranteeing execution of the first and last entries.

7 Discussion and Related Works

  • Repeated sampling need not improve outputs monotonically: it may converge to the most probable rather than optimal solution, especially when correctness probability is below 0.5.This qualification is attributed to Chen et al. (2024).Source: §7 · p. 9
  • RINS is reported to outperform other recursive architectures, including latent recurrent thinking and the Repeat All Over strategy identified as state-of-the-art for mobile LLMs.The paper presents RINS as complementary to other inference techniques.Source: §7 · p. 9
  • Unlike earlier parameter-sharing approaches whose scaling exponents were smaller than vanilla Transformers', RINS improves both scaling exponents and asymptotic performance limits.The discussion names Universal Transformers and ALBERT as examples of earlier parameter-sharing designs.Source: §7 · p. 9

A Recursive Inference Scaling (RINS) Pseudocode

  • Stochastic RINS expands ps into a tuple with zero-probability first and last entries, guaranteeing those entries execute while recursion rounds are sampled from a binomial distribution.When stochastic depth is not applied, ps = 0.Source: Figure 7 · p. 14

B Retrieval and Multilinguality Detailed Results

This appendix section points readers to detailed retrieval and multilinguality results for SigLIP-RINS-B/16, including per-language Crossmodal-3600 retrieval evaluations.

B Retrieval and Multilinguality Detailed Results

  • The section directs readers to full retrieval and multilinguality results for SigLIP-RINS-B/16.The supplied text does not include the result values.Source: B Retrieval and Multilinguality… · p. 15
  • Table 4 reports per-language retrieval evaluations using the Crossmodal-3600 dataset.The caption names the dataset but supplies no comparative values.Source: Table 4 · p. 15
  • Table 5 covers multilingual SigLIP models across datasets under overtraining, with all models identical in size; its caption says stochastic RINS outperforms the others.The caption directs readers to Appendix B for full retrieval and multilinguality results.Source: Table 5 · p. 16

C Zero-shot Evaluation on Common Sense Reasoning Tasks

The appendix specifies zero-shot common-sense evaluation and discusses recursive depth through a self-similarity hypothesis about language. The evaluation uses six benchmarks and task-specific prompting choices.

C Zero-shot Evaluation on Common Sense Reasoning Tasks

D Recursion and Self-Similarity

  • Self-similarity in temporal language suggests reusing the same block to produce increasingly abstract features, but the authors describe this conceptual account as a hypothesis.They also cite empirical self-similarity in language log-perplexity as evidence consistent with the account.Source: D Recursion and Self-Similarity · p. 18

E.1 Architecture Sweep

The appendix describes an architecture sweep intended to compare similarly sized recursive models and provides training configurations for language, vision, and language-image models.

E.1 Architecture Sweep

  • The architecture sweep varies parameter-sharing signatures and recursive degrees, with the stated goal of comparing architectures of similar size.Source: §E.1 · p. 19

E.2.1 Language Modeling

  • The decoder-only language-model configuration uses a 32,000-token vocabulary, context lengths of 1,024 or 1,536 for the long-sequence baseline, and equally weighted C4 and SlimPajama data.Source: §E.2.1 · p. 20

E.2.2 Vision: Image Classification

  • The ViT configuration specifies 1,000 classes, batches of 1,024, 224-pixel inputs, and MixUp with probability 0.2.Source: §E.2.2 · p. 20

E.2.3 Language-Image Pretraining

  • The SigLIP configuration uses batches of 1,024 × 32, multilingual C4 text, and a maximum text length of 64.Source: §E.2.3 · p. 21

F C4 Evaluation Results with Stochastic RINS

The stochastic RINS results vary with the dropout parameter and inference cost, with performance approaching the baseline as the parameter approaches one.

F C4 Evaluation Results with Stochastic RINS

  • Perplexity scores exceed 3 for stochastic RINS@1x when ps = 0, and performance converges to baseline as ps →1.Figure 9 plots training compute cost on the x-axis and labels each configuration by inference cost relative to baseline.Source: Figure 9 · p. 22

G Vision

The vision evaluation finds no advantage from parameter sharing in supervised image classification. At matched inference cost, the higher-resolution non-recursive baseline outperforms the recursive architectures.

G Vision

  • Recursive ViTs are trained for fewer epochs to match the total training FLOPs of baselines trained for 300 or 1,000 epochs.The setup uses ImageNet-ILSVRC2012, batch size 1,024, MixUp, and learned position embeddings.Source: G Vision · p. 23
  • The A@336 baseline matches recursive models' inference cost by using higher-resolution images, unlike the language-model setting where RINS gains persist against sequence-length scaling.Source: G Vision · p. 23, Table 6 · p. 23
  • Parameter-sharing methods, including RINS, provide no advantage in supervised image classification; baseline A@336 outperforms them at matched inference cost.The table compares recursive architectures with the baseline trained on 224- and 336-resolution images.Source: Table 6 · p. 23, G Vision · p. 23

Related papers

Loading 2502.07503v4…