PaperlayerView exact revision on arXiv

Source-linked AI summary

Recursive Language Models

Alex L. Zhang, Tim Kraska, Omar Khattab

arXiv:2512.24601v3cs.AIcs.CL

TL;DR

LLMs struggle with arbitrarily long prompts because context windows are limited and quality can degrade as prompts grow. The paper introduces RLMs, which externalize prompts into an environment for programmatic inspection, decomposition, and recursive sub-calls. RLMs process 10M+ token inputs and outperform common long-context scaffolds with comparable costs, while small-scale post-training improves RLM-Qwen3-8B.

  • Problem

    Limited context windows and context rot constrain LLM processing of long prompts, even as long-horizon tasks may require tens or hundreds of millions of tokens.

  • Method

    RLMs wrap a base model with an external environment that stores the prompt and supports programmatic examination, decomposition, and recursive sub-calls.

  • Results

    RLMs scale to 10M+ tokens and outperform base models and task-agnostic scaffolds by up to 2× while maintaining comparable or cheaper average token costs.

  • Takeaways & Limitations

    RLMs provide a general inference framework for long-context processing, including information-dense tasks where base models make little progress.

  • Takeaways & Limitations

    Evaluations on more difficult natural long-context tasks and guardrail mechanisms remain underexplored, while added RLM complexity may cause exploding sub-call costs.

Abstract

from arXiv · show

We study allowing large language models (LLMs) to process arbitrarily long prompts through the lens of inference-time scaling. We propose Recursive Language Models (RLMs), a general inference paradigm that treats long prompts as part of an external environment and allows the LLM to programmatically examine, decompose, and recursively call itself over snippets of the prompt. We find that RLMs can successfully process inputs up to two orders of magnitude beyond model context windows and, even for shorter prompts, dramatically outperform the quality of vanilla frontier LLMs and common long-context and coding scaffolds (e.g., on GPT-5 by a median across the evaluated benchmarks of $26\%$ against compaction, $130\%$ against CodeAct with sub-calls, and $13\%$ against Claude Code) across four diverse long-context tasks while having comparable cost. At a small scale, we post-train the first model around the RLM. Our model, RLM-Qwen3-8B, outperforms the underlying Qwen3-8B model by $28.3\%$ on average and even approaches the quality of vanilla GPT-5 on three long-context tasks. Code is available at https://github.com/alexzhang13/rlm.

1 Introduction

The paper addresses context-window limits and context rot by introducing RLMs, which treat prompts as an external environment for programmatic decomposition and recursion. Across diverse long-context tasks, RLMs maintain strong performance at scales where vanilla models degrade, and post-training improves a small recursive model.

  • Motivation and contribution: GPT-5 performance degrades with increasing input length and task complexity, while a depth-1 RLM maintains strong performance beyond GPT-5’s 272K-token context window.Figure 1 compares S-NIAH, OOLONG, and OOLONG-Pairs across input lengths from 2^13 to 2^20.
  • Motivation and contribution: RLMs treat arbitrarily long prompts as an external environment rather than feeding them directly into the Transformer.The LLM symbolically and recursively interacts with prompt content through this environment.
  • Evaluation: RLMs are evaluated against direct calls, compaction, retrieval agents, and coding agents across four long-context task types.The evaluation uses GPT-5 and Qwen3-Coder-480B-A35B.
  • Results: RLMs show strong 10M+ token performance, outperform other long-context approaches with double-digit gains in many cases while maintaining comparable cost.They also exhibit less degradation as contexts lengthen and tasks become more sophisticated.
  • Results: 28.3% median improvement comes from post-training RLM-Qwen3-8B on only 1,000 unrelated-domain samples.The result demonstrates that recursive models can improve quickly with little additional training.

2 Recursive Language Models

An RLM wraps a base model with a persistent REPL that stores the prompt externally, enables programmatic inspection and transformation, and supports recursive sub-calls. Its design keeps prompt content and intermediate results out of the root model’s context history while allowing symbolic management of long computations.

  • Definition: An RLM is an inference-time scaffold around a base model that treats an arbitrary-length prompt as part of a persistent external environment.The framework targets input lengths much larger than the base model’s maximum context K.
  • Core mechanism: The initial prompt is stored in a REPL variable, and the root model receives only constant-size metadata before generating code to inspect and transform it.The code can invoke sub-RLMs on programmatically constructed prompt snippets.
  • Core mechanism: Each loop iteration executes REPL code, updates symbolic state, and appends only bounded metadata about printed output to the model history.The loop returns the value of Final once that variable is set.
  • Design choices: The RLM design differs from weaker scaffolds by keeping prompt data external, managing intermediate values symbolically, and placing recursion inside executable code.These choices avoid copying the prompt into the model context and enable repeated programmatic sub-calls.
  • Design choices: A direct Finish action cannot produce outputs longer than the base model’s context window, while non-programmatic sub-calls permit only a few explicitly verbalized delegations.The paper identifies both as limitations of the alternative scaffold.

3 Scaling Long Context Tasks

The evaluation varies task complexity with context length and compares RLMs with direct models and common task-agnostic scaffolds. Tasks range from constant-information retrieval to linear and quadratic aggregation, with experiments spanning frontier models, coding agents, and fine-tuned recursive models.

  • Task design: Task difficulty is characterized by how processing complexity scales with prompt length, rather than by context length alone.The paper contrasts constant-complexity NIAH with linear OOLONG and quadratic OOLONG-Pairs tasks.
  • Task design: S-NIAH keeps the sought information constant at O(1) relative to input length, whereas OOLONG requires nearly all questions and scales linearly.These tasks expose different long-context scaling behaviors.
  • Task design: OOLONG-Pairs requires aggregating nearly all pairs of dataset entries, creating quadratically many processing items relative to input length.Performance is reported with F1 scores over list-valued answers.
  • Methods and baselines: The study compares GPT-5 and Qwen3-Coder-480B-A35B RLMs with direct models, compaction, retrieval, CodeAct, and coding-agent baselines.RLM implementations use a Python REPL and vary maximum recursion depth from 0 to 3.
  • Methods and baselines: Fine-tuning uses 1,000 filtered trajectories to improve the root model’s ability to manipulate the REPL and launch recursive calls.The training recipe treats effective sub-call modeling as related to general-purpose reasoning.

4 Results and Discussion

RLMs scale long-context processing beyond base model context windows while often improving performance at comparable cost. Results also show that REPL access, recursive sub-calling, and training contribute differently across task complexity, with runtime remaining implementation-sensitive.

  • Overall scaling and efficiency: Up to 2× performance gains are reported over base models and common long-context scaffolds while maintaining comparable or cheaper average token costs.On BrowseComp-Plus (1K), RLM(GPT-5, depth=1) costs $0.99 on average and outperforms compaction and retrieval baselines by over 29%.
  • Information-dense tasks: 58.0% and 23.1% F1 scores on OOLONG-Pairs substantially exceed GPT-5 and Qwen3-Coder base scores of ≤0.1%.The result highlights RLMs’ ability to handle extremely information-dense inputs, including tasks within the models’ context windows.
  • Ablations and mechanisms: The REPL is necessary for handling long inputs, whereas recursive sub-calling provides strong benefits on information-dense inputs.On CodeQA, RLM(depth=0) outperforms all sub-calling variants; on OOLONG-Pairs, higher-depth GPT-5 RLMs outperform other methods by a large margin.
  • Scaling with length and complexity: RLM performance degrades more slowly than GPT-5 as input length and problem complexity increase, consistently outperforming GPT-5 beyond 2^14 context lengths.RLM costs scale proportionally to task complexity while remaining in the same order of magnitude as GPT-5.
  • Long reasoning: Beyond context length, RLM(GPT-5.2, depth=1) outperforms the base model on LongCoT-mini, with explicit decomposition hints increasing performance by 69.5% overall.The hinted RLM generates a problem graph and solves nodes through programmatic sub-calls.
  • Training and generalization: Training on unrelated tasks improves RLM-Qwen3-8B across all evaluated tasks, while RLVR training on a smaller MRCRv2 split generalizes to a longer, harder split.The fine-tuned model also has lower inference costs and runs more than 3× faster because of better decisions and fewer mistakes.

5 Analyses of RLM Trajectories

RLM trajectories probe and decompose long-context tasks, with decomposition quality and syntax errors shaping performance. In-context examples improve initial decomposition and overall OOLONG performance, while model choice affects syntax-error rates and recursion-depth outcomes.

  • Observed RLM decomposition patterns: RLMs probe prompts, decompose tasks into recursive sub-tasks, and can stitch sub-LM outputs beyond the context window.This behavior is required for tasks such as OOLONG-Pairs.
  • Figure 4: Figure 4 varies RLM(GPT-5) in-context examples on OOLONG and compares syntax-error prevalence across correct and incorrect depth-1 rollouts.Panel (a) categorizes first decomposition attempts; panel (b) reports trajectories with at least one syntax error.
  • First decomposition and errors: In-context decomposition examples improve both overall OOLONG performance and the RLM’s initial decomposition, even when unrelated to the task.The first decomposition attempt remains important because later recovery from an incorrect pattern is not always sufficient.
  • First decomposition and errors: Qwen3-Coder RLM trajectories contain significantly more syntax errors than GPT-5 trajectories, including among correct rollouts.These errors help explain why higher recursion depths perform worse on average for RLM(Qwen3-Coder).

6 Related Works

Related work follows two main paths: changing language-model architectures for longer contexts or building scaffolds that manage context externally. RLMs belong to the latter path and combine external-context handling with model-selected recursive calls.

  • Long-Context LM Systems: Long-context systems either retrain architectures for longer contexts or build scaffolds that implicitly manage context; RLMs pursue the scaffold approach.Lossy context management is one class of scaffold-based strategy.
  • Task Decomposition through sub-LM calls: Prior agents use multiple language-model calls, but many place them through human-engineered workflows; other methods defer sub-call selection to the language model.Examples include ViperGPT, THREAD, ReDel, Context Folding, and AgentFold.

7 Limitations and Future Work

RLM evaluations show strong performance beyond existing context limits, but harder natural tasks and effective guardrails remain underexplored. Initial Qwen3-8B evidence suggests explicit RLM training can rapidly improve performance beyond the training domain.

  • Limitations: Evaluations of more difficult and natural long-context tasks and mechanisms for implementing RLM guardrails remain highly underexplored.The paper also identifies complexity from adding RLM layers and possible exploding sub-call costs as an unresolved concern.
  • Future Work: Explicit training as an RLM provides rapid Qwen3-8B performance improvements, including outside the training domain.The experiments provide initial evidence rather than a broad evaluation of natively trained RLMs.

8 Conclusion

The paper concludes that RLMs offload context and enable recursive sub-querying, allowing language models to reason over context through code and recursive calls. The authors position native RLM training as a possible additional scaling axis for future language-model systems.

  • Conclusion: RLMs offload input context and enable language models to recursively sub-query language models before producing an output.The explored implementation stores context in a Python REPL variable.
  • Conclusion: RLMs operate over context through code and recursive language-model calls rather than purely in token space.The authors connect this framework to future training of models that reason as RLMs.

A Additional Training Details

The authors distilled filtered recursive trajectories into RLM-Qwen3-8B and evaluated both its training setup and efficiency. The recipe used limited training resources, while the post-trained model showed improved RLM efficiency.

  • Training recipe: 2250 candidate trajectories were collected from 750 English LongBenchPro tasks before filtering.Trajectories were sampled from Qwen3-Coder-480B-A35B-Instruct and then distilled to Qwen3-8B.
  • Filtering: 1,072 candidate trajectories remained after removing zero-scoring or single-turn trajectories.Each root RLM turn was then separated into an input-output supervised fine-tuning sample.
  • Filtering: Turns exceeding approximately 100k characters were removed, and programmatic corrections fixed small RLM-template mistakes.The corrections addressed errors such as mixing FINAL(answer) with FINAL(variable in REPL).
  • Compute: 300 training steps with batch size 64 required 48 H100 hours using the prime-rl library.The authors describe this as an exceedingly simple recipe and call for larger-scale future training.

B Negative Results: Things We Tried That Did Not Work.

The authors report several failure modes and practical lessons for prompting and recursive execution. Model capability, output limits, asynchronous execution, and prompt calibration all materially affect RLM behavior.

  • Prompting: Using one identical RLM system prompt across models caused undesirable behavior, requiring a Qwen3-Coder-specific warning against excessive recursive sub-calls.Without the warning, Qwen3-Coder attempted thousands of subcalls for basic tasks.
  • Model capability: Models with insufficient coding capabilities struggled to reason through and manipulate context in the REPL environment.Small-scale experiments specifically found smaller models such as Qwen3-8B struggled in this setting.
  • Output limits: Insufficient output tokens caused thinking models to run out of tokens during individual RLM calls.For Qwen3-235B-A22B, this contributed to a smaller performance gap despite positive base-model results, including OOLONG increasing from 30% to 38%.
  • Execution: Sequential blocking sub-LM calls made RLM experiments slow, especially relative to the base model.The authors expect robust asynchronous implementation to resolve this issue.
  • Output control: The FINAL() and FINAL_VAR() protocol made distinguishing a final answer from an intermediate thought brittle for some models.Models sometimes emitted their plan as the final answer, motivating minor safeguards and future native RLM training.
  • Prompt strategies: For large contexts, the prompt examples recommend chunking context, querying sub-LLMs, storing intermediate buffers, and aggregating them into a final answer.The prompts distinguish llm_query for simple subtasks from rlm_query for recursively complex subtasks.

C.3 LongCoT-mini experiment.

The LongCoT-mini experiment uses an RLM workflow that decomposes problems into dependent subproblems, delegates solving, verifies answers, and assembles the result. Decomposition hints alone often confuse the model, although they improve performance on harder math splits.

  • Experimental finding: Decomposition hints often confuse GPT-5 on some LongCoT-mini splits, while improving performance on the more difficult math split.The hints omit sub-calling details and do not provide the REPL-like mechanism used by RLMs.
  • Verification: Every propagated answer is independently checked or tested for plausibility before downstream nodes consume it.Failed checks trigger re-dispatch and re-verification of the affected node.
  • Cycles: Cyclic dependencies are handled by proposing candidate seed values, solving downstream nodes, and checking the cycle constraint.Trials are cached, and answers are frozen once the constraint is satisfied.
  • Assembly: Final answers are assembled from verified values by dictionary lookup rather than recomputation.The workflow can use a sub-LM for aggregation if needed, but the root process performs no mathematical solving.
  • Planning and solving: The workflow first extracts a DAG of self-contained nodes, then solves all dependency-ready nodes in parallel through sub-LM calls.Each subprompt copies the node question and verified parent values, while the plan tracks dependencies and cycles.

D.1 OOLONG-Pairs Benchmark

OOLONG-Pairs evaluates whether models can identify every user-ID pair satisfying semantic and temporal conditions over long question collections. RLMs maintain strong scaling behavior and solve more tasks than competing approaches, though trajectories can use widely varying numbers of sub-calls.

  • Benchmark design: OOLONG-Pairs requires returning all user-ID pairs satisfying semantic and temporal conditions, enforcing quadratic scaling rather than allowing counting shortcuts.The benchmark contains 20 synthetic tasks over contexts ranging from 1,024 to 1,048,576 tokens.
  • Scaling results: RLM(GPT-5) is the only evaluated method maintaining perfect performance at the 1,000-document scale.The no-recursion ablation reaches 90%, while conditioned base GPT-5 variants show performance dropoff as document count increases.
  • Cost: RLM inference cost scales log-linearly and is cheaper than an infinite-context GPT-5 extrapolation in this setup.The reported cost is reasonably bounded relative to strategies such as ReAct + BM25.

E.2 RLM(Qwen3-Coder) on OOLONG-Pairs-Query_3

The Qwen3-Coder trajectory solves an OOLONG-Pairs query by classifying entries recursively and then forming qualifying pairs programmatically. Although the approach finds the answer, repeated verification and regeneration can waste calls and ultimately cause the root model to discard correct intermediate results.

  • Task: The 32k-token OOLONG-Pairs query requires semantically classifying thousands of entries before outputting all qualifying user-ID pairs.The task combines information-dense input with a long-output requirement.
  • Cost: The trajectory costs $0.38 for the OOLONG-Pairs example.The cost is reported for the example whose answer is stored in a variable after programmatic pair construction.
  • Trajectory: Qwen3-Coder classifies input chunks through recursive sub-LM calls, then categorizes the returned labels and constructs unique pairs programmatically.The root LM stores target users and pairs in variables for final output.
  • Failure mode: Repeated regeneration eventually produces a wrong root-LM answer instead of returning the correct answer stored in the code environment.The example is explicitly identified as an RLM failure after five repeated regeneration cycles.
  • Trajectory: Qwen3-Coder can launch thousands of recursive calls because its classification function invokes a sub-LM once per input line.This behavior differs from GPT-5’s more limited use of sub-calls.

F.1 Additional Quantitative Analysis of Main Results

Additional analyses compare which tasks RLMs solve relative to baselines and examine how sub-call counts vary across models and outcomes. These analyses show broad RLM coverage alongside substantial trajectory-level behavioral differences.

  • Fine-grained success: RLMs generally solve the same tasks as, and more tasks than, baseline methods, especially when GPT-5 is the underlying model.Figure 9 separates RLM-only, shared, and baseline-only correct answers.
  • Sub-calling behavior: Sub-call counts vary widely across models and between correct and incorrect RLM trajectories.Figure 10 plots average depth-1 sub-calls by task and correctness.
  • Model differences: GPT-5 uses significantly more sub-calls than other models on BrowseComp-Plus, whereas Qwen3-Coder averages about 500 on correct OOLONG rollouts.The comparison indicates that correctness is not associated with one uniform sub-calling pattern across models.

F.2 Additional Runtime and Cost Analysis of RLMs

RLM cost and runtime are generally comparable at typical cases, but their trajectories can become long-tailed, producing sharp cost and runtime increases at high percentiles.

  • RLM cost trajectories are generally long-tailed and high-variance across both GPT-5 and Qwen3-Coder.The analysis focuses on RLMs with depth=0 and depth=1 and includes per-task cost histograms.
  • 95th-percentile runtime can become extremely long, mainly because sequential sub-LLM calls consume most of the runtime.These cases occur infrequently and can be early-stopped with timeout logic.
  • Asynchrony and prompting that discourages long sub-LM calls or code are identified as ways to improve runtime substantially.The stated runtime improvements target the sequential-call bottleneck and long trajectories.
  • At the 50th percentile, RLMs have comparable or lower API costs than the baselines.Figure 11 compares RLM(depth=1) with baselines at the 25th, 50th, 75th, and 95th cost percentiles.
  • Runtime tail behavior is evaluated through quartile plots across OOLONG, OOLONG-Pairs, CodeQA, and BrowseComp+ (1K).Figures 12 and 13 report the 25th, 50th, 75th, and 95th runtime percentiles for GPT-5 and Qwen3-Coder-480B-A35B-Instruct.
Loading 2512.24601v3…