PaperlayerView exact revision on arXiv

Source-linked AI summary

Reasoning Cache: Continual Improvement Over Long Horizons via Short-Horizon RL

Ian Wu, Yuxiao Qu, Amrith Setlur, Aviral Kumar

arXiv:2602.03773v2cs.LG

TL;DR

Standard RL is limited by fixed distributions and training budgets, making long-horizon test-time extrapolation difficult. The paper introduces Reasoning Cache, which alternates reasoning with summarization and trains models to use summaries across iterations. RC-trained models extrapolate substantially beyond their 16k-token training budget, while the approach remains limited by myopic rewards and untrained summary generation.

  • Problem

    Standard RL operates over fixed prompt distributions and bounded training rollouts, limiting extrapolation when test distributions or reasoning budgets differ.

  • Method

    Reasoning Cache replaces standard decoding with iterative reasoning and summarization, while RL trains models to perform summary-conditioned reasoning.

  • Results

    Accuracy on HMMT 2025 improves from 40% to 70% as the test budget increases from 16k to 512k tokens, and RC-trained models outperform the base model across benchmarks and token budgets.

  • Takeaways & Limitations

    RC enables models to continually improve over reasoning horizons much longer than those used during training and to leverage summaries for later reasoning.

  • Takeaways & Limitations

    The current objective uses myopic per-trace rewards and does not explicitly train summary generation, limiting multi-turn strategies and summary-generation capability.

Abstract

from arXiv · show

Large Language Models (LLMs) that can continually improve beyond their training budgets are able to solve increasingly difficult problems by adapting at test time, a property we refer to as extrapolation. However, standard reinforcement learning (RL) operates over fixed problem distributions and training budgets, which limits extrapolation amidst distribution shift at test time. To address this, we introduce RC, an iterative decoding algorithm that replaces standard autoregressive decoding during both training and inference. RC exploits an asymmetry between the response generation and summarization capabilities of LLMs to construct reasoning chains that consistently improve across iterations. Models trained to use RC can extrapolate and continually improve over reasoning horizons more than an order of magnitude longer than those seen during training. Empirically, training a 4B model with RC using a 16k-token training budget improves performance on HMMT 2025 from 40% to nearly 70% with 0.5m tokens at test time, outperforming both comparably sized models and many larger reasoning LLMs. Finally, we also show that models trained with RC can more effectively leverage existing scaffolds to further scale test-time performance, due to the improved summary-conditioned generation abilities learned through training.

1. Introduction

LLMs may improve through longer test-time reasoning, but standard training limits that extrapolation to fixed distributions and rollout budgets. RC addresses this with iterative summary-conditioned reasoning, producing strong long-horizon gains and better use of external scaffolds.

  • Longer test-time reasoning is intended to let LLMs solve harder problems and continually improve without changing model weights.
  • Standard RL uses fixed prompt distributions and bounded rollouts, encouraging premature termination and exposing longer generations to distribution shift.
  • RC generates reasoning, summarizes it into a cache, discards the original trace, and conditions subsequent reasoning on the summary.
  • RC-trained models show strong and consistent extrapolation, substantially outperforming base and stronger reasoning models on mathematical benchmarks.
  • Optimizing summary-conditioned generation improves both RC use and the ability to use external context for reasoning.

2. Related Work

Prior work scales test-time reasoning through longer traces, modified training, rewards, iterative transformations, or memory. These approaches motivate RC’s focus on extrapolation and its use of summaries as compressed states for self-generated reasoning.

  • Longer autoregressive traces scale test-time compute, but models often cannot extrapolate far beyond lengths seen during RL training.
  • Dataset and curriculum methods enable extrapolation to around 3–4× the training budget, but performance typically saturates beyond this range.
  • Iterative decoding research includes self-correction, self-refinement, parallel-compute scaffolds, and training models to apply transformations more effectively.
  • Prior memory methods store external context, whereas RC summaries store abstractions of self-generated reasoning traces for later steps.

3. Preliminaries and Notation

The paper defines test-time performance as reward-based performance under a token budget and contrasts fixed-budget training objectives with larger-budget, shifted-distribution evaluation. Extrapolation occurs when the larger test budget improves performance.

  • At test time, an autoregressive policy allocates a specified token budget to reasoning, with performance evaluated as the budget increases.
  • Standard on-policy RL optimizes expected rollout reward over a training distribution of prompt-answer pairs.
  • Training rollouts encode reasoning traces and are sampled within a fixed training budget H_train, with rewards based on answer correctness.
  • Test-time evaluation may involve a different prompt distribution and a larger budget H_test than training.
  • A model extrapolates when performance at H_test exceeds its performance at H_train.

4. Problem Statement

Standard RL does not ensure extrapolation because it rewards termination within a fixed training budget and exposes longer reasoning to unseen conditional distributions. RC changes decoding to avoid this shift while making longer-horizon training practical.

  • Positive reward only for rollouts terminating within H_train encourages premature termination and does not optimize performance at larger H_test.
  • Continuing beyond H_train forces the model to operate on conditional distributions absent from training, and it is unclear whether finite-prompt RL learns the required operators.
  • RC circumvents this challenge by altering decoding so long-horizon reasoning avoids significant distribution shift.
  • Increasing H_train to match H_test is costly and may fail on harder future test distributions that require still larger budgets.
  • RC output alternates between reasoning and summarization, with later reasoning conditioned on the generated summary.

5. Enabling Extrapolation with Reasoning Cache

Reasoning Cache (RC) replaces autoregressive decoding with alternating bounded-length reasoning and summarization turns, enabling longer effective test-time reasoning while preserving tractable rollout lengths. Its benefits depend on informative summaries and suitable instruction-following models, and performance extrapolates beyond the training horizon.

  • RC decoding: RC alternates reasoning generation with summarization, discarding the original trace and conditioning subsequent reasoning on the updated summary.The process uses separate reasoning and summarization instructions, with summaries retaining high-level strategies and conclusions.
  • RC decoding: Increasing the number of bounded-length turns increases test-time compute while keeping each autoregressive generation within the training horizon.RC defines the effective budget through repeated turns rather than a single response longer than H_train.
  • Extrapolation results: 17%: 4B model accuracy increases when the test token budget scales from 16k to 192k, while cumulative token usage grows linearly with the provided budget.The corresponding 30B model accuracy increase is 12%.
  • Extrapolation results: RC consistently outperforms self-verification and self-refinement across test-time budgets.Summary conditioning keeps contexts bounded and removes redundant distractor tokens that can obscure useful information for later reasoning.
  • Design factors: Summary detail has an optimum: very short summaries hurt performance, additional detail improves it, and performance peaks with summaries of at least two paragraphs.Omitting summarization entirely also degrades performance at H_test = 192k.
  • Design factors: RC gains are smaller with Qwen3-4B-Thinking-2507 than with the instruction-following base model, while reducing H_R from 16k to 8k has minimal impact.Reducing H_R further to 4k substantially degrades performance because nearly half of traces terminate early.
  • Summary-conditioned reasoning: Verification is the most common strategy in RC traces, followed by exploration and refinement.Very few samples restart from scratch without using the summary.

6. Training Models to Extrapolate with RC Decoding

The RC training method strengthens summary-conditioned generation so each iteration can produce reasoning that improves on accumulated information. It combines iterative on-policy rollouts with replay-buffer summaries to train later reasoning turns without requiring equally long on-policy trajectories.

  • Training objective: RC training optimizes summary-conditioned generation so the model produces improved reasoning given a problem and a prior summary.The objective targets reasoning likely to yield a correct answer while exploiting information accumulated across iterations.
  • Training objective: Training runs RC for T_train turns, samples unique generated summaries, and generates multiple reasoning traces conditioned on each summary for reward-based optimization.Outcome rewards and advantages are computed over the conditioned reasoning samples.
  • Training objective: Local optimization of each iteration is aligned with the full trajectory when later steps can use intermediate results and partial analyses preserved in summaries.Conditioning rollouts on summaries explicitly trains the model to exploit accumulated information in subsequent responses.
  • Replay-buffer training: A summary replay buffer enables training on later reasoning turns without generating long on-policy decoded trajectories.The buffer stores problem-summary pairs and extends the maximum effective training horizon by T_train steps per epoch.
  • Replay-buffer training: From the second epoch onward, RC rollouts are initialized from sampled replay-buffer summaries rather than freshly generated summaries.New summaries replace older summaries for the same input problem, and newly generated summaries seed optimized training rollouts.

7. Experimental Evaluation: Training with RC

The experiments evaluate RC training across mathematical and scientific benchmarks, comparisons with autoregressive and iterative baselines, scaffold use, and ablations of training design choices. RCT-4B shows stronger long-horizon extrapolation than the base model and standard RL, with gains extending to an unseen scientific domain.

  • Benchmark Results: RCT-4B improves over the base model across all four reasoning benchmarks as the test-time token budget exceeds the training budget.The performance gap widens with increasing token budget, indicating improved extrapolation rather than only short-horizon gains.
  • Benchmark Results: RCT-4B outperforms autoregressive approaches on HMMT 2025, IMO-AnswerBench, and FrontierScience, and achieves competitive results against much larger reasoning models.The comparison uses benchmarks selected for low contamination risk.
  • Benchmark Results: RC training improves FrontierScience performance despite training exclusively on mathematics, whereas standard RL achieves no gains on that out-of-domain benchmark.The standard RL model also remains substantially weaker than specialized reasoning models.
  • Benchmark Results: RCT-4B outperforms iterative decoding baselines, while nearly all evaluated iterative methods outperform autoregressive decoding on FrontierScience.The authors attribute RCT-4B’s advantage to exploiting the summarization–generation asymmetry during training.
  • Benchmark Results: At 256k tokens, RCT-4B reaches nearly 35% pass@16 versus 20% for the base model.The trained model’s advantage increases as the reasoning token budget grows.
  • Ablation Studies: At 192k tokens, Stage II with the summary replay buffer improves accuracy by 9.4% over Stage I, compared with approximately 4% without the buffer.At 16k tokens, the corresponding improvement with replay is 2.7%.
  • Ablation Studies: Training with three turns performs best among Ttrain ∈ {2, 3, 4}, consistent with balancing gradient signal across early turns.The authors associate this trade-off with allocating insufficient signal to early turns when the training horizon is too long or too short.
  • Ablation Studies: Standard outcome-reward RL yields modest base-model improvements but falls far short of RC training, showing that effective RC extrapolation requires its structured multi-turn objective.The RL comparison uses the same number of training steps and a two-stage curriculum, with the RL training budget increased from 16k to 32k tokens.

8. Conclusion and Perspectives on Future Work

The paper concludes that RC trains models to improve reasoning across horizons much longer than their training rollouts by alternating summarization with summary-conditioned reasoning. It also identifies non-myopic rewards and explicit summary-generation training as important directions for harder problems.

  • Conclusion: RC enables continual reasoning improvement across long horizons by replacing autoregressive decoding with alternating summarization and summary-conditioned reasoning.Outcome-reward RL trains models to use this iterative decoding process more effectively.
  • Conclusion: RC-trained models gain performance on challenging mathematical and scientific benchmarks by extrapolating to reasoning horizons much longer than those used in training.The trained model can solve problems it cannot solve within its training budget and leverage existing test-time scaffolds.
  • Future Work: The current objective assigns rewards to individual traces, which discourages exploratory early turns whose value is realized only in later turns.The authors propose non-myopic extensions of RC for potentially improving performance on harder problems.
  • Future Work: The current approach trains summary-conditioned generation but does not explicitly optimize summary generation, although changing summaries can significantly affect RC performance.The authors identify direct summary-generation optimization as a potential improvement.

Appendices

Figure 11 compares RCT-4B with selected reasoning models on IMO-AnswerBench and reports that combining RC training with RC decoding outcompetes many larger and newer models.

  • Figure 11: RCT-4B outcompetes many larger and newer reasoning models on IMO-AnswerBench when RC training and decoding are combined.Inference hyperparameters are set using the recommended values from each model’s Hugging Face page.

B. Comparison with Majority Voting

RC decoding uses additional sequential reasoning depth more effectively than majority voting over parallel samples for the tested settings. This comparison is reported for both the base Qwen3 model and RCT-4B.

  • RC decoding is evaluated on IMO-AnswerBench using Qwen3-4B-Instruct-2507 and RCT-4B.The figure caption specifies accuracy and Maj@16 as the plotted measures.
  • RC increases reasoning depth more effectively than majority voting increases breadth for the tested value of k.The comparison evaluates accuracy and Maj@16 against sequential reasoning token budgets.
  • The supplied passages also identify RC evaluations on gpt-oss models and report that both models benefit from extrapolation through RC.These evaluations use H_R = 64k and different reasoning efforts for generation and summarization.

E. Limitations of RC and Directions for Future Work

RC has limitations in reward design, model applicability, and problem structure. Its computational advantages rely on settings where summaries preserve useful reasoning information and iterative training remains effective.

  • Training limitations: RC training does not optimize summary generation because summary-conditioned reasoning is treated as the primary bottleneck.Optimizing summary generation alone hurts efficacy, while jointly optimizing both objectives performs worse than reasoning-only training.
  • Training limitations: Summary-generation rewards suffer from credit-assignment problems when later reasoning fails for reasons unrelated to summary quality.The paper leaves alternative reward-assignment schemes for future work.
  • Training limitations: RC training uses myopic rewards assigned independently at each iteration, without explicit credit assignment to future turns.The authors argue these rewards can align with trajectory success when each step makes monotonic progress.
  • Model scope: RC is limited to models with strong summarization-generation asymmetry, which instruction-following models generally possess but highly specialized reasoning models do not.Using a separate summarization model would require maintaining two models and may create practical challenges.
  • Problem scope: RC may underperform on search-heavy problems because summaries can discard individual states needed to track the search process.The method is better suited to solution graphs with compactly summarizable clusters and sparse connections between them.
  • Efficiency: RC can be approximately T times cheaper than autoregressive decoding for a fixed effective reasoning budget in attention-dominated regimes.The comparison assumes T H_R >> C and H_R >> H_S.

F.2. Training

RC training separates long-horizon trajectory construction from bounded policy optimization, avoiding the quadratic rollout-cost growth of standard long-context RL. It keeps optimized rollouts and within-turn memory bounded while supporting longer effective horizons at inference.

  • Standard RL: Standard long-context RL generates B·K·N tokens with attention cost that grows with the sequence context.The resulting training cost scales quadratically with the rollout horizon when N is large.
  • RC training: RC training first generates summary trajectories and then optimizes bounded reasoning rollouts conditioned on sampled summaries.The policy-optimization stage uses N_summ summaries and K_reason rollouts of length at most H_R.
  • RC training: All RC optimized rollouts remain bounded by H_R regardless of the total effective reasoning horizon supported at inference time.This decouples optimized rollout length from the effective reasoning horizon.
  • Memory scaling: RC requires approximately T times lower KV memory at the same effective reasoning horizon.Its within-turn memory is independent of the number of turns T.
  • Scaffolds: RC replaces initial solution generation and refinement steps in RSA with RC decoding.The reported experiment uses k = 2, M = 8, T_RSA = 10, and T = 8 RC steps.
  • Scaffolds: RC replaces initial candidate generation in DSM Agent to improve the quality of its candidate pool.The experiment uses n_g = 8, n_v = 4, T_DSM = 6, and T = 8 RC steps.
  • Baselines: Budget forcing yields modest mathematical-reasoning gains but degrades FrontierScience performance, whereas RC-based methods substantially outperform it across benchmarks.The authors associate budget forcing with unchanged answers, repetition, and unnatural out-of-distribution prefixes.

K.2. Delethink

Delethink extends reasoning by chunking continuation from the previous trace, whereas RC summarizes prior reasoning and conditions subsequent iterations on those summaries. The paper reports that RC outperforms Delethink across tested token budgets.

  • Delethink: Delethink generates H_R reasoning tokens conditioned on the final H_chunk tokens from the previous iteration.It continues until EOS or the maximum iteration count and supports long-horizon RL through chunking.
  • Results: Delethink generally improves as token budget increases but plateaus near the performance of the base model.The cited experiment evaluates H_R values of 4k and 8k and excludes 16k because the model usually terminates there.
  • Results: RC outperforms Delethink across all tested token budgets using Qwen3-4B-Instruct-2507 without additional training.The comparison uses the same base model for both methods.
  • Mechanism: RC explicitly drives verification and refinement through summaries, whereas Delethink can terminate after generating a final answer.The paper attributes significant performance gains to this continued summary-conditioned process.
  • Mechanism: RC uses structured summaries that are more in-distribution for suitable base models than Delethink’s prompt-plus-last-chunk prefixes.The paper hypothesizes that summary conditioning better supports difficult problems after chain-of-thought progress stalls.

L. Hardware, Hyperparameters, and Implementation Details

The paper reports its training and inference hardware, software stack, and a naive sequential implementation of RC decoding. The supplied passages also include a coordinate-based geometry example whose construction considers both square orientations and identifies a non-degenerate solution.

  • Hardware: Training uses a single node of 8×H100 GPUs, while inference uses either 8×H100 or 4×GH200 GPUs.
  • Software and implementation: The implementation uses verl with FSDP for training and vLLM for inference, with a naive RC decoding implementation.
  • Software and implementation: Each RC turn generates reasoning traces and summaries sequentially in batches, leaving faster sequences idle while waiting for slower sequences.The paper identifies asynchronous decoding across turns as a more efficient alternative.
  • Example geometry: The geometry example parameterizes E as (1,e), considers clockwise and counterclockwise constructions for square CEFG, and tests the segment constraint C ∈ DG.The supplied solution reports the non-degenerate value e = 2 - √3 and therefore FG = √3 - 1.
Loading 2602.03773v2…