PaperlayerView exact revision on arXiv

Source-linked AI summary

Rethinking Thinking Tokens: LLMs as Improvement Operators

Lovish Madaan, Aniket Didolkar, Suchin Gururangan, John Quan, Ruan Silva, Ruslan Salakhutdinov, Manzil Zaheer, Sanjeev Arora, Anirudh Goyal

arXiv:2510.01123v1cs.LGcs.AI

TL;DR

Long reasoning traces can improve accuracy but entangle reasoning depth with sequence length and inherit long-context failure modes, motivating inference strategies that meet accuracy goals under explicit budgets. The paper studies bounded-summary iteration and trains a matching improvement policy; iterative pipelines outperform single-pass baselines at matched sequential budgets, with PDR reporting the largest gains.

  • Problem

    The paper asks what task accuracy is achievable under constraints on generated tokens, latency, context length, and compute, given that long traces entangle reasoning depth with sequence length and inherit long-context failure modes.

    Source: §1 · pp. 1–2

  • Method

    The paper studies phased generation with bounded round-wise summaries that condition fresh generations, and trains an operator-consistent policy using verifiable rewards.

    Source: §1 · p. 2

  • Results

    Iterative pipelines surpass single-pass baselines at matched sequential budgets, with shallow PDR delivering the largest gains, including +11% on AIME 2024 and +9% on AIME 2025.

    Source: §5 · p. 12

  • Takeaways & Limitations

    The findings suggest that iteration with short contexts and compact summaries can substitute for long traces while holding latency fixed.

    Source: §1 · p. 3

Abstract

from arXiv · show

Reasoning training incentivizes LLMs to produce long chains of thought (long CoT), which among other things, allows them to explore solution strategies with self-checking. This results in higher accuracy, but inflates context length, token/compute cost, and answer latency. We ask: Can current models leverage their metacognition to provide other combinations on this Pareto frontier, e.g., better accuracy with lower context length and/or latency? Abstractly, we view the model as an improvement operator on its own "thoughts" with a continuum of possible strategies. We identify an interesting inference family Parallel-Distill-Refine (PDR), which performs the following: (i) generate diverse drafts in parallel; (ii) distill them into a bounded, textual workspace; and (iii) refine conditioned on this workspace, producing an output that seeds the next round. Importantly, context length (hence compute cost) is controllable via degree of parallelism, and is no longer conflated with the total number of generated tokens. We report PDR instantiations of current models that give better accuracy than long CoT while incurring lower latency. Setting degree of parallelism to 1 yields an interesting subcase, Sequential Refinement (SR) (iteratively improve a single candidate answer) which provides performance superior to long CoT. Success of such model orchestrations raises the question whether further training could shift the Pareto frontier. To this end, we train an 8B thinking model with Reinforcement Learning (RL) to make it consistent with PDR as the inference method. On math tasks with verifiable answers, iterative pipelines surpass single-pass baselines at matched sequential budgets, with PDR delivering the largest gains (e.g., +11% on AIME 2024 and +9% on AIME 2025).

1 Introduction

The paper explores inference strategies that accumulate reasoning through bounded summaries rather than a single long trace, and asks how accuracy trades off against inference budgets. It studies sequential refinement and parallel-distill-refine, then trains an operator-consistent policy and reports gains over single-pass baselines.

1 Introduction

  • The paper asks what task accuracy is achievable under constraints such as total generated tokens, generation-chain depth, context length, and compute.Longer reasoning traces can correlate with accuracy but entangle reasoning depth with sequence length and inherit long-context failure modes.Source: §1 · pp. 1–2
  • A viable iterative scheme needs a bounded state that preserves salient facts and intermediate results, flags disagreements and open subgoals, and avoids replaying all prior attempts.Simply asking the model to try again risks forgetting useful partial results and repeating mistakes; appending attempts recreates long-context failures and scales cost with rounds.Source: §1 · p. 2
  • The proposed inference strategies generate solutions in phases, write bounded round-wise summaries, and use those summaries to condition fresh generations.Sequential refinement updates one artifact, while PDR samples parallel drafts and distills them into a bounded summary; the workspace is freshly synthesized each round.Source: §1 · pp. 2–3
  • Operator-consistent RL unrolls short drafts, distillation, and refinement during training with verifiable rewards, addressing the information-flow mismatch with PDR inference.The training objective conditions a refined attempt on the prompt and compact summary.Source: §1 · p. 3
  • +11% on AIME 2024 and +9% on AIME 2025 are reported for shallow PDR, which delivers the largest gains among iterative pipelines at matched sequential budgets.An operator-consistent RL objective yields further improvements, e.g., ∼5% on AIME 2024 and AIME 2025 when mixing standard and operator RL.Source: §1 · p. 3

2 Background & Related Work

The related-work discussion places the paper among approaches that scale reasoning through longer traces, self-improvement, aggregation, search, memory, and multi-turn training. Its distinction is a round-wise improvement operator that analyzes compute at matched sequential budgets and aligns training with inference.

2 Background & Related Work

  • The paper unifies existing primitives as round-wise operators, analyzes compute-normalized depth at matched Bseq while varying Btotal, and aligns training with deployment through operator-consistent RL.It explicitly notes that parallel sampling, sequential revision, structured search, and summarization are established primitives.Source: §2 · p. 3
  • Long-trace reasoning and self-consistency trade increased test-time tokens and latency for accuracy, whereas this approach accumulates evidence across rounds through bounded, re-synthesized summaries.Chain-of-thought improves multi-step reasoning (Wei et al., 2022), while self-consistency aggregates sampled traces (Wang et al., 2023).Source: §2 · p. 3
  • Self-Refine, Reflexion, CRITIC, and debate refine outputs through feedback, memory, tools, or argumentation; this operator instead constrains per-call context with a compact state.PDR also uses parallel inference-time exploration and distillation, analyzed under fixed Bseq while varying Btotal.Source: §2 · pp. 3–4
  • Aggregation methods filter or combine parallel traces, while Parallel-R1 uses parallel thinking as an exploration scaffold; this paper treats aggregation as a meta-cognitive ability within round-wise operators.Parallel-R1 reports gains over a sequential-thinking RL baseline on MATH, AMC23, and AIME.Source: §2 · p. 4
  • Tree of Thoughts, Graph of Thoughts, and Least-to-Most structure exploration, but typically grow tokens with breadth or depth or rely on long contexts; PDR distills rounds into bounded states.The cited methods are associated with Yao et al. (2023), Besta et al. (2024), and Zhou et al. (2022).Source: §2 · p. 4
  • Rather than persistent memory or cross-instance reasoning reuse, the paper uses non-persistent round-wise summaries and focuses on inference-time operators plus aligned RL.Related work also includes adaptive multi-turn RL and tree-search RL; the paper's training unrolls one iterative round with outcome supervision, without deep tree search.Source: §2 · p. 4

3 LLMs as Improvement Operators

The section frames an LLM as an improvement operator that refines artifacts through bounded workspaces under explicit token budgets. It instantiates sequential refinement and parallel-distill-refine, then describes training that aligns the model with the short-context inference interface.

3.1 Problem setting and notation

  • A compact workspace is a bounded summary of agreements, contradictions, intermediate results, and open subgoals; accuracy is evaluated against latency and compute proxies.The paper distinguishes tokens on the accepted path from tokens across all model calls, including discarded branches.Source: §3.1 · p. 5

3.2 Operator Instantiations

  • The section introduces two short-context iterative refinement pipelines: sequential refinement and Parallel-Distill-Refine.Sequential refinement iterates over one artifact, while PDR samples drafts in parallel and distills them into a workspace.Source: §3.2 · p. 5, §3.2.1 · p. 5, §3.2.2 · p. 5

3.2.1 Sequential refinement (SR; depth over a single candidate).

  • Sequential refinement iteratively improves a single artifact without an explicit workspace; an optional error-analysis step supplies transient notes between rounds.The error-analysis variant asks the model to identify and explain flaws before generating a revised solution.Source: §3.2.1 · p. 5

3.2.2 Parallel-Distill-Refine (PDR; round-wise workspace)

  • PDR samples parallel drafts, distills them into a bounded summary, and uses that workspace to guide subsequent refinement without replaying prior text.The workspace is re-synthesized each round, and the final round uses a single generation as the returned artifact.Source: §3.2.2 · pp. 5–6

3.3 Operator-Consistent Training

  • Operator-consistent training mixes standard long-trace RL with short-context generate-distill-refine rollouts to align training with inference.Training uses one PDR round for stability and cost control, while inference can run multiple rounds.Source: §3.3 · pp. 6–7

4 Experiments

The experiments compare short-context iterative operators with long chain-of-thought under matched sequential budgets, then test whether operator-consistent RL training improves performance. Iterative pipelines outperform single-pass baselines, with PDR delivering the largest gains and operator-consistent training adding further improvements.

4 Experiments

  • The evaluation measures mean@16 accuracy on AIME 2024 and AIME 2025, comparing SR and PDR with long-CoT baselines under matched sequential budgets.The experiments also report outcomes as functions of total token budget.Source: §4 · p. 7, §4.1 · p. 8

4.1 Experiments to understand SR and PDR

  • Adding error analysis before solution generation improves SR performance for o3-mini but not for gemini-2.5-flash.Other settings, including prompts and budgets, match standard SR.Source: §4.1 · p. 9, Table 1 · p. 10
  • PDR uses parallel drafts, distills selected candidates into a textual workspace, and refines solutions across rounds; SR iteratively refines a single candidate.The tested PDR settings vary rounds and parallel generations; the workspace carries selected candidates forward.Source: §4.1 · pp. 8, 10
  • At Bseq 49k, o3-mini accuracy rises from 76.9 (Long CoT) to 81.5 (SR) and 86.7 (PDR); at equal total tokens, PDR matches SR accuracy with a 2.57× smaller sequential budget.Figure 3 compares accuracy at matched sequential budgets; Figure 5 reports the token-matched comparison.Source: Figure 3 · p. 8, §4.1 · p. 10
  • Per-sample top-k and global-summary distillation generally outperform shared top-k and random-k, with wider margins at larger thinking budgets.For o3-mini on AIME 2025, global summary outperforms the alternatives.Source: §4.1 · p. 11, Table 2 · p. 11
  • Admitting only incorrect candidates into PDR summaries sharply reduces accuracy, especially for o3-mini, while correct-only candidates improve over Random-k.The same degradation pattern is reported across AIME 2024 and AIME 2025.Source: §4.1 · pp. 10–11, Figure 6 · p. 10

4.2 Operator-consistent RL Training

  • Operator-consistent RL uses PDR with four parallel generations, one round, and two candidates forwarded, mixing operator and standard RL batches.The training setup also increases prompt length to fit the compact workspace.Source: §4.2 · pp. 11–12
  • Continual PDR RL yields gains of +5.00 and +4.59 percentage points on AIME 2024 and AIME 2025, respectively.PDR RL first improves over baseline by +3.34 points on AIME 2024 and +1.67 points on AIME 2025; the reported larger gains are from continual updates starting at a baseline RL checkpoint.Source: §4.2 · p. 12, Table 3 · p. 12

5 Conclusion

Compact-memory iteration outperforms long-trace baselines at matched sequential budgets, with PDR delivering the largest reported gains. Operator-consistent RL and directly measured or trained meta-skills are highlighted as further routes to improvement.

5 Conclusion

  • PDR delivers the largest gains over single-pass baselines at matched sequential budgets: +11% on AIME 2024 and +9% on AIME 2025.The conclusion also reports that operator-consistent RL further improves performance by ∼5% on AIME 2024 and AIME 2025 when mixing standard and operator RL.Source: §5 · p. 12
  • Measuring and training diversity, verification, and refinement directly is proposed as a way to accelerate improvement-operator gains under fixed latency budgets.The conclusion suggests models can learn the meta-skills that make iteration effective.Source: §5 · p. 12
  • Future work includes trainable summaries, uncertainty-conditioned round and fan-out schedules, budget-aware compute allocation, verifier and tool integration, and broader scaling and domain evaluations.The proposed domains include reasoning, coding, and planning.Source: §5 · p. 12

A Extended Related Work

Related work is organized around budget-aware test-time reasoning and improvement procedures. The paper distinguishes its textual, round-wise operators from prior search and persistent-workspace approaches.

A Extended Related Work

  • The evaluation protocol reports sequential budget Bseq and total budget Btotal to compare single-pass, long-trace, sampling-heavy, and iterative pipelines.Bseq is a latency proxy along the accepted path; Btotal includes tokens from discarded branches.Source: A Extended Related Work · p. 16
  • Unlike search-based improvement loops, SR and PDR use textual round-wise operators to keep per-call Bseq small, with parallel drafts optionally increasing Btotal.Operator-consistent RL then amortizes the improvement procedure into model weights.Source: A Extended Related Work · p. 16
  • Unlike a persistent shared workspace, the paper’s textual summary is re-synthesized each round and used at inference time, avoiding long-context replay and architectural changes.The comparison draws on the coordination intuition of a small communication bottleneck.Source: A Extended Related Work · p. 16

B Prompts

The appendix provides prompts for sequential refinement and parallel-draft refinement, alongside training settings. It also illustrates a space-bounded machine with read-only input and bounded working memory.

B.1 Sequential Refinement

  • The sequential-refinement prompt treats the previous response as unverified and asks for a better answer without starting from scratch.It supplies the problem and requests a clearly presented final answer.Source: §B.1 · p. 16

B.2 Parallel-Distill-Refine

  • The non-summary refinement prompt asks the model to solve the problem from its prompt without supplying a prior-round summary.The appendix labels this prompt “Refinement Prompt (Non-summary).”Source: §B.2 · p. 17
  • A further refinement prompt presents multiple previous-round candidate responses as unverified and asks for a better answer without starting from scratch.The appendix shows candidate responses indexed from 1 to k.Source: §B.2 · p. 17
  • The summary-based prompt provides a summary of other solvers’ previous-round reasoning and asks the model to treat it as unverified context.The output is requested as a final answer to the problem.Source: §B.2 · p. 17

B.3 Training Setup

  • The SFT setup uses a pretrained 8B base model, 2M-token batches, sequence length 32768, and learning rate 2 × 10−5 for about four epochs and 8B tokens.The run uses AdamW on 32 H100 GPU nodes.Source: §B.3 · p. 17
  • RL training runs for 800 steps with learning rate 5 × 10−7; baseline RL uses 80 H200 GPUs, while PDR and continual PDR RL use 288.Evaluation uses temperature and top-p of 1.0; the GPU splits are 64/16 and 256/16 generators/trainers, respectively.Source: §B.3 · pp. 17–18
  • The space-bounded Turing machine reads an input of size N while using working memory S(N) with S(N) ≥ log N.The figure describes the input as read-only and accessible by moving a tape head over its bits.Source: Figure 7 · p. 18

C Complexity of Space-bounded computation

The appendix relates probabilistic language-model reasoning to randomized space-bounded computation. It uses graph connectivity and local graph access to illustrate reasoning with limited working memory.

C Complexity of Space-bounded computation

  • The paper’s fixed-context LLM is compared to a randomized space-bounded machine because LLM outputs are probabilistic unless temperature is zero.The discussion concerns reasoning traces longer than the context size.Source: C Complexity of Space-bounded… · p. 18
  • A randomized space-bounded machine can determine connectivity of an N-vertex graph using only O(log N) space.The graph may be too large to fit in working memory.Source: C Complexity of Space-bounded… · p. 18
  • When local edge queries are available, a machine can perform a random walk “in its mind” to determine connectivity without access to the full graph tape.The paper presents this as the closest setting to its own, with connectivity reasoning in less than O(N) space.Source: C Complexity of Space-bounded… · p. 18

D.1 Oracle

Incorrect solutions in the context workspace can substantially reduce performance, with a more pronounced effect for o3-mini.

D.1 Oracle

  • Incorrect solutions in the context workspace can heavily degrade performance, and the effect is more noticeable for o3-mini.Source: §D.1 · p. 18

D.2 SR and PDR operators

On AIME 2025, SR and PDR improve on Long CoT at matched sequential budgets, with PDR giving the strongest result. The section also examines selection-policy effects and an SR variant that improves results without increasing the sequential budget.

D.2 SR and PDR operators

  • At a 49k sequential token budget, o3-mini scores 73.5 with Long CoT, 77.1 with SR, and 82.9 with PDR on AIME 2025.Source: §D.2 · p. 18
  • On AIME 2025, admitting only incorrect candidates into PDR summaries reduces accuracy, while admitting only correct candidates improves over Random-k; the drop is larger for o3-mini than gemini-2.5-flash.Source: Figure 8 · p. 19
  • At comparable sequential budgets on AIME 2025, both SR and PDR outperform Long CoT, with PDR yielding the largest gains by using parallel compute without increasing per-call context.Bseq counts tokens along the accepted path, including the distilled summary, and excludes unused parallel proposals; it serves as a latency proxy.Source: Figure 9 · p. 19
  • An SR variant using error analysis followed by solution generation improves o3-mini results without affecting Bseq.Source: §D.2 · p. 19
  • Figure 10 reports that PDR forms the Pareto frontier and consistently gains over Long CoT and SR across Bseq and Btotal token budgets for o3-mini.Source: Figure 10 · p. 20

E Mechanics of Improvement operator: Source of accuracy gain

Hard-case analyses show that improvement depends on preserving useful draft evidence and successfully refining it, while added compute does not help uniformly. The analysis points to verification, refinement, and diversity as remaining skill gaps.

E Mechanics of Improvement operator: Source of accuracy gain

  • With more compute, gemini-2.5-flash accuracy changes from 0.4375 to 0.625 on Q1, 0.0625 to 0 on Q2, and stays at 0.1875 on Q3 and 0 on Q4.The comparison is between 4- and 5-round schedules; the higher-compute setting uses 32 first-round generations.Source: E Mechanics of Improvement operator:… · p. 20, Table 4 · p. 20
  • When a round-one draft is correct, improvement requires the summary to carry its evidence and the refine step to use it; otherwise later rounds cannot exploit that signal.The Q1 gain suggests both steps succeeded, while Q3's flat result despite 3/32 correct drafts indicates a verification or refinement gap.Source: E Mechanics of Improvement operator:… · pp. 20–21
  • For Q1 and Q3, only 3 of 32 first-round drafts are correct, leaving the summary to distinguish useful evidence from many incorrect distractors.Source: E Mechanics of Improvement operator:… · pp. 20–21
  • When round one has no correct draft, the summary must extract partial progress or eliminated avenues; on Q2, the added-compute regression is consistent with a summary reinforcing a wrong pattern.Q4's unchanged result may reflect a capability ceiling or a need for still more first-round generations.Source: E Mechanics of Improvement operator:… · pp. 20–21
  • The analysis identifies verification, refinement, and diversity as skill gaps relevant to PDR and to settings that combine these skills.Source: E Mechanics of Improvement operator:… · p. 21
Loading 2510.01123v1…