PaperlayerView exact revision on arXiv

Source-linked AI summary

InftyThink+: Effective and Efficient Infinite-Horizon Reasoning via Reinforcement Learning

Yuchen Yan, Liang Jiang, Jin Jiang, Shuaicheng Li, Zujie Wen, Zhiqiang Zhang, Jun Zhou, Jian Shao, Yueting Zhuang, Yongliang Shen

arXiv:2602.06960v3cs.CLcs.AI

TL;DR

Long chain-of-thought reasoning faces cost, context, and information-access limits, while existing iterative methods do not learn key summarization and continuation decisions. InftyThink+ combines supervised cold-start with trajectory-level reinforcement learning and reports stronger accuracy, efficiency, and generalization.

  • Problem

    Long reasoning traces face quadratic cost, context-window limits, and lost-in-the-middle degradation, while existing iterative methods do not optimize when to summarize, what to preserve, or how to continue.

  • Method

    InftyThink+ uses supervised cold-start followed by end-to-end reinforcement learning over complete iterative reasoning trajectories.

  • Results

    InftyThink+ consistently outperforms SFT-based iterative reasoning and standard long-context RL, including a 21% AIME24 accuracy improvement and reduced inference latency.

  • Takeaways & Limitations

    Trajectory-level optimization enables learned adaptive reasoning behaviors while improving reasoning performance and efficiency.

  • Takeaways & Limitations

    Evaluation focuses on mathematical and scientific reasoning benchmarks, so generalization to other domains requires further investigation.

Abstract

from arXiv · show

Large reasoning models achieve strong performance by scaling inference-time chain-of-thought, but this paradigm suffers from quadratic cost, context length limits, and degraded reasoning due to lost-in-the-middle effects. Iterative reasoning mitigates these issues by periodically summarizing intermediate thoughts, yet existing methods rely on supervised learning or fixed heuristics and fail to optimize when to summarize, what to preserve, and how to resume reasoning. We propose InftyThink+, an end-to-end reinforcement learning framework that optimizes the entire iterative reasoning trajectory, building on model-controlled iteration boundaries and explicit summarization. InftyThink+ adopts a two-stage training scheme with supervised cold-start followed by trajectory-level reinforcement learning, enabling the model to learn strategic summarization and continuation decisions. Experiments on DeepSeek-R1-Distill-Qwen-1.5B show that InftyThink+ improves accuracy by 21% on AIME24 and outperforms conventional long chain-of-thought reinforcement learning by a clear margin, while also generalizing better to out-of-distribution benchmarks. Moreover, InftyThink+ significantly reduces inference latency and accelerates reinforcement learning training, demonstrating improved reasoning efficiency alongside stronger performance.

1. Introduction

Long chain-of-thought reasoning is limited by computational cost, context windows, and lost-in-the-middle degradation. InftyThink+ applies trajectory-level reinforcement learning to iterative reasoning, improving accuracy and efficiency.

  • Motivation: Quadratic self-attention cost, bounded context windows, and lost-in-the-middle effects constrain very long reasoning traces.These barriers can make long traces expensive, prematurely terminated, or less effective.
  • Motivation: Iterative reasoning periodically compresses or summarizes accumulated context, decoupling reasoning depth from context length while bounding per-step computation.Consecutive reasoning rounds use refreshed summaries rather than the entire history.
  • Research gap: Existing methods leave unresolved when to compress, what to preserve, and how to resume, while fixed chunks or supervised imitation impose strategic limitations.The paper frames these as sequential decisions with long-horizon consequences.
  • Approach: InftyThink+ combines supervised cold-start with trajectory-level reinforcement learning to optimize the complete iterative reasoning trajectory.The RL stage learns summarization timing, information preservation, and continuation across iterations.
  • Results: InftyThink+ consistently outperforms SFT-based iterative reasoning and standard long-context RL while generalizing to larger models and out-of-distribution tasks.The paper attributes the improvements to learned adaptive iteration behaviors rather than fixed heuristics.

2. Related Work

Prior work manages long reasoning through context compression or segmentation, but InftyThink+ adds end-to-end reinforcement learning for summarization and continuation strategy.

  • Reasoning models: Reasoning models improve performance through long processes of decomposition, planning, inference, and reflection, but limited context windows constrain this capability.The related work frames context limitation as a central challenge for current reasoning models.
  • Context management: Input-side methods compress or discard earlier context, while output-side methods segment long trajectories into shorter reasoning segments.Both directions aim to reserve context for subsequent reasoning or expand the usable context horizon.
  • Iterative reasoning: InftyThink propagates information across iterations with explicit textual summaries, whereas prior training relies on supervised learning and heuristic data construction.InftyThink+ replaces this training approach with end-to-end trajectory-level RL optimization.
  • Iterative reasoning: InftyThink+ learns effective summarization and continuation strategies through trajectory-level feedback.This distinguishes it from methods that use fixed heuristics or supervised imitation alone.

3. Methods

InftyThink+ first teaches an iterative reasoning format, then optimizes complete multi-round trajectories with task and efficiency rewards shared across iterations.

  • InftyThink paradigm: InftyThink distributes reasoning across iterations connected by explicit summaries and terminates when the model generates a conclusion instead of another summary.Each round conditions on the query and the previous iteration’s summary.
  • InftyThink paradigm: Bounded per-iteration context decouples total reasoning depth from context length and requires summaries to preserve information needed later.The model sees the query and most recent summary rather than the full reasoning history.
  • Cold start: Cold-start supervised fine-tuning converts existing reasoning data into multi-iteration outputs with explicit summaries and teaches syntactically valid InftyThink formatting.The transformed data partitions reasoning traces and adds generated summaries before supervised initialization.
  • Trajectory-level RL: Reinforcement learning then optimizes the complete multi-generation trajectory rather than only learning the output format.The method separates format acquisition from strategy optimization.
  • Trajectory-level RL: Trajectory rollouts repeatedly generate outputs from the query and previous summary until the model concludes, fails to produce a usable summary, or reaches the iteration bound.The maximum number of iterations is controlled by φ to maintain training efficiency.
  • Reward design: Task rewards evaluate final correctness, while efficiency rewards penalize additional iterations with a quadratic decay.The efficiency reward is maximal at one iteration and decreases as iteration count increases.
  • Reward design: Efficiency rewards affect only correct trajectories, preventing premature termination from being rewarded at the expense of accuracy.Task and efficiency rewards are combined multiplicatively.
  • Policy optimization: GRPO shares trajectory-level advantages across all iterations, allowing early summaries that support later success to receive learning signal.This treats all outputs in a trajectory as jointly optimized under the trajectory outcome.

4. Experiments

Experiments evaluate InftyThink+ across models and benchmarks, finding stronger accuracy, lower inference latency, and improved effectiveness–efficiency trade-offs than conventional settings.

  • Generalization: InftyThink+ extends its gains across larger-scale models and out-of-distribution code and scientific reasoning tasks.The experiments include DeepSeek-R1-Distill-Qwen-1.5B and Qwen3-4B-Base, with evaluations including MATH500, AIME24, AIME25, and GPQA Diamond.
  • Experimental setup: The evaluation reports accuracy, generated-token counts, and inference latency across multiple benchmarks, using 32 samples at temperature 0.7.Table 1 distinguishes cold start, task-only RL, and task-plus-efficiency RL settings.
  • Effectiveness: InftyThink+ achieves larger reinforcement-learning accuracy gains than Vanilla, averaging +9.89 ACC versus +5.62 under task-only RL.On AIME24, gains are +21.46 points for InftyThink+ versus +12.08 for Vanilla.
  • Effectiveness–efficiency trade-off: The efficiency-reward configuration improves average accuracy by +6.51 points while reducing latency from 77.57s to 48.37s.Compared with task-only RL, it reduces latency from 100.21s to 48.37s and tokens from 20.02K to 10.66K, with average accuracy changing from 53.96 to 50.58.

5. Analyses

Analyses show that InftyThink+ learns adaptive timing, informative summaries, and effective continuation policies, while also accelerating inference and RL training.

  • Learning when to compress: Adaptive interruption timing outperforms Random and Fixed strategies, with incorrect timing causing larger accuracy degradations after RL.With RL, Random and Fixed timing degrade AIME24 by -2.50 to -3.02 and AIME25 by -2.00 to -2.83.
  • Learning how to compress: Replacing internal summaries with external summaries raises SFT-only AIME24 accuracy from 29.48% to 32.40% but lowers RL-trained accuracy from 50.94% to 48.42%.The reversal indicates that RL couples summary generation with downstream reasoning.
  • Learning how to continue: Vanilla continuation gains nearly saturate after the second iteration, whereas InftyThink+ converts later summaries into monotonic performance improvements.This comparison tests continuation from summaries produced at the first through fourth iterations.
  • Inference efficiency: InftyThink+ reduces inference latency by 30%–40%, and efficiency reward increases the reduction to 60%–70%.The analysis attributes these gains to O(n · ℓ2) iterative complexity versus O(L2) for Vanilla.
  • RL training efficiency: Vanilla long-context RL averages 300 seconds per step, versus 225 seconds for InftyThink+ and 175 seconds with efficiency reward.These correspond to approximately 25% and 40% speedups, respectively.

6. Conclusion

InftyThink+ uses trajectory-level reinforcement learning to learn effective and efficient iterative reasoning, improving accuracy while reducing inference latency.

  • Contribution: InftyThink+ separates format learning from strategy optimization so models learn when to compress, how to compress, and how to continue.The framework combines supervised initialization with end-to-end trajectory-level reinforcement learning.
  • Findings: Experiments show consistent accuracy gains over SFT-based iterative reasoning and standard long-context RL alongside substantially lower inference latency.The reported improvements arise from learned adaptive behaviors rather than heuristics.

Impact Statement

The paper identifies computational and energy benefits from more efficient reasoning, while noting misuse risks, possible usage rebound, and limited domain coverage.

  • Potential benefits: InftyThink+ may reduce computational costs and energy consumption for reasoning tasks.These are presented as primary societal benefits of more efficient reasoning systems.
  • Risks: Improved reasoning capabilities could be misused to generate more convincing misinformation or enable harmful applications.The paper states that these risks also apply broadly to advances in language-model reasoning.
  • Environmental considerations: Lower per-query costs may increase overall usage, partially offsetting environmental benefits.This is identified as a broader-impact consideration rather than a measured result.
  • Scope limitation: The evaluation focuses on mathematical and scientific reasoning benchmarks, so generalization to other domains requires further investigation.The paper frames this as a limitation of the current evaluation scope.

A.2. Limitations

InftyThink+ has limitations tied to task structure, natural-language summaries, and cold-start data requirements. These constraints bound its benefits, controllability, and ease of adaptation.

  • Task-structure assumptions: InftyThink+ assumes reasoning can be decomposed into relatively independent stages whose essential information can be summarized for later reasoning.This assumption fits mathematical reasoning and multi-constraint planning better than highly entangled tasks.
  • Limitations of natural language summaries: Natural-language summaries encode importance, logical status, and priority implicitly, limiting fine-grained control over compression granularity and information fidelity.The model must reinterpret and rebalance these factors during continuation.
  • Dependence on cold-start training: The cold-start stage provides structural scaffolding rather than directly optimizing reasoning strategies, creating dependence on task-specific cold-start data design.Adapting the framework to new domains or task distributions therefore introduces additional engineering complexity.
  • Future directions: Extending InftyThink+ to long-horizon agentic tasks is identified as a promising direction involving longer time scales, interaction loops, tool use, and evolving contexts.
  • Future directions: Future work may explore latent tokens, learned memory slots, or hybrid symbolic–continuous summaries to represent reasoning states more compactly and faithfully.

B. Context Hit Analysis

Context-hit analysis shows that longer vanilla reasoning budgets do not reliably produce completed solutions and can reduce efficiency. InftyThink instead bounds per-iteration context through summaries while continuing reasoning across iterations.

  • Context Hit Analysis: 8k, 16k, 32k, 48k, and 64k token budgets are evaluated using completion rate and accuracy across MATH500, AIME24, AIME25, and AMC23.Completion rate measures the fraction of instances that generate an eos token within the token budget.
  • Context Hit Analysis: Even at 32k–64k tokens, vanilla reasoning fails to complete some challenging AIME24 and AIME25 tasks.
  • Context Hit Analysis: At 48k and 64k tokens, completion rate remains nearly unchanged, consistent with lost-in-the-middle effects and repetitive or unproductive deliberation.
  • Context Hit Analysis: Increasing generation length substantially decreases tokens generated per second, motivating extended reasoning that preserves inference efficiency.
  • InftyThink Reasoning Paradigm: InftyThink decomposes long reasoning into bounded segments and uses summaries to preserve global coherence across iterations.The first segment produces a compact summary, subsequent segments condition on the previous summary, and the final iteration produces a conclusion.
  • Vanilla Reasoning Paradigm: Vanilla long-context reasoning records explicit thinking and a final conclusion in a single-round generation process.
  • InftyThink Reasoning Paradigm: Periodic summarization bounds context usage and memory footprint while allowing reasoning to continue beyond fixed context-length limits.The process repeatedly generates segments and summaries until a conclusion is produced or the maximum iteration count φ is reached.
  • Summary conditioning: InftyThink+ conditions each intermediate summary only on the current reasoning segment and previous summary, matching inference-time information constraints.This differs from generating summaries using the full preceding reasoning trajectory.

H.1. Training-time Metrics

Training-time analyses show that InftyThink+ reduces per-step RL training time and improves reward dynamics. Efficiency-aware rewards further reduce training time and iteration counts while moderating generation length.

  • Training efficiency: 225 seconds versus around 325 seconds: InftyThink+ has lower average per-step training time than the vanilla method.The reduction spans rollout, log-probability computation, and actor updates.
  • Training efficiency: 225 seconds to 175 seconds: enabling the efficiency reward produces a downward trend in InftyThink+ per-step training time during learning.
  • InftyThink metrics: InftyThink+ consistently achieves higher task rewards than vanilla RL, while efficiency rewards substantially reduce the number of iteration rounds.

I.1. Evaluation Across More Domains

Across mathematical, scientific, and code reasoning, InftyThink+ improves the accuracy–efficiency trade-off relative to vanilla approaches, with benefits extending to out-of-distribution settings. Training analyses further show stronger policy optimization and efficiency-sensitive generation dynamics.

  • Mathematical Reasoning: 50.94% on AIME24: InftyThink+ raises accuracy beyond vanilla RL’s 38.75% while adding almost no inference time.Vanilla RL improves from 26.67% to 38.75%, whereas InftyThink+ reaches 50.94%.
  • Scientific Reasoning: +6.56 average accuracy gain: InftyThink+ with task reward outperforms vanilla RL on out-of-distribution scientific reasoning benchmarks.Vanilla RL yields only a marginal average gain of +1.67 while substantially increasing inference cost.
  • Code Reasoning: 26.93 seconds versus 64.17 seconds: cold-start InftyThink+ reduces average code-reasoning latency while improving accuracy relative to vanilla.Adding efficiency reward preserves most accuracy gains while slightly reducing generated tokens and latency relative to cold start.
  • Code Reasoning: Average ΔACC +6.48: task-plus-efficiency reward preserves most code-reasoning accuracy gains while improving the accuracy–efficiency Pareto frontier.
  • Training progression: Across three benchmarks, actor performance improves throughout RL training, with task-reward RL producing rapid and stable accuracy gains.Efficiency reward slightly slows early accuracy growth but yields smoother dynamics and competitive late-stage performance.
  • Training progression: Efficiency reward suppresses token growth and reduces latency dynamics during training, whereas vanilla latency steadily increases across benchmarks.

J.1. Training Stability

Across repeated runs, InftyThink+ shows consistent training dynamics and evaluation trends despite short-term fluctuations. Accuracy and latency measurements remain reproducible, with larger accuracy variability on the smaller AIME benchmarks.

  • Training dynamics: Three independent runs show closely aligned global training trends despite short-term computational-nondeterminism fluctuations.Training reward rises steadily, while policy-gradient loss and entropy follow consistent trajectories.
  • Accuracy stability: Evaluation accuracy trends are largely consistent across MATH500, AIME24, and AIME25, with aligned upward trajectories and comparable final performance.MATH500 is smoother because it contains 500 problems, whereas AIME24 and AIME25 contain 30 each.
  • Accuracy stability: Repeated checkpoint evaluations closely overlap across benchmarks, supporting that observed accuracy gains reflect model improvements rather than evaluation noise.AIME24 and AIME25 show somewhat larger pointwise fluctuations because their smaller test sets make accuracy estimates more sensitive to nondeterminism.
  • Latency stability: Latency curves remain highly consistent across repeated evaluations, with stable relative ordering and temporal evolution throughout training.Only minor fluctuations appear at individual training steps when identical checkpoints and evaluation settings are used.

K.1.3. LIMITATION OF SUPERVISED LEARNING

The paper argues that supervised learning cannot optimize summaries for answer informativeness or align them with the policy’s own continuation capabilities. It motivates trajectory-level reinforcement learning and reports iterative reasoning’s stability, accuracy-scaling, and latency benefits.

  • Limitation of Supervised Learning: Supervised fine-tuning does not optimize the Information Bottleneck objective for iterative summaries.Its objective is independent of answer correctness and therefore does not optimize the informativeness term.
  • Limitation of Supervised Learning: SFT-generated summaries can mismatch the continuation capabilities of the policy being trained.Summaries generated by an external model may preserve information needed by that model but not by the current policy.
  • Limitation of Supervised Learning: Reinforcement learning addresses both limitations by rewarding answer correctness and training the policy on summaries it generates itself.Outcome-based rewards encourage answer-relevant summaries, while closed-loop optimization aligns summaries with the policy’s continuation behavior.
  • Computational Complexity Analysis: When m ≪ ℓ and n > 1, iterative reasoning has strictly lower computational cost than vanilla long-context reasoning.The cost ratio is approximately 1/n under the stated assumptions.
  • Reinforcement Learning without Cold Start: Cold start stabilizes reinforcement learning, whereas removing it causes reward collapse, policy-gradient spikes, and entropy explosion.Repeated no-cold-start runs reproduce the same failure pattern, indicating systematic instability.
  • Iterative Reasoning Performance: Increasing reasoning rounds improves accuracy across three benchmarks, with the largest gains in early rounds and later saturation.InftyThink+ trained with task-reward RL maintains higher accuracy and stronger scaling across additional rounds than the pre-RL model.
  • Inference Efficiency: InftyThink+ shifts latency distributions left, reduces heavy right tails, and achieves its largest latency reduction with efficiency-aware reward shaping.RL(T+E) reduces mean latency to 17.71s on MATH500, 57.50s on AIME24, and 68.39s on AIME25.

O.1. Ablation of Iteration Cap Parameter φ.

The iteration cap φ controls InftyThink+’s reasoning horizon, creating a trade-off between optimization efficiency and downstream task performance. Larger caps support deeper refinement and generally stronger accuracy, while smaller caps produce cheaper, higher-reward rollouts.

  • φ bounds the maximum number of InftyThink-style reasoning rounds, controlling the effective reasoning horizon and computation budget.
  • RL optimization dynamics: Smaller caps optimize faster and achieve higher aggregate training rewards, while φ=10 learns more slowly because longer rollouts increase variance and complicate credit assignment.
  • InftyThink-specific behavior: Larger caps enable more realized turns and iterative refinement: task-only training reaches approximately 2 turns for φ=3, approximately 3 for φ=5, and approximately 5–6 for φ=10.
  • Accuracy–efficiency trade-off: Efficiency rewards favor smaller caps, while φ=10 often delivers stronger task performance; φ therefore defines an accuracy–efficiency frontier, with φ=5 often balancing both objectives.
  • Downstream reasoning performance: Larger iteration budgets generally improve downstream accuracy, especially on AIME24/AIME25, whereas φ=3 produces slower and lower improvement curves.
  • Context-window interaction: Larger context windows generally improve accuracy but increase latency, while η=6k can offer comparable accuracy to η=8k at lower latency.

P. Ablation on Efficiency Reward Decay

InftyThink+ evaluates how efficiency-reward decay shapes training and reasoning length. Quadratic decay provides the most stable performance because it encourages efficiency without excessively suppressing additional reasoning.

  • The efficiency reward depends on the normalized number of reasoning iterations before termination, with the trajectory’s turn count K(τ) compared against the maximum budget ϕ.
  • All decay variants reward one-turn trajectories most and reduce efficiency reward as additional reasoning iterations are used.
  • Quadratic decay is the most conservative early and mid-trajectory penalty, encouraging efficiency without aggressively suppressing further reasoning iterations.
  • Benchmark comparison: Quadratic decay produces the most stable training curves and strongest final performance across MATH500, AIME24, and AIME25.
  • Training dynamics: Linear and logarithmic decay reduce reasoning turns faster, increasing efficiency rewards but causing later declines in task reward and benchmark accuracy.
  • Design choice: The study adopts quadratic decay because steeper efficiency pressure can bias the policy toward premature termination and harm reasoning performance.

Q.2. Experimental Comparison

The experiments compare InftyThink+ with DeleThink under matched model, training, and dataset conditions. InftyThink+ consistently achieves higher AIME24 and AIME25 accuracy during training, while its format is implicitly enforced by task reward.

  • The DeleThink comparison uses the same base model, training configuration, and dataset, with checkpoint-level accuracy evaluated on AIME24 and AIME25.
  • Performance comparison: InftyThink+ consistently outperforms DeleThink throughout training on both AIME24 and AIME25.
  • Performance comparison: The task-reward-only variant achieves the strongest overall accuracy, while the task-plus-efficiency variant remains above DeleThink despite optimizing an additional efficiency objective.
  • Evaluation scope: DeleThink’s AIME24 curve ends at 700 training steps because its source paper reports checkpoint evaluations only through that point.
  • Format reward: Malformed output prevents the iterative loop from reaching a valid terminal answer, so task reward implicitly enforces format compliance and makes a separate format reward largely redundant.
Loading 2602.06960v3…