PaperlayerView exact revision on arXiv

Source-linked AI summary

SimpleTIR: End-to-End Reinforcement Learning for Multi-Turn Tool-Integrated Reasoning

Zhenghai Xue, Longtao Zheng, Qian Liu, Yingru Li, Xiaosen Zheng, Zejun Ma, Bo An

arXiv:2509.02479v2cs.LG

TL;DR

Multi-turn TIR reinforcement learning is destabilized when external tool feedback induces distributional drift and low-probability tokens, with effects compounding across turns. SimpleTIR filters trajectories containing void turns from policy updates, stabilizing training and improving mathematical reasoning. Starting from Qwen2.5-7B, it raises AIME24 from 22.1 to 50.5 and encourages diverse reasoning patterns without cold-start SFT.

  • Problem

    Multi-turn TIR reinforcement learning suffers from distributional drift caused by external tool feedback, compounding low-probability tokens, and training instability.

  • Method

    SimpleTIR filters trajectories containing void turns from policy-loss computation to block harmful high-magnitude gradients.

  • Results

    50.5 AIME24 score, up from 22.1 for the Qwen2.5-7B text-only baseline, with state-of-the-art multi-turn TIR performance and stable training.

  • Takeaways & Limitations

    SimpleTIR supports end-to-end Zero RL for multi-turn TIR while encouraging diverse reasoning patterns such as cross-validation and self-correction.

  • Takeaways & Limitations

    Void turns may not apply directly beyond multi-turn TIR, mathematical reasoning is capped at 10 turns, and training relies on a highly parallel code-execution sandbox.

Abstract

from arXiv · show

Large Language Models (LLMs) can significantly improve their reasoning capabilities by interacting with external tools, a paradigm known as Tool-Integrated Reasoning (TIR). However, extending TIR to multi-turn scenarios using Reinforcement Learning (RL) is often hindered by training instability and performance collapse. We identify that such instability is primarily caused by a distributional drift from external tool feedback, leading to the generation of low-probability tokens. This issue compounds over successive turns, causing catastrophic gradient norm explosions that derail the training process. To address this challenge, we introduce SimpleTIR , a plug-and-play algorithm that stabilizes multi-turn TIR training. Its core strategy is to identify and filter out trajectories containing void turns, i.e., turns that yield neither a code block nor a final answer. By removing these problematic trajectories from the policy update, SimpleTIR effectively blocks the harmful, high-magnitude gradients, thus stabilizing the learning dynamics. Extensive experiments show that SimpleTIR achieves state-of-the-art performance on challenging math reasoning benchmarks, notably elevating the AIME24 score from a text-only baseline of 22.1 to 50.5 when starting from the Qwen2.5-7B base model. Furthermore, by avoiding the constraints of supervised fine-tuning, SimpleTIR encourages the model to discover diverse and sophisticated reasoning patterns, such as self-correction and cross-validation.

1. Introduction

Multi-turn TIR offers iterative reasoning with external tools but remains difficult to train because distributional drift produces low-probability tokens and unstable gradients. SimpleTIR filters void-turn trajectories to stabilize training and improves mathematical reasoning performance while supporting diverse reasoning patterns.

  • Motivation: Multi-turn TIR lets LLMs iteratively reason, generate code, execute it, and use tool outputs in later turns.It can improve computational accuracy and address knowledge cutoffs through tools such as Python interpreters and search engines.
  • Training instability: External tool feedback can shift the model away from its pretrained distribution, increasing stochasticity and the generation of low-probability tokens.Although feedback tokens are masked from the policy loss, subsequent generations inherit and compound the distributional shift.
  • SimpleTIR: SimpleTIR filters trajectories containing void turns—responses with neither a complete code block nor a final answer—from policy updates.This blocks harmful high-magnitude gradients associated with problematic sequences and requires minimal integration changes.
  • Results: 50.5 AIME24 score, up from the Qwen2.5-7B text-only baseline of 22.1, demonstrates SimpleTIR’s reported improvement on challenging mathematical reasoning tasks.Ablations identify void-turn trajectory filtering as the crucial component for stabilizing training against naive multi-turn approaches.
  • Implications: SimpleTIR encourages diverse reasoning patterns, including cross-validation, progressive reasoning, and self-correction, without a cold-start SFT phase.The paper presents this as an advantage of its Zero RL approach.

2. Preliminaries

The paper formulates multi-turn TIR as a hierarchical process with turn-level and token-level decisions, while training one unified policy. GRPO supplies relative advantages, and feedback-token masking restricts policy loss to agent-generated responses.

  • Hierarchical MDP formulation: Multi-turn TIR is modeled as a Hierarchical Markov Decision Process separating turn-level decisions from token generation.The high-level policy governs conversational turns, while the low-level policy generates tokens within each turn.
  • Hierarchical MDP formulation: A trajectory alternates between model responses and subsequent tool feedback across turns.The conversation state before each turn contains the original question and all preceding responses and feedback.
  • High-level MDP: At the high level, actions generate complete responses, transitions append responses and tool feedback, and the terminal reward reflects overall trajectory success.The high-level state is the complete conversation history before the current turn.
  • Low-level MDP: At the low level, each action is one vocabulary token, transitions append selected tokens, and the low-level policy receives no intrinsic reward.Its goal is to complete the chosen high-level response.
  • Policy optimization: A single unified policy is trained with GRPO, which computes advantages from relative performance among grouped trajectories without a learned value function.The discount factors are set to 1 for both levels.
  • Feedback masking: Feedback-token masking accumulates loss only on model-generated response tokens, excluding environment-provided tool feedback from gradient computation.The objective uses a binary response-token mask together with a standard PPO clipped surrogate objective and importance-sampling ratio.

3. Methodology

SimpleTIR stabilizes multi-turn TIR by diagnosing low-probability-token failures and filtering trajectories containing void turns before policy updates. The method targets both gradient explosions and misaligned credit assignment caused by distributional drift across turns.

  • Instability diagnosis: Multi-turn TIR becomes unstable when external tool feedback shifts the input distribution, causing later generations to contain increasingly low-probability tokens.These tokens compound across turns and can produce gradient explosions and performance collapse.
  • Instability diagnosis: Low-probability tokens create gradient spikes because an unclipped importance ratio can explode when the old policy assigned the sampled token a minute probability.The gradient norm can also remain large through a probability-dependent term when the sampled token has low probability.
  • Instability diagnosis: Sparse terminal rewards misassign credit by penalizing valid early-turn reasoning together with faulty later-turn tokens in failed trajectories.Later-turn low-probability tokens are therefore not distinguished from earlier correct behavior.
  • SimpleTIR: High-perplexity masking and importance-ratio clipping do not resolve multi-turn instability because their thresholds are difficult to tune and they leave credit assignment unsolved.SimpleTIR is presented as a filtering approach that is agnostic to the specific RL algorithm and orthogonal to other RL improvements.
  • SimpleTIR: A void turn is an LLM response containing neither a complete code block nor a final answer, and it serves as a heuristic for detecting abnormal trajectories.Void turns often reflect distributional drift and premature end-of-sequence sampling.
  • SimpleTIR: SimpleTIR masks the policy loss for every trajectory containing a void turn, removing it before the GRPO update.This prevents large gradients from backpropagating and avoids penalizing successful early turns after a later collapse.

4. Experiments

Experiments evaluate SimpleTIR across mathematical reasoning tasks, training-turn settings, ablations, and emergent reasoning behaviors. Results show stable training, strong benchmark performance, and diverse multi-turn strategies.

  • Setup: SimpleTIR uses Qwen-2.5 base models, Math3-5 training data, and evaluation on Math500, AIME24, AIME25, AMC23, and Hmmt Feb 25.Evaluation reports average@32 scores at temperature 1.
  • Training Curves and Ablation Studies: Scaling interaction turns from 1 to 5 benefits SimpleTIR, while additional turns increase response length and Math500 scores but do not clearly improve AIME24.The results suggest tasks differ in how many external-feedback steps they require.
  • Training Results: SimpleTIR demonstrates significant improvement over base models and outperforms Zero RL baselines with or without TIR.It also outperforms baselines starting from the Qwen2.5-Math-7B series, including ToRL and Effective TIR.
  • Training Curves and Ablation Studies: High-importance-ratio and low-probability-token filtering remain unstable, whereas void-turn filtering produces more stable gradient norms and training curves.Terminating on void turns without filtering the resulting trajectories is also inferior because of misaligned credit assignment.
  • Emergence of Diverse Reasoning Behaviors: SimpleTIR responses combine Cross Validation, Progressive Reasoning, and Error Correction, with SimpleTIR-32B showing more progressive reasoning and error correction than ReTool.Reasoning-pattern frequencies were counted only among responses reaching the correct final answer.

5. Related Work

Related work establishes Zero RL for emergent reasoning and surveys reinforcement-learning approaches to tool use. Existing methods span search-based tasks, mathematical tool use, cold-start supervised fine-tuning, and direct base-model training.

  • Zero RL: Zero RL emerged from large-scale outcome-reward RL starting from unaligned base models and was followed by reproducible recipes such as SimpleRL.Open-Reasoner-Zero further studies scaling vanilla PPO with GAE and without KL regularization.
  • Tool-Integrated Reasoning: Recent tool-use RL work applies search tools to question answering and Python interpreters to mathematical calculations or enumerations.Examples include Search-R1, R1-Search, ReTool, ToRL, and Effective CIR.
  • Training Stability: The paper leaves related work on stabilizing reinforcement-learning training to Appendix A.

6. Conclusion

SimpleTIR stabilizes and enhances multi-turn TIR under Zero RL by filtering trajectories containing void turns. It improves mathematical reasoning performance while encouraging diverse reasoning patterns without cold-start human data.

  • 6. Conclusion: SimpleTIR filters trajectories with void turns to address harmful negative samples and stabilize multi-turn TIR training.The framework is designed for the Zero RL setting.
  • 6. Conclusion: SimpleTIR improves reasoning performance across a variety of mathematical benchmarks, including state-of-the-art results from Qwen2.5-7B.
  • 6. Conclusion: SimpleTIR encourages diverse reasoning patterns without relying on cold-start human data.

A.1. Stabilizing RL Training

Multi-turn TIR RL can become unstable through distribution shift from external tool feedback and compounding low-probability generations. SimpleTIR’s trajectory filtering targets this TIR-specific failure mechanism.

  • A.1. Stabilizing RL Training: Existing stabilization approaches include entropy control, importance-ratio constraints, data filtering, and updates focused on low-probability or high-entropy tokens.
  • A.1. Stabilizing RL Training: Void turns and compounding errors can precipitate collapsed generations and gradient spikes during Zero RL.A void turn can disrupt subsequent decoding and lead to corrupted outputs.

B.2. Response with Emergent Reasoning Behaviors

SimpleTIR elicits diverse multi-turn behaviors without instruction-level biases. Qualitative rollouts and quantitative pattern analysis support richer reasoning strategies than cold-start SFT.

  • B.2. Response with Emergent Reasoning Behaviors: SimpleTIR produces diverse multi-turn behaviors without instruction-level biases.
  • B.2. Response with Emergent Reasoning Behaviors: Progressive reasoning with code improvement is illustrated as an emergent behavior in the qualitative rollouts.
  • B.2. Response with Emergent Reasoning Behaviors: The reported cases support richer strategies from Zero RL with TIR than from cold-start SFT.

C. Experiments

The experiments specify a standardized multi-turn TIR interaction format and present example responses, including a void turn followed by corrupted outputs. The supplied passages also include a geometry-solving example involving tangent circles.

  • C. Experiments: The prompt template uses complete Python scripts, printed intermediate quantities, and a standardized final-answer channel.These choices are intended to stabilize interpreter interaction and detect valid tool calls versus void turns.
  • C. Experiments: Table 4 shows a multi-turn AIME24 response whose void turn is followed by corrupted responses during and after that turn.
  • C. Experiments: The geometry example asks for a triangle’s inradius from arrangements of eight radius-34 circles and 2024 radius-1 circles.
  • C. Experiments: For the larger circles, the example computes consecutive-center spacing as 68 and the seven-gap chain distance as 476.
  • C. Experiments: The example relates the circle arrangement to triangle side tangencies, vertical distances, and the inradius relationship.

C.2. Hyperparameters

SimpleTIR uses multi-turn rollouts with bounded response length and interaction turns, while tuning PPO and sampling choices to preserve exploration and control training instability.

  • C.2. Hyperparameters: 16,384-token responses and 5 interaction turns bound the initial episode length while accommodating complete code blocks and execution traces.The response-length cap avoids premature truncation, while the turn limit bounds compute and permits planning, execution, and verification.
  • C.2. Hyperparameters: Temperature = 1.0 preserves candidate diversity, with selection and credit assignment replacing explicit entropy bonuses for exploration.The rollout design uses sampling rather than an entropy term to encourage diverse tool-use strategies.
  • C.2. Hyperparameters: 1,280 responses per update and n = 16 rollouts per prompt provide broad tool-use coverage while keeping variance manageable.The batch and rollout settings are presented as a balance between coverage and variance.
  • C.2. Hyperparameters: PPO uses a 0.2 / 0.28 low/high clip ratio, 4 epochs, train batch size 512, learning rate 1e-6, and gradient clipping 1.The settings constrain policy updates, reuse samples, balance gradient quality and memory, and limit instability from long contexts and sparse-reward advantages.
  • C.2. Hyperparameters: Undiscounted episodes use γ = 1.0, while entropy and KL coefficients are both zero.Exploration is induced through temperature and multi-rollout sampling rather than entropy or KL regularization.

C.3. Prompt for Extracting Reasoning Patterns

The paper uses a constrained labeling prompt to identify reasoning patterns in multi-turn TIR trajectories, enabling consistent frequency comparisons across methods.

  • C.3. Prompt for Extracting Reasoning Patterns: The labeling prompt first records bookkeeping features such as code-block counts and whether execution matches the final answer.These checks precede the behavioral pattern queries.
  • C.3. Prompt for Extracting Reasoning Patterns: It then queries four patterns: decomposed sub-solves, induction, self-verification, and bug-fixing through additional code.The first pattern specifically concerns non-final execution.
  • C.3. Prompt for Extracting Reasoning Patterns: A constrained output format improves label consistency and supports fair frequency comparisons across methods.The stated purpose is to make extracted behavior frequencies comparable.
  • C.3. Prompt for Extracting Reasoning Patterns: Table 5 presents a SimpleTIR-7B AIME24 example of progressive reasoning and code improvement across turns.Responses from different turns are separated by vertical lines.
Loading 2509.02479v2…