Source-linked AI summary
Agent RL Scaling Law: Agent RL with Spontaneous Code Execution for Mathematical Problem Solving
Xinji Mai, Haotian Xu, Zhong-Zhi Li, Xing W, Weinong Wang, Jian Hu, Yingying Zhang, Wenqiang Zhang
TL;DR
The paper addresses how base LLMs can autonomously learn to use code execution for precise mathematical reasoning, a capability less studied than text-only RL or controlled tool use. It introduces ZeroTIR, trains models with outcome-based RL and a decoupled execution environment, and finds predictable training dynamics alongside stronger benchmark performance than non-tool baselines. The authors frame these results as a foundation for studying autonomous tool learning, while noting that the precise mathematical form of the scaling relationships remains unresolved.
Problem
LLMs struggle with precise mathematical computation, while evidence about how agents autonomously acquire external tool use through outcome-based RL remains limited.
Method
ZeroTIR trains base LLMs to spontaneously generate and execute Python code through outcome-based RL using a decoupled code-execution environment and interactive rollout frameworks.
Results
ZeroTIR significantly outperforms non-tool ZeroRL baselines and SFT-based TIR methods; 7B ZTRL reaches 52.3% average across AIME24, AIME25, and MATH500.
Takeaways & Limitations
Training progression produces predictable changes in tool usage frequency, code quality, response length, and task accuracy, providing a reproducible basis for studying Agent RL scaling.
Takeaways & Limitations
The work demonstrates the qualitative existence of scaling laws but does not rigorously determine their precise mathematical form.
Abstract
from arXiv · showhide
Large Language Models (LLMs) often struggle with mathematical reasoning tasks requiring precise, verifiable computation. While Reinforcement Learning (RL) from outcome-based rewards enhances text-based reasoning, understanding how agents autonomously learn to leverage external tools like code execution remains crucial. We investigate RL from outcome-based rewards for Tool-Integrated Reasoning, ZeroTIR, training base LLMs to spontaneously generate and execute Python code for mathematical problems without supervised tool-use examples. Our central contribution is we demonstrate that as RL training progresses, key metrics scale predictably. Specifically, we observe strong positive correlations where increased training steps lead to increases in the spontaneous code execution frequency, the average response length, and, critically, the final task accuracy. This suggests a quantifiable relationship between computational effort invested in training and the emergence of effective, tool-augmented reasoning strategies. We implement a robust framework featuring a decoupled code execution environment and validate our findings across standard RL algorithms and frameworks. Experiments show ZeroTIR significantly surpasses non-tool ZeroRL baselines on challenging math benchmarks. Our findings provide a foundational understanding of how autonomous tool use is acquired and scales within Agent RL, offering a reproducible benchmark for future studies. Code is released at \href{https://github.com/yyht/openrlhf_async_pipline}{https://github.com/yyht/openrlhf\_async\_pipline}.
1 Introduction
The paper studies how base LLMs autonomously acquire Python code execution through outcome-based RL for mathematical reasoning, and characterizes predictable scaling dynamics. It introduces ZeroTIR and reports improved performance over non-tool and supervised tool-integrated baselines.
- Motivation: LLMs often struggle with precise, multi-step mathematical reasoning because next-token prediction favors likely text over computational correctness.Existing tool-augmented approaches commonly rely on supervised trajectories, prompt structures, or predefined triggers.
- Framework: The work combines mainstream RL frameworks and algorithms with an environment server to support empirical analysis and reproducible study of agentic tool use.The listed frameworks include Open-Reasoner-Zero and OpenRLHF, while the algorithms include PPO and Reinforce++.
- Contribution: The paper identifies Agent RL Scaling Law governing autonomous acquisition of spontaneous code execution skills during ZeroTIR training.The study examines relationships among training progression, tool usage, response length, and task accuracy.
- Approach: ZeroTIR trains base LLMs to spontaneously generate and execute Python code for mathematical problem-solving through reinforcement learning.The framework is designed to study tool acquisition without starting from a fine-tuned model with existing tooling capabilities.
- Results: ZTRL models trained with ZeroTIR significantly outperform non-instrumental ZeroRL baselines and SFT-based tool-integrated reasoning methods on challenging mathematical benchmarks.The reported comparison includes a 7B ZTRL model trained from a general base model.
2 Related Works
Related work spans tool-integrated reasoning, autonomous LLM agents, and outcome-based Agent RL. This paper focuses on systematic analysis of spontaneous code-execution learning rather than controlled or search-focused tool invocation.
- Tool-Integrated Reasoning: Tool-integrated reasoning augments LLMs with calculators, search engines, or code interpreters to address calculation and factual lookup limitations.Prior methods often use supervised tool-invocation examples, prompts, or controlled triggers.
- Agent Reinforcement Learning: Agent RL trains LLM agents for autonomous planning, decision-making, environmental interaction, and tool use.Examples include outcome-based training for autonomous search-engine querying.
- Agent Reinforcement Learning: Outcome-based rewards can foster exploration and emergent strategies, including autonomous use of computational tools for mathematical reasoning.Contemporaneous work such as TORL applies ZeroRL with code execution to mathematics.
- Positioning: The paper distinguishes its contribution by systematically identifying and analyzing Agent RL Scaling Law for code-execution tool acquisition.This focus builds on ZeroRL while differing from search-focused agents.
3 Methodology
ZeroTIR trains base LLMs with reinforcement learning to generate and execute Python code during mathematical reasoning. Its methodology combines policy-gradient optimization, masked handling of environment-generated tokens, stability techniques, and decoupled asynchronous tool interaction.
- ZeroTIR trains a base LLM to autonomously use a Python execution environment for mathematical problem-solving through reinforcement learning.
- RL algorithms: PPO optimizes a policy and value network, while REINFORCE-style methods estimate policy gradients directly from sampled trajectories.PPO uses a clipped surrogate objective; REINFORCE variants use baseline-based advantage estimates without the same learned value function.
- Token handling: Environment-generated tokens are excluded from value-loss and policy-gradient updates because they are not produced by the LLM policy.The framework masks these states when computing value loss and can set their advantages to zero during policy-loss calculation.
- Training objective: The objective maximizes expected outcome-based reward for trajectories interacting with the code environment while regularizing divergence from a reference policy.The input is x, the trajectory is y, and β controls KL regularization.
- Stability and rollout control: Replay filtering retains intermediate-accuracy response groups, while dynamic stop tokens coordinate iterative reasoning, code generation, execution, and feedback integration.Groups above 0.8 or below 0.2 accuracy are filtered out, and the state-machine interaction is more efficient than post-hoc code parsing.
- Environment interaction: A decoupled network-accessible execution service isolates failures from training and supports independent scaling, while asynchronous pipelining overlaps rollout generation with parameter updates.The asynchronous pipeline was approximately 1.6 times faster than basic asynchronous rollout and over 4 times faster than the initial synchronous interaction.
4 Experiments
Experiments evaluate ZeroTIR from pretrained Qwen base models across RL frameworks, algorithms, datasets, and mathematical benchmarks. Results show strong gains from tool-integrated reasoning and systematic effects of interaction budgets, model scale, training data, decoding, and training dynamics.
- Experimental setup: ZeroTIR is evaluated from Qwen 2.5 Base 7B/32B weights using OpenRLHF and Open-Reasoner-Zero, PPO and Reinforce++, and verifiable math datasets.Evaluation covers MATH500, AIME24/25, HMMT, CMIMC, and other challenging mathematical reasoning benchmarks.
- Comparative performance: 52.3% average accuracy across AIME24, AIME25, and MATH500 lets 7B ZTRL surpass SimpleRL-Zero at 39.1%, Eurus-2-PRIME at 39.7%, and Qwen 2.5 Math Instruct at 41.6%.ZTRL also reaches 54.0% versus TORL’s 51.8% on the same three benchmarks, despite starting from general Qwen 2.5 Base rather than a math-specialized base.
- Interaction budgets: Increasing Nmax from zero to four or twenty raises average scores by as much as fifteen percentage points, with gains tapering beyond four calls.Accuracy increases monotonically with the interaction cap across model scales.
- Model and algorithm scaling: Performance increases with model size: 32B surpasses 7B, which surpasses 1.5B under identical hyperparameters.At 7B, Reinforce++ and PPO reach similar final accuracies, but Reinforce++ converges roughly three hundred steps sooner.
- Data and decoding: DeepMath achieves 60% Max on HMMT Feb. 25, while Orz-trained models reach 53% Maj on CMIMC versus 33% for DeepMath.The results indicate a trade-off between peak reasoning and majority robustness, with decoding entropy affecting Max, Maj, and Avg.
- Training dynamics: Code Proportion initially dips and then rises for TIR-enabled models, while Code in Correct rises with Raw Reward Avg during training.These dynamics associate increasing effective code use with task success, although response length does not perfectly mirror reward improvements in every setting.
- Training dynamics: More than 90% of correct code-assisted responses use one code execution, while fewer than 10% use two or more calls despite larger permitted budgets.Average code count often stabilizes between one and two calls, suggesting agents predominantly learn few-interaction strategies.
- Joint budget scaling: Training and inference interaction budgets interact: models trained with more interactions benefit most from generous inference budgets but can underperform low-budget models in the one-step regime.On AIME24/25, the 8-interaction versus 2-interaction training gap exceeds ten percentage points at sixteen inference steps; on HMMT24/25 it is around seven points.
5 Conclusion
This work studies autonomous Python code execution acquired by base LLMs through outcome-based RL and characterizes predictable scaling in tool use and mathematical reasoning. It also identifies interaction-budget patterns and acknowledges that precise quantitative scaling laws remain unresolved.
- ZeroTIR studies spontaneous Python code execution for mathematical reasoning in base LLMs trained with outcome-based reinforcement learning.
- Increasing inference interactions improves benchmark performance, with gains flattening near 8 interactions and varying by dataset.
- Training progression produces predictable changes in tool-usage frequency, code quality, response length, and task accuracy.
- The study qualitatively demonstrates scaling-law existence, while leaving the precise mathematical form of these relationships for future work.
- The resulting framework provides a reproducible basis for studying autonomous tool learning and Agent RL scaling effects.