Source-linked AI summary
Towards Effective Code-Integrated Reasoning
Fei Bai, Yingqian Min, Beichen Zhang, Zhipeng Chen, Wayne Xin Zhao, Lei Fang, Zheng Liu, Zhongyuan Wang, Ji-Rong Wen
TL;DR
Reasoning models remain constrained by issues such as imprecise numerical computation, while tool-augmented RL can make training more intricate and unstable. The paper develops code-integrated reasoning with training strategies that balance exploration and stability. It reports improved mathematical reasoning benchmark performance and analyzes gains in capability boundaries and reasoning efficiency.
Problem
Reasoning models remain constrained by imprecise numerical computation and restricted knowledge coverage, while tool-augmented RL adds training complexity and instability challenges.
Source: §1 · p. 1Method
The paper trains code-integrated reasoning models with strategies balancing exploration and stability, including progressive tool budgets and precise code-block matching.
Source: §1 · p. 2Results
CIR achieves 52.4% average accuracy across evaluation benchmarks, consistently outperforming baseline models.
Source: §3.2.1 · p. 6Takeaways & Limitations
Code integration extends reasoning models’ capacity boundaries and improves reasoning efficiency, with gains varying across problem types.
Source: §1 · p. 2, §5 · p. 13The paper leaves comparison with long-CoT models for future work because their lengthy responses would raise training costs relative to these experiments.
Source: §4.2 · p. 11
Abstract
from arXiv · showhide
In this paper, we investigate code-integrated reasoning, where models generate code when necessary and integrate feedback by executing it through a code interpreter. To acquire this capability, models must learn when and how to use external code tools effectively, which is supported by tool-augmented reinforcement learning (RL) through interactive learning. Despite its benefits, tool-augmented RL can still suffer from potential instability in the learning dynamics. In light of this challenge, we present a systematic approach to improving the training effectiveness and stability of tool-augmented RL for code-integrated reasoning. Specifically, we develop enhanced training strategies that balance exploration and stability, progressively building tool-use capabilities while improving reasoning performance. Through extensive experiments on five mainstream mathematical reasoning benchmarks, our model demonstrates significant performance improvements over multiple competitive baselines. Furthermore, we conduct an in-depth analysis of the mechanism and effect of code-integrated reasoning, revealing several key insights, such as the extension of model's capability boundaries and the simultaneous improvement of reasoning efficiency through code integration. All data and code for reproducing this work are available at: https://github.com/RUCAIBox/CIR.
1 Introduction
Reasoning models can deliberate yet remain limited by imprecise numerical computation and restricted knowledge coverage, while tool-augmented RL introduces its own training challenges. This paper studies code-integrated reasoning and proposes training strategies that balance exploration and stability while analyzing its effects.
Motivation
- Tool-augmented RL must teach both reasoning and tool use, while tool interruptions and added interaction rounds can destabilize training and increase computational demands.Prior work addressed some challenges, but comprehensive solutions considering these factors remained largely unexplored.Source: §1 · p. 2
Code-Integrated Reasoning
- Code-integrated reasoning uses generated code as part of the reasoning process, integrating natural language with formal programming through a code interpreter.Unlike search queries or mathematical expressions, generated code snippets directly participate in reasoning.Source: §1 · p. 2
Training Strategies
- The proposed training strategies balance exploration and stability, including removing the KL term, using clip-higher, increasing interaction budgets progressively, and stabilizing tool feedback and code boundaries.The authors report state-of-the-art performance across multiple mainstream benchmarks, outperforming competitive baselines.Source: §1 · p. 2
Analysis of Code Integration
- Code integration extends reasoning models’ capacity boundaries and produces more concise, efficient paths, though gains vary by problem type and geometry improves minimally.Executable but logically flawed code can hinder performance, while non-executable code may still prompt reflection and revision.Source: §1 · p. 2
2 Methodology
The methodology combines inference-time code execution with interactive tool-augmented reinforcement learning, then addresses instability by balancing exploration and stability. It organizes the challenges and mitigation strategies into distinct stages.
2.1 Code-Integrated Reasoning
- At inference, the model selectively generates code when precise computation or symbolic transformations are useful, executes it externally, and reintegrates results into its reasoning context.The updated context guides subsequent reasoning toward a final answer; generated code also serves as a formal representation of reasoning steps.Source: §2.1 · pp. 2–3
2.2 Tool-Augmented RL Framework
- Tool-augmented RL incorporates interaction information into training trajectories so external execution feedback can inform policy updates.The framework extends conventional RL trajectories, which otherwise exclude external feedback.Source: §2.2 · p. 3
2.3.1 Exploration-Stability Trade-off Issues in Tool-Augmented RL
- Tool-augmented RL instability is associated with disrupted interaction boundaries, shifts between model reasoning and tool feedback, and response homogenization under fixed interaction budgets.Fixed budgets can narrow rewarded behaviors, making outputs more deterministic and reducing the diversity of reasoning explored.Source: §2.3.1 · p. 4
2.3 RL with Exploration-Stability Balance
- The approach groups its targeted training strategies into stability maintenance and exploration enhancement to address instability, distributional shifts, and reduced exploration.The methodology first frames the optimization challenges and then presents corresponding mitigations.Source: §2.3.2 · p. 4, §2.3 · p. 4
2.3.2 Improved Training Strategies
- For stability, precise code-block matching, masking external-tool feedback during loss calculation, and disabling the entropy bonus target noisy boundaries, distribution mismatch, and uncontrolled entropy growth.Exact matching triggers execution on complete, well-formed code blocks rather than relying on stop tokens.Source: §2.3.2 · pp. 4–5, Figure 1 · p. 5
- For exploration, clip-higher preserves policy entropy, while progressively increasing tool invocations encourages broader reasoning paths and delays early convergence.Clip-higher increases the upper bound in the surrogate loss; progressive scheduling aims to maintain output diversity longer.Source: §2.3.2 · pp. 5–6
3 Experiments
Experiments compare code-integrated RL (CIR) with text-only and code-integrated baselines across five mathematical reasoning benchmarks. CIR leads the reported comparisons, while training analyses trace reward, response length, accuracy, and code-use patterns as the interaction budget expands.
3.1 Experiment Settings
- Experiments train Qwen2.5-Math-7B and non-thinking Qwen3-8B on 38,000 STILL-3 samples, comparing text-only and code-integrated baselines across five math benchmarks.Training uses REINFORCE++ and progressively increases tool-interaction rounds from 2 to 4; evaluation uses AVG@16 on four benchmarks and greedy decoding on MATH500.Source: §3.1 · p. 6
3.2.1 Main Results
- CIR reaches 52.4% average accuracy, exceeding EURUS-2-7B-PRIME by 14.8 points, OAT-ZERO-7B by 13.1 points, and TORL-7B’s 50.5%.The paper reports consistent superiority across all evaluation benchmarks; Table 1 reports greedy decoding for MATH500 and AVG@16 for the other four benchmarks.Source: §3.2.1 · pp. 6–7, Table 1 · p. 7
3.2.2 Learning Dynamics
- Training reward rises as the interaction budget expands, while response length drops early and later grows with the increasing number of interactions.The budget progresses from 2 to 3 and then 4; the authors describe the early shortening as a shift toward tool invocation.Source: Figure 2 · p. 7, §3.2.2 · pp. 7–8
- Test accuracy improves as the interaction budget expands, with the analysis reporting this trend on AIME2025 and MATH500.The Figure 3 caption identifies AIME2024 and MATH500, while the accompanying prose names AIME2025 and MATH500.Source: Figure 3 · p. 8, §3.2.2 · p. 8
- Code generation exceeds 85% early and approaches 100% later, while pass rate declines as the budget expands, which need not indicate poorer performance.The model averages more than one code snippet per response from early training; pass rate first rises during the initial 100 steps before declining.Source: Figure 4 · p. 8, §3.2.2 · p. 8
4 Mechanistic Analysis of Code-Integrated Reasoning
The mechanistic analysis examines how code execution affects reasoning capacity, response efficiency, and performance across problem types. It finds that benefits depend on both execution feedback and the kind of reasoning a problem requires.
4.1 Impact on Model’s Capability Boundaries
- Across all four benchmarks, code-integrated models outperform text reasoning in PASS@k; Basecode leads at large k, while removing execution sharply lowers RL performance.The authors interpret the drop for RLtext as evidence that code generation and execution become integrated model functionalities after RL training.Source: Figure 6 · p. 9, §4.1 · p. 10
4.2 Comparison between Code-Integrated Reasoning and Long-CoT Reasoning
- The model reaches nearly 80% of DEEPSEEK-R1-DISTILL-QWEN-7B accuracy at under 15% of its response length, and matches QWEN3-8B (THINKING-10K) with under 20% of its response length.Code-integrated responses typically start with a brief, complete solution overview before executable code, unlike long-CoT responses that continue decomposing and reflecting.Source: Table 2 · p. 10, §4.2 · p. 10
4.3 Effect of Non-Executable Code in Code-Integrated Reasoning
- 36.0% of incorrect responses stemmed from executable but logically flawed code; about one-third of correct responses still contained non-executable code.Executable feedback can prematurely end reasoning, so code executability alone does not reliably indicate answer quality.Source: §4.3 · p. 12
- 39.4% of correct responses included non-executable code, while their Full Pass Rate was 60.6% and Final Pass Rate was 65.5%, consistent with successful code revision.The authors suggest error feedback may prompt revisions that contribute to valid reasoning paths.Source: §4.3 · p. 12, Table 4 · p. 12
4.4 Performance Gain Analysis of Code-Integrated Reasoning
- Code integration improves algebra, number theory, and combinatorics more than geometry across AIME2024 and OlymMATH; geometry gains remain marginal even with RL.The authors attribute the contrast to geometry’s need for correct spatial frameworks and conceptual reasoning before computation can help.Source: §4.4 · p. 12, Figure 7 · p. 13
5 Conclusion
The paper presents training strategies for code-integrated reasoning that balance exploration and stability, and analyzes how code integration affects reasoning capacity and efficiency.
5 Conclusion
- The QWEN2.5-MATH-7B-based model achieves 42.3% accuracy on AIME2024; the study also finds code integration extends reasoning capacity boundaries and improves efficiency.The framework focuses on code execution and is described as extensible to other external tools, which the authors leave for future work.Source: §5 · p. 13