Source-linked AI summary
Learning to Use Tools: Reinforcement Learning for Tool-Integrated Mathematical Reasoning
Minghui Xu, Zi Wang
TL;DR
The paper asks whether calculator tools can improve reliable mathematical reasoning and whether reinforcement learning can further improve tool-integrated performance on Countdown. It constructs tool-use SFT data, applies several on-policy RL methods with verifiable rewards, and evaluates them on a fresh held-out benchmark. Tool integration improves SFT and RL baselines, while Tool-DAPO is strongest among the tested RL variants; the authors note that RL remains limited when no correct trajectory is sampled.
Problem
Arithmetic and logical mistakes remain difficult for reasoning-oriented LLMs on multi-step tasks requiring exact computation and verification.
Method
The paper trains calculator-using SFT policies, applies RLOO, RLOO++, GRPO, and DAPO with automatically verifiable final-answer rewards, and evaluates on Countdown benchmarks.
Results
Tool integration consistently improves SFT and RL models, and Tool-DAPO achieves the strongest performance among the tested RL methods.
Takeaways & Limitations
Calculator tools reduce arithmetic and verification failures, while RL increases the probability of correct reasoning traces in this Countdown setting.
Takeaways & Limitations
RL helps most when a prompt already yields at least one correct trajectory; prompts with zero correct samples remain difficult.
Abstract
from arXiv · showhide
Current large language models (LLMs) increasingly benefit from external tool integration, especially for tasks requiring reliable computation and verification. Motivated by this, we study calculator tool calling for improving mathematical reasoning on the Countdown task. We first analyze reasoning failures and find that calculation errors account for a substantial portion of incorrect responses. We then construct supervised fine-tuning datasets to teach the model useful tool-use patterns and how to interpret returned outputs. Building on this tool-formatted policy, we apply several on-policy reinforcement learning methods, including RLOO, RLOO++, GRPO, and DAPO, using automatically verifiable final-answer rewards. To enable a more reliable evaluation, we construct a fresh 1,024-problem held-out Countdown benchmark with no exact overlap with the training data. Our results show that calculator tool integration consistently improves both SFT and RL baselines, yielding roughly 10 percentage-point gains across pass@k. Among the RL methods, Tool-DAPO achieves the strongest performance, improving pass@1 from 35.8% for Tool-SFT to 66.0%. Further analysis shows that RL encourages more effective tool use even when only final-answer rewards are provided. These findings suggest that tool integration reduces arithmetic and verification errors, while RL increases the probability of correct reasoning traces.
1 Introduction
The paper studies calculator-integrated reasoning on Countdown, combining supervised tool-use training with online reinforcement learning and fresh evaluation. It argues that tools improve both SFT and RL models, with Tool-DAPO strongest among tested methods.
- Motivation: Countdown tests combinatorial reasoning and exact arithmetic, while calculator validation and correction target arithmetic and verification errors.The task combines given numbers with arithmetic operations to reach a target, and final answers are automatically verifiable.
- Approach: The pipeline builds tool-integrated SFT data and applies RLOO, RLOO++, GRPO, and DAPO using automatically computed verifier rewards.Tool rollouts execute calculator expressions and insert returned observations before generation continues.
- Evaluation: Evaluation covers the original 50-problem public test split and a fresh 1,024-problem held-out benchmark without exact training overlap.The larger benchmark is designed to provide a more reliable evaluation.
- Contribution: The study presents live calculator execution during model rollouts as a core component of its tool-integrated Countdown training and evaluation pipeline.This pipeline supports direct interaction between reasoning generation and calculator outputs.
- Findings: Tool integration consistently improves both SFT and RL models, while Tool-DAPO achieves the strongest performance among the tested methods.The paper also analyzes error categories, tool-call dynamics, and correctness distributions to characterize improvements and remaining failures.
2 Related Work
Related work positions the paper at the intersection of reinforcement learning for reasoning and tool-augmented language-model inference. Prior studies motivate both online optimization with verifiable rewards and external actions for improving reliability.
- Reinforcement learning for reasoning: Recent large-scale reinforcement learning has been used to encourage long-chain reasoning in models such as OpenAI o1 and DeepSeek-R1.These examples situate the work within sequence-level training for reasoning capabilities.
- Policy optimization: IPO and REINFORCE-style methods such as RLOO provide alternatives for preference or online policy optimization, especially when final answers are automatically verifiable.RLOO reduces reliance on a learned critic through its leave-one-out formulation.
- Tool use: Tool-augmented reasoning methods interleave language-model reasoning with external actions, with newer work training tool invocation through reinforcement learning.ReAct, ReTool, and in-context reinforcement learning represent successive approaches to integrating external tools.
- Coverage and sampling: Prior RLVR studies suggest that reinforcement learning may increase the probability of already-available correct trajectories without expanding high-pass@k reasoning coverage.Reported gains at pass@1 can coexist with unchanged or lower performance at larger pass@k.
3 Method
The method builds calculator-aware Countdown training data, then applies several on-policy RL algorithms with automatically verifiable rewards to study whether tool use improves reasoning.
- Tool-integrated SFT: Tool-SFT teaches models where to place calculator checks and how to condition reasoning on returned observations.The tool format uses executable expressions and environment-inserted observations during generation.
- Tool-integrated SFT: The SFT dataset construction corrects arithmetic-error prefixes, samples model continuations, and assembles calculator-augmented completions.Correct continuations are selected when available; otherwise, the exact solver supplies the answer.
- RLOO: RLOO estimates response advantages using each reward relative to the average reward of the other sampled responses.Its loss combines the policy-gradient term with entropy regularization and KL divergence from a frozen SFT reference policy.
- RLOO++: RLOO++ applies batch-wide advantage normalization and rescales advantages to a target standard deviation of 0.3.The modification is intended to stabilize updates when rewards are sparse or sampled groups have nearly uniform rewards.
- GRPO: GRPO estimates baselines from group scores instead of a value model, while using clipped token-level importance weighting and a direct KL penalty.This design reduces memory and computational cost relative to PPO.
- Tool-DAPO: Tool-DAPO normalizes advantages within prompt groups, filters uninformative all-correct or all-wrong groups, and uses asymmetric policy-ratio clipping.In the run, 60.7% of generated prompt groups were filtered; the update still permits increasing high-reward tool-using trajectories.
4 Experimental Setup
The experiments evaluate no-tool and calculator-enabled SFT/RL models on Countdown using pass@k, including a fresh 1,024-problem held-out benchmark to reduce overlap and estimation concerns.
- Task and metric: Countdown instances contain a target and either three or four numbers, and correctness requires using every number exactly once to reach the target.Allowed arithmetic operations determine whether the final expression is valid.
- Task and metric: pass@k is computed from k sampled responses per problem.The metric summarizes performance across multiple sampled completions.
- Compared systems: The comparisons include no-tool SFT, Tool-SFT, no-tool RLOO, Tool-RLOO, Tool-RLOO++, Tool-GRPO, and Tool-DAPO.Tool models may call the calculator during generation and receive observations before answering.
- Held-out evaluation: The fresh held-out test set contains 1,024 problems with no exact training overlap under the target-and-sorted-numbers key.It supplements the 50-example public split, whose pass@k estimates are described as noisy.
5 Results
The results diagnose arithmetic and planning failures, compare tool-integrated and non-tool models on fresh and public benchmarks, and evaluate several tool-integrated RL methods. Tool use improves SFT and RL performance, while Tool-DAPO provides the strongest reported RL improvement.
- SFT failure analysis: Calculation errors occur across both difficulty levels, while target/planning errors become substantially more common in 4-number problems.The error analysis separates local arithmetic mistakes from broader target-aware planning failures.
- Tool integration: The evaluation compares no-tool and tool-augmented SFT and RLOO variants on a fresh 1,024-problem benchmark and the original 50-problem split.The fresh benchmark is presented as the clearer comparison because the small public split has wide, overlapping confidence intervals.
- Tool integration: With matched training budgets, Tool-RLOO raises pass@1 from 50.6% to 56.6% and pass@16 from 66.6% to 74.0%.Extending Tool-RLOO training to 200 steps further reaches 60.7% pass@1 and 76.5% pass@16.
- RL method comparison: Tool-RLOO and Tool-RLOO++ improve pass@1 over Tool-SFT, but Tool-RLOO++ does not consistently improve high-k performance.Tool-GRPO exceeds Tool-SFT on the 1,024-problem set, while its small-test-set behavior is less stable.
- RL method comparison: Tool-DAPO increases pass@1 by 9.3 percentage points over Tool-RLOO and also improves pass@16.Its dynamic sampling filters zero-variance prompt groups before policy updates; 60.7% of generated groups were filtered in this run.
6 Discussion
Across training, reinforcement learning shifts sampled responses toward highly correct sets, while Tool-DAPO also reduces examples with no correct samples and Tool-RLOO increases useful calculator invocation.
- RL increases the mass near fully correct response sets, reinforcing trajectories that already contain correct reasoning traces.
- Examples with zero correct responses remain difficult because they provide limited positive signal for policy improvement.
- Tool-DAPO reaches more fully correct examples with fewer optimization steps and reduces examples with zero correct samples.
- Tool-RLOO increases executable calculator calls from below one to approximately one per sampled rollout while improving Countdown accuracy.
- Figures 5–7 track the fraction of examples with each count of correct responses among 16 sampled responses across RLOO, Tool-RLOO, and Tool-DAPO training.
7 Conclusion
The paper finds that calculator tool use and reinforcement learning are complementary for Countdown reasoning. Tool-DAPO is strongest among the tested RL variants, but RL remains limited on prompts where no sampled trajectory is correct.
- Calculator integration improves both SFT and RL models, while Tool-DAPO achieves the strongest result among the tested RL variants.
- RL helps most when a prompt already yields at least one correct sampled trajectory; prompts with zero correct samples remain difficult.
- Future work should combine tool execution with stronger planning or search to discover correct trajectories for hard prompts.
A Additional Analysis
The additional analysis contrasts a plain RLOO failure with a Tool-RLOO success on the same arithmetic challenge. Calculator verification converts a promising decomposition into a checked final expression.
- The calculator returns 44 for (99 - 11) / (48 - 46), supporting the final answer after the proposed plan is checked.
- Plain RLOO explores nearby subtractive patterns but misses the decomposition 48 −46 = 2 and 99 −11 = 88.
- Tool-RLOO identifies the key decomposition and invokes the calculator at the verification point.
- The tool distinguishes a valid expression from the near-miss expressions produced in the plain RLOO trace.
B Implementation Details
The implementation compares RL configurations with matched or method-specific hyperparameter settings and enables calculator execution only in tool-augmented rollouts. Training costs vary across methods and budgets on a single H100 GPU.
- All RL experiments ran on a single H100 GPU, with reported runtimes ranging from about 5 hours to 16.15 hours.
- RLOO and Tool-RLOO share the same RL hyperparameters, while Tool-RLOO additionally enables tool execution during rollout.
- Tool-RLOO++ follows Tool-RLOO and modifies advantage normalization.
- Tool-GRPO uses a single-update trainer where the importance ratio is one and PPO-style clipping is inert.
- Tool-DAPO has its own reported hyperparameter configuration.