START: Self-taught Reasoner with Tools

Chengpeng Li; Mingfeng Xue; Zhenru Zhang; Jiaxi Yang; Beichen Zhang; Xiang Wang; Bowen Yu; Binyuan Hui; Junyang Lin; Dayiheng Liu

Source-linked AI summary

START: Self-taught Reasoner with Tools

Chengpeng Li, Mingfeng Xue, Zhenru Zhang, Jiaxi Yang, Beichen Zhang, Xiang Wang, Bowen Yu, Binyuan Hui, Junyang Lin, Dayiheng Liu

arXiv:2503.04625v2Mar 6, 2025cs.CL

TL;DR

Large reasoning models can hallucinate and struggle with complex computations when relying only on internal reasoning. START combines hint-guided tool use with self-training to integrate long CoT and Python execution, and evaluations report improved performance across challenging reasoning benchmarks. The paper limits its external-tool study to a Python interpreter.

Problem
Long-CoT reasoning models remain limited by hallucinations and inefficiencies caused by reliance on internal reasoning without external tools.
Method
START uses Hint-infer to elicit Python-interpreter calls and Hint-RFT to score, filter, modify, and fine-tune tool-using reasoning trajectories.
Results
START markedly surpasses existing tool-integrated and long-CoT models across mathematical, scientific, coding, and GPQA evaluations.
Takeaways & Limitations
Integrating external tools into long CoT enables START to perform complex computations and self-debugging as an open-source tool-integrated reasoning model.
Takeaways & Limitations
The study focuses exclusively on Python interpreter integration, leaving broader toolsets for future work.

Abstract

from arXiv · show

Large reasoning models (LRMs) like OpenAI-o1 and DeepSeek-R1 have demonstrated remarkable capabilities in complex reasoning tasks through the utilization of long Chain-of-thought (CoT). However, these models often suffer from hallucinations and inefficiencies due to their reliance solely on internal reasoning processes. In this paper, we introduce START (Self-Taught Reasoner with Tools), a novel tool-integrated long CoT reasoning LLM that significantly enhances reasoning capabilities by leveraging external tools. Through code execution, START is capable of performing complex computations, self-checking, exploring diverse methods, and self-debugging, thereby addressing the limitations of LRMs. The core innovation of START lies in its self-learning framework, which comprises two key techniques: 1) Hint-infer: We demonstrate that inserting artificially designed hints (e.g., ``Wait, maybe using Python here is a good idea.'') during the inference process of a LRM effectively stimulates its ability to utilize external tools without the need for any demonstration data. Hint-infer can also serve as a simple and effective sequential test-time scaling method; 2) Hint Rejection Sampling Fine-Tuning (Hint-RFT): Hint-RFT combines Hint-infer and RFT by scoring, filtering, and modifying the reasoning trajectories with tool invocation generated by a LRM via Hint-infer, followed by fine-tuning the LRM. Through this framework, we have fine-tuned the QwQ-32B model to achieve START. On PhD-level science QA (GPQA), competition-level math benchmarks (AMC23, AIME24, AIME25), and the competition-level code benchmark (LiveCodeBench), START achieves accuracy rates of 63.6%, 95.0%, 66.7%, 47.1%, and 47.3%, respectively. It significantly outperforms the base QwQ-32B and achieves performance comparable to the state-of-the-art open-weight model R1-Distill-Qwen-32B and the proprietary model o1-Preview.

1 Introduction

START addresses hallucinations and limited tool use in long-CoT reasoning by combining hint-guided inference with self-training for Python-interpreter use. The resulting model integrates long CoT with external execution and is presented as an open-source tool-integrated reasoner.

Long-CoT models can hallucinate on complex computations because they rely exclusively on internal reasoning, motivating tool-integrated reasoning.
Hint-infer inserts context-aware hints during inference to stimulate large reasoning models to invoke a Python interpreter without demonstration data.
Hint-infer can also provide sequential test-time scaling: adding hints before the stop token increases thinking time and problem-solving success.
Hint-RFT scores, filters, and modifies hint-generated trajectories before fine-tuning QwQ-32B-Preview, while subsequent rejection sampling fine-tuning produces START.
START combines long CoT with code-interpreter execution for complex reasoning, enabling self-debugging where QwQ-32B-Preview can hallucinate and produce incorrect answers.
START is presented as the first open-source long-CoT reasoning model that utilizes both long CoT and a code interpreter for complex reasoning tasks.

2 Related Work

Long CoT extends chain-of-thought reasoning with reflection, verification, correction, and test-time scaling, but reasoning models still face hallucinations and computational inaccuracies. START combines these approaches with Python-based tools to improve mathematical and coding reasoning.

Long CoT adds reflection, verification, correction, and multi-path exploration to conventional chain-of-thought reasoning.
Long CoT also exhibits test-time scaling, with increased computational resources associated with improved reasoning performance.
Reasoning models still encounter hallucinations and computational inaccuracies that limit complex-task performance.
START synthesizes Python-based tools with long CoT to improve performance across mathematical and coding benchmarks.

3 Methodology

START is trained to use Python tools during long CoT through scenario-specific hints, active data selection, and rejection-sampling fine-tuning. Hint-infer inserts prompts at reasoning junctures to encourage tool use and continued reasoning.

The training data combines math and code problems, with decontamination applied to reduce potential test-data leakage.
The Hint-Library provides math hints for reflection, logical verification, and alternative methods, plus coding hints for self-debugging and test-case checking.
Hint-infer inserts hints after conjunctions such as “Alternatively” and “Wait,” then sends generated code to Python before reasoning continues.
Active learning retains problems solved with hint inference but not greedy inference, producing Dseed with 10K math and 2K code examples.
Rejection-sampling fine-tuning scores, filters, and manually modifies sampled long-TIR data before a second fine-tuning stage produces START.

4 Experiment

START is evaluated on graduate-level science, competition mathematics, and coding benchmarks against general LLMs and long-CoT models. It improves over QwQ-32B-Preview across reported science, math, and code comparisons, while tool effects vary by task and dataset.

Evaluation setup: The evaluation covers Python-tool integration on GPQA, MATH500, AMC23, AIME24, AIME25, and LiveCodeBench.
Task differences: Physics benefits more from Python tools, whereas Biology benefits more from internet knowledge according to the reported GPQA subject comparison.
Math benchmarks: 3.8%, 15.0%, 16.7%, and 7.1% absolute improvements over QwQ-32B-Preview occur on MATH500, AMC23, AIME24, and AIME25, respectively.
Ablations: QwQ-RFT remains nearly on par with QwQ-32B-Preview, suggesting START’s advantage is predominantly associated with tool invocation rather than expanded training data alone.
Test-time scaling: Hint-infer improves all reported QwQ-32B-Preview benchmarks modestly, while START gains more substantially through Hint-RFT.

5 Conclusion

The paper presents START as a tool-integrated long-CoT model trained through Hint-infer and Hint-RFT. Its reported benchmark improvements support combining external tools with long-CoT reasoning for advanced reasoning tasks.

START integrates external tools and self-learning techniques to mitigate limitations of existing large reasoning models.
Hint-infer and Hint-RFT enable long-CoT models to use coding interpreters for computation and self-debugging.
Empirical results report significant improvements across challenging benchmarks and position START as an open-source solution for advanced reasoning tasks.

6 Limitations

The paper identifies limitations involving its tool scope, manually designed hints, benchmark coverage, and potential misuse risks. It frames these constraints as directions for future improvement and responsible deployment.

Tool scope: START integrates only a Python interpreter, leaving broader toolsets such as search engines, specialized libraries, and other computational resources unexplored.The authors suggest diverse tools could improve performance, versatility, and robustness across domains.
Hint design: Manually designed hints may disrupt the model’s original reasoning flow, while hint effectiveness can vary by task or context.The authors suggest more nuanced criteria for hint types and placement could improve reasoning fluidity and accuracy.
Evaluation scope: The empirical evaluation uses a limited set of benchmarks, so generalizability across broader and more diverse datasets remains unestablished.Performance may vary with task complexity, domain specificity, and input-data characteristics.
Responsible use: The model’s code-generation and problem-solving capabilities could be misused for disinformation or automating harmful tasks.The paper calls for safeguards and ethical guidelines to monitor and mitigate these risks.
Conclusion: The authors present acknowledging these limitations as important for future improvements and responsible development of tool-integrated reasoning models.This conclusion connects the identified constraints to subsequent research and deployment practices.

A.2 More results about Hint-infer

Hint-infer generally improves QwQ-32B-Preview across repeated hint-insertion rounds, but START-Hint-infer shows dataset-dependent gains, including modest or absent improvements on some tasks.

QwQ-Hint-infer: 60.0% on AIME24 is reached by QwQ-32B-Preview after three hint-insertion rounds, up from 50.0% at Round 0.The passage also reports improvements across aime25, gpqa, amc23, MATH500, and LiveCodeBench.
START-Hint-infer: 60.0% Pass@1 is reached on AIME25 by START-Hint-infer at Round 3.LiveCodeBench increases from 47.3% to 50.0%, while other datasets show more modest or absent gains.
Cross-dataset variation: Hint-infer’s effectiveness varies with dataset characteristics, producing improvements on some tasks but limited or no gains on others.The reported disparity indicates that repeated hint insertion does not yield uniform performance changes across benchmarks.

A.3 Prompting Methods for Data annotation

The prompting study compares direct, well-designed, and in-context approaches for eliciting Python tool calls, finding that hint-infer succeeds where the prompt-based methods do not.

Prompting methods: Three prompt-based methods were investigated to trigger long CoT reasoning with Python tool calls: direct, well-designed, and in-context prompting.The direct prompt explicitly instructs models to integrate natural-language reasoning with programs, while the well-designed prompt provides detailed tool-use instructions.
Evaluation: The evaluation compares QWQ-Hint-infer and START-Hint-infer on PhD-level science QA, mathematics, and coding benchmarks using Pass@1.The table caption identifies the evaluated task categories and metric.
Results: 100% success was achieved by hint-infer in triggering Python code generation, whereas QwQ-32B-Preview, DeepSeek-R1, and o1-mini failed with the three prompt-based methods.The comparison concerns generating long CoTs containing code-interpreter calls.

A.4 Hint-infer for test time scaling

The paper applies Hint-infer for test-time scaling by inserting progressively varied Python-use hints and using code templates to test candidate solutions on coding tasks.

Hint rounds: Hint-infer uses three rounds of Python-oriented hints that prompt checking, repeated utilization, and deeper problem analysis.The listed hints are used for GPQA and MATH, with the same Hint-Library approach applied to LiveCodeBench.
Starter-code tasks: Coding tasks with starter code use a template that instantiates the solution, runs example inputs, prints outputs, and checks the results.The template includes placeholders for the starter code, test inputs, and expected output inspection.
No-starter-code tasks: Coding tasks without starter code use a function template followed by example-input execution and output checking.The template supplies a function definition before testing the candidate implementation.

A.5.1 A case form AMC23

This appendix traces tool-assisted solutions across AMC23, AIME24, AIME25, MATH500, GPQA, and LiveCodeBench, including successful computations, verification, and debugging.

A.5.1 A case form AMC23: The AMC23 reasoning considers enumeration, modular arithmetic, Frobenius numbers, and non-negative combinations of 6, 10, and 15.The passage notes that no pair of denominations is coprime, so the two-coin Frobenius formula does not directly apply.
A.5.1 A case form AMC23: The appendix includes code fragments for testing coin combinations, checking divisibility and primality, and executing candidate solutions.Several fragments are incomplete or contain a reported SyntaxError, so the surrounding reasoning supplies the substantive results.
A.5.1 A case form AMC23: 11 is the reported AMC23 answer: 29 cents is the largest non-purchasable value, and its digit sum is 11.The solution uses code to search values below an arbitrary upper limit.
A.5.2 A case form AIME24: The AIME24 solution rejects p = 2 because n^4 + 1 is never divisible by 4, then considers odd primes with p ≡ 1 (mod 8).It connects the condition to the order of n modulo p^2 and Euler’s totient function.
A.5.2 A case form AIME24: AIME24 reports p = 17 and m = 110 for the least prime and least positive solution to m^4 + 1 divisible by p^2.The reported verification uses 17^2 = 289 and checks the congruence for 110.
A.5.3 A case form AIME25: AIME25 reports a final value of a + b = 510, while the MATH500 case reports fixed points n = 1 and n = −2.These are presented as final answers for their respective case studies.
A.5.5 A case form GPQA: The GPQA case first identifies a calculation mistake, then reports answer choice B, approximately 4.5, after recalculating the temperature-related factor.The passage explicitly contrasts the corrected result with an earlier approximate choice of 1.1.
A.5.6 A case form LiveCodeBench: The LiveCodeBench case detects incorrect example outputs, revises the implementation, and then reports that the revised code produces the correct examples.The initial outputs were 2 and 6 instead of the expected 4 and 126.