CoRT: Code-integrated Reasoning within Thinking

1 Introduction

• LRMs struggle with accuracy and efficiency on complex mathematical operations, while Code Interpreters provide tools suited to precise computation and equation solving.Integrating them is challenging because CIs introduce external deterministic knowledge beyond models’ internal representations.Source: §1 · pp. 1–2
• A prompt-hint after <think> raised code triggering from 50% to 90% on 100 problems, but models still used tools inefficiently.The hint asks models to solve step by step using multiple Python calls.Source: §1 · p. 2
• Hint-Engineering inserts targeted hints to redirect delayed manual calculations to code and discourage unnecessary checking of CI outputs.It aims to help models incorporate external knowledge into their reasoning.Source: §1 · p. 2
• 4% and 8% absolute accuracy gains for DeepSeek-R1-Distill-Qwen-32B and -1.5B accompany 30% and 50% fewer tokens on AIME benchmarks, respectively.The accuracy gains are reported across five challenging mathematical reasoning datasets.Source: §1 · pp. 2–3
• The paper presents data synthesis, post-training, and evaluations of code-integrated reasoning across five challenging mathematical benchmarks as its contributions.The introduction frames data scarcity as a challenge for this emerging domain.Source: §1 · p. 3

2 Methodology

• CoRT’s training framework covers code-integrated reasoning, a Cold Start–SFT–RFT pipeline for 32B models, and strong-to-weak distillation plus RL for 1.5B models.Source: §2 · p. 3

2.1 Task Formulation

• At each step, the model uses the problem and prior interaction trace to generate text and code; an executor returns output that joins the trace for subsequent reasoning.The trace contains textual reasoning steps, program snippets, and execution outputs, and continues until a definitive answer.Source: §2.1 · p. 3

2.2.1 Prompt-hint

• Prompt-hint places a general instruction after <think> to encourage code use, increasing the likelihood of code-integrated reasoning.The method generated 800 training instances from STILL3 and used them for SFT of DeepSeek-R1-Distill-Qwen-32B.Source: §2.2.1 · pp. 3–4

2.2.2 Hint-engineering

• Hint-Engineering places targeted hints at delayed calculations and distrust of execution results, redirecting work to Python while retaining checks of code logic.The authors manually inserted hints in 30 AIME problems; RFT then filtered sampled trajectories for incorrect answers and inefficient code behaviors.Source: §2.2.2 · pp. 4–5

2.3 Strong-to-weak Distillation

• Under infrastructure constraints, the authors distilled 32B models into 1.5B models and selected and preprocessed 10,000 public examples for distillation.Source: §2.3 · p. 5

2.4 Code-integrated Reinforcement Learning

• Code-integrated RL uses GRPO rollouts with CI interactions, a persistent execution environment, output masking, and accuracy plus code-execution rewards.A tool-use limit controls rollout overhead; the reward penalizes responses whose code attempts all fail.Source: §2.4 · p. 5

3.1 Main Results

• At 32B, Hint-Engineering-RFT scores 81.3% average accuracy versus Prompt-Hint-SFT’s 81.8%; at 1.5B, Hint-Engineering-RL reaches 56.4%, while Prompt-Hint-RL reaches 58.3%.The 1.5B RL models improve over their SFT stages by approximately 8% absolute.Source: §3.1 · p. 6

3.2 Token Efficiency Analysis

• At comparable accuracy, Hint-Engineering uses 40–50% fewer tokens than Prompt-Hint-SFT-32B on AIME24; it also cuts token use approximately 30% versus R1-distill-32B.Figure 4 also reports lower token consumption for both correct and incorrect responses.Source: Figure 4 · p. 7, §3.2 · p. 7

3.3 Code Behavior Analysis between Prompt-Hint and Hint-Engineering

• Prompt-Hint uses code mainly for verification (68.2%), versus Hint-Engineering’s near-even calculation and verification split (51.1% and 48.9%).Hint-Engineering also shows more diverse and more uniformly distributed mathematical code functions.Source: §3.3 · p. 8, Figure 5 · p. 8

3.4 Impact of Code Reward in RL

• On AIME24, code reward lifts Prompt-Hint accuracy to 43.1% from 37.9% and Hint-Engineering to 41.0% from 38.3%; a stronger reward ratio of 0.5 degrades performance.The reported optimal code reward ratio is ω = 0.1.Source: §3.4 · p. 8, Figure 6 · p. 9

3.5 Pass@K Analysis

• Across AIME24 and MATH500-Level5, SFT does not significantly raise the Pass@k upper bound, while RL raises performance across k, especially at lower sampling budgets.The analysis compares the DeepSeek-R1-1.5B base model with SFT and RL variants.Source: Figure 7 · p. 9, §3.5 · pp. 9–10

4 Conclusion

• The paper concludes that code tools can enhance mathematical reasoning across model scales, with Hint-Engineering reducing token use by 30–50% while maintaining competitive performance.It reports that RL improves performance beyond SFT, particularly for smaller models.Source: §4 · p. 10

A.1 Reasoning in LLMs

• The reviewed work traces large reasoning models to scaling, RL, and long-form CoT [23] [24] [25] [3], alongside growing interest in code interpreters for computation [44] [45] [11].Source: §A.1 · p. 16

A.2 Code-Integrated Reasoning for LLMs

• CoRT positions its approach against prior tool-integrated reasoning: START uses hints [20], STILL3 uses prompting [16], and Retool rewrites long CoT [61].The paper characterizes CoRT as sample-efficient, using 30 human-annotated examples with RFT [15] [62] and RL.Source: §A.2 · p. 16

B.1 Training Implementation

• Hint-Engineering SFT-32B uses 30 annotated samples versus 800 for Prompt-Hint SFT-32B; its RFT uses 800 filtered samples, while RL training selects 1,000 challenging problems.The RL selection comes from 20,000 randomly selected NuminaMath-1.5 problems.Source: §B.1 · pp. 16–17

B.2 Evaluation Methodology

• Evaluations re-run public models with original scripts, verify final answers against ground truth using Math-Verify, and use avg@16 or avg@4 as pass@1 depending on dataset.Inference uses a 32,768-token maximum sequence length, up to 15 function calls, temperature 0.6, and top-p 0.95.Source: §B.2 · p. 17

B.3 Function Call Limiting Strategy

• After 15 code executions, a system message directs the model to continue reasoning without code, limiting computational resources while allowing it to finish.Source: §B.3 · p. 17

B.4 Computational Hardware

• Training used four servers with eight A100 GPUs each; evaluation used a single server with eight A100 GPUs.Source: §B.4 · p. 18

C Evaluation Dataset Description

• The five evaluation sets are AIME24, AIME25, AMC23, MATH500, and OlympiadBench, spanning competition and olympiad-level mathematics.MATH500 contains 500 problems, and OlympiadBench includes 675 olympiad-level problems.Source: C Evaluation Dataset Description · p. 18

D.1 Code Behavior Evolution During RL

• During RL, Hint-Engineering’s code-use rate rises from 86% to above 95%, while its code blocks fall from 4.4 to around 3.5; Prompt-Hint’s rise from 1.9 to around 2.7 reflects a different trend.Prompt-Hint’s code success rate stays around 78%; Hint-Engineering varies more but reaches higher peaks.Source: Figure 8 · p. 18, §D.1 · p. 19

D.2 RL Training Data Ablation

• Increasing RL training data from 1K to 5K or 20K samples does not improve optimal performance under uniform difficulty and topic distributions.The authors suggest data quality may matter more than quantity for RL fine-tuning.Source: §D.2 · p. 19, Figure 9 · p. 19
• Hard-query training takes longer to reach its optimum but ultimately achieves the best results; easy queries peak earlier at lower accuracy.The comparison uses 1K samples across easy (avg@8=7/8), uniform, and hard (avg@8=1/8) difficulty distributions.Source: §D.2 · p. 19, Figure 9 · p. 19
• Uniform and hard-query-matched topic distributions produce minimal differences in optimal performance with 1K samples.The authors suggest difficulty level matters more than specific topic coverage.Source: §D.2 · pp. 19–20, Figure 9 · p. 19

D.3 Code Reward Ablation

• With penalty=0.1, code reward reaches 43.1% peak accuracy at step 100, versus 37.9% without code reward.The improvement persists through most of training.Source: §D.3 · p. 20, Figure 10 · p. 20
• Penalty=0.1 yields more stable performance than penalty=0.5, whose different-SFT-model run spikes to 41.4% at step 60 before declining sharply.The penalty=0.1 configuration converges more smoothly and maintains higher accuracy across training steps.Source: §D.3 · p. 20, Figure 10 · p. 20

D.4 Token Efficiency Analysis for Lightweight Models

• At a 4k token budget, Hint-Engineering-SFT achieves 22.7% accuracy, versus 12.1% for DeepSeek-R1-1.5B and 11.0% for Prompt-Hint-SFT.Hint-Engineering-SFT outperforms both approaches across the token budgets examined.Source: §D.4 · p. 20, Figure 11 · p. 21
• Hint-Engineering-RL uses 32% fewer total tokens than Prompt-Hint-RL (6,684 vs. 9,891), with comparable accuracy (41.0% vs. 43.1%).For correct responses, Hint-Engineering-SFT uses 47% fewer tokens than Prompt-Hint-SFT (4,711 vs. 8,862).Source: §D.4 · p. 21, Figure 11 · p. 21

D.5 Pass@K Analysis for Frontier Models

• At k=1 on AIME24, Prompt-Hint-32B-SFT reaches 77.3% accuracy versus 72.9% for DeepSeek-R1-32B; as k increases, models converge to 93.3% on AIME24 and 99.2% on MATH500-Level5.Hint-Engineering-32B-SFT also improves over the baseline at lower k values.Source: §D.5 · p. 21, Figure 12 · p. 22
• Hint-Engineering-32B-RFT reaches 93.3% accuracy on AIME24 with k=8 samples, while other approaches require 2–4 times more samples for comparable results.The analysis covers AIME24 and MATH500-Level5.Source: §D.5 · p. 21, Figure 12 · p. 22

E Prompts

• The solving prompt pairs step-by-step natural-language reasoning with Python execution, code blocks, displayed outputs, and multiple calls when useful.It directs Python use toward computation, symbolic manipulation, and data analysis.Source: E Prompts · pp. 22–23

1. Is it Verification or Calculation?

• The code-analysis prompt classifies snippets as Verification when they confirm an existing result and Calculation when they derive a new result, then assigns one or more functions.The prompt asks for the code index, classification, and function label.Source: §1 · p. 23

E Prompts

• The function menu spans equation solving, numerical approximation, optimization, combinatorial enumeration, and FFT-related tasks.It also includes data visualization and pattern analysis.Source: §1 · p. 23, §3 · p. 24, §6 · p. 24

F.1 5 Samples of DHint−engineering−SF T

• 797 is the least positive n for which 2^n + 5^n − n is divisible by 1000.The solution uses the Chinese Remainder Theorem, checks modular periods, and verifies the result modulo 1000.Source: §F.1 · pp. 26, 28
• 349 is the largest prime below 1000 satisfying the complex-number and triangle conditions, with (x,y)=(-5,-18), Re=4735, and Im=4482.The reported triangle inequalities all hold for these values.Source: §F.1 · p. 30
• 125 is the requested sum m+n for the two noncongruent parallelepipeds, whose reported volumes are 63√2/2 and 31√2.The solution gives m=63 and n=62.Source: §F.1 · pp. 30–31
• 75 is the reported value of tan^2(∠AEP) for the equilateral-triangle configuration.The calculation solves for P=(1525/38, 25√3/38) before evaluating the tangent.Source: §F.1 · pp. 31–32
• For the 13-14-15 triangle problem, the circumcircle has center (7,33/8) and radius 65/8, and the second intersection of AM with it is (567/74,−147/37).The supplied part then begins calculating Q from the condition ∠PBQ=∠PCQ but does not report the requested AQ value.Source: §F.1 · pp. 32–34
• The sample finds t = 99/148 or 197/148 and concludes m + n = 247 using m = 99 and n = 148.The reported integers m and n are coprime positive integers.Source: §F.1 · pp. 35–36

F.2 Test case of Prompt-Hint-SFT-32B

• The Prompt-Hint-SFT-32B examples conclude p + q = 721 for the box-radius problem and area(BGE) = 468 for the triangle problem.For the box, the reported fraction is r^2 = 657/64; the triangle result follows from the coordinates B(18,0), G(58,24), and E(57,0).Source: §F.2 · pp. 37–38

F.3 Test case of Hint-Engineering-RFT-32B

• The Hint-Engineering-RFT-32B examples conclude p + q = 721 for the box-radius problem and area(BGE) = 468 for the triangle problem.Numerical optimization reports a maximum a² + b² + c² of 41.0625 for the box; the triangle area is also verified as 468 using base and height.Source: §F.3 · pp. 40–41

G Stron-to-Weak Distillation Data

• Approximately 10,000 problems from AIME, MATH Level 5, and Numina-math make up the mathematical dataset for strong-to-weak distillation.It includes 800 AIME problems, 2,280 MATH training problems, and 7,000 Numina-math problems.Source: G Stron-to-Weak Distillation Data · p. 41
• Numina-math problems with figures, proof-based problems, multiple-choice questions, or answers in their statements were excluded before sampling.The remaining problems were randomly sampled to create a clean, standardized dataset for distillation training.Source: G Stron-to-Weak Distillation Data · p. 41
• The dataset draws on AIME competitions prior to 2024 and the MATH training set’s Level 5 problems.The sources contribute 800 AIME problems and 2,280 MATH problems, respectively.Source: G Stron-to-Weak Distillation Data · p. 41

H Baselines

• Frontier comparisons include o1, QwQ-32B, and DeepSeek-R1, which the paper treats as an upper bound for performance.At 32B, comparisons also include DeepSeek-R1-32B, START-32B [20], STILL-3-TOOL-32B [16], and ReTool-R1-32B [61].Source: H Baselines · p. 41
• At 1.5B, baselines include DeepSeek-R1-1.5B, DeepScaleR-1.5B-Preview, and ToRL-1.5B [59].DeepScaleR-1.5B-Preview is described as the current state-of-the-art reinforcement learning model at this scale.Source: H Baselines · p. 41
• The baseline set spans model sizes, SFT, RL and RFT training, and mathematical reasoning with and without explicit tool use.The authors use this comparison to evaluate Prompt-Hint and Hint-Engineering against established benchmarks and recent approaches.Source: H Baselines · p. 42

I Broader Impacts

• Potential benefits include educational and research applications, lower token-related resource consumption, and more reliable AI calculations.The passage presents these as potential societal benefits of improved mathematical reasoning efficiency and accuracy.Source: I Broader Impacts · p. 42
• Enhanced reasoning could be misused; the authors recommend access controls, transparency about limitations and intended use, and responsible deployment strategies.The models are trained on public mathematical datasets and intended for educational and research purposes.Source: I Broader Impacts · p. 42

• Training and inference requirements for the 32B and 1.5B DeepSeek-R1-Distill-Qwen models may affect broader accessibility.Source: J Limitation · p. 42
• The rapid evolution of LRM capabilities creates opportunities to continue enhancing the work as new architectures emerge.Source: J Limitation · p. 42
• The work focuses on code interpreters in long-form reasoning, leaving additional synergies between external tools and models’ reasoning capabilities for future investigation.Source: J Limitation · p. 42