PaperlayerView exact revision on arXiv

Source-linked AI summary

Teaching Language Models to Think in Code

Hyeon Hwang, Jiwoo Lee, Jaewoo Kang

arXiv:2605.07237v2cs.CL

TL;DR

Precise multi-step mathematical reasoning remains challenging for natural-language reasoners because intermediate computations can contain errors. THINC makes code the primary reasoner after brief planning, and THINC-4B reaches 78.1% across five competition-level math benchmarks while outperforming all evaluated baselines.

  • Problem

    Natural-language reasoners remain vulnerable to errors in precise multi-step mathematical computation.

  • Method

    THINC uses one brief natural-language planning step, then performs all subsequent reasoning through code blocks linked by interpreter outputs.

  • Results

    78.1% average accuracy: THINC-4B outperforms all evaluated tool-integrated reasoning baselines across five competition-level math benchmarks.

  • Takeaways & Limitations

    99.2% of THINC-4B final answers come from interpreter output, and it maintains 64–69% recovery through three consecutive code failures.

  • Takeaways & Limitations

    Experiments are limited to 1.7B and 4B models and competition-level mathematics, leaving scalability and transfer to other tool-integrated domains untested.

Abstract

from arXiv · show

Tool-integrated reasoning (TIR) has emerged as a dominant paradigm for mathematical problem solving in language models, combining natural language (NL) reasoning with code execution. However, this interleaved setup has three key limitations: code often acts as a post-hoc verifier, intermediate NL computations are error-prone, and NL and code play overlapping rather than clearly distinct roles. We propose ThinC (Thinking in Code), a framework in which code itself serves as the reasoner rather than as a tool invoked by NL. A ThinC trajectory begins with a brief NL planning step, after which all reasoning unfolds through code blocks connected only by their execution outputs. We distill 12.2k code-centric trajectories from a teacher model and train ThinC-1.7B and ThinC-4B with supervised fine-tuning followed by reinforcement learning. ThinC-4B consistently outperforms every TIR baseline on five competition-level math benchmarks and even surpasses the much larger Qwen3-235B-A22B-Thinking. Further analysis shows that ThinC reasons through code: 99.2% of its final answers are grounded in interpreter output, and the model recovers reliably from code execution failures without intermediate NL reasoning. Our code and models will be released soon.

1 Introduction

THINC makes code the primary reasoner after a brief NL plan, addressing structural weaknesses of interleaved tool-integrated reasoning. THINC-4B reaches 78.1% average accuracy across five competition-level math benchmarks, surpassing every evaluated TIR baseline and outperforming a much larger NL reasoner on four benchmarks.

  • Related work: Prior approaches such as PAL and PoT [2] established reliable executable computation but generally generated single-pass programs without iterative interaction with execution results.Later systems including ReTool [4], ASTER, and Tool-Star [3] expanded tool interaction, reinforcement-learning optimization, or multi-tool collaboration.
  • Motivation: Interleaved tool-integrated reasoning often uses code for post-hoc verification, propagates NL arithmetic errors as hard-coded constants, and duplicates algorithms across NL and code.These three limitations are summarized in Figure 1 and described as recurring structural problems of the paradigm.
  • THINC framework: THINC begins with one brief NL planning step, then conducts all reasoning through code blocks connected only by execution outputs.The framework treats code as the reasoner rather than as a tool driven by NL reasoning, and combines trajectory distillation, supervised fine-tuning, and reinforcement learning with verifiable rewards.
  • Evaluation: 78.1% average accuracy across five competition-level math benchmarks lets THINC-4B surpass every evaluated TIR baseline and Qwen3-235B-A22B-Thinking on four of five benchmarks.THINC-1.7B reaches 42.8% average accuracy, exceeding Qwen3-1.7B by 10.6 percentage points.

2 Preliminaries

This section defines tool-integrated reasoning (TIR) as alternating language-model text generation and external-tool execution, focusing on Python-based mathematical reasoning. It also summarizes the SFT and verifiable-reward RL procedures used to train TIR systems.

  • 2 Preliminaries: TIR alternates natural-language thought blocks and Python-interpreter outputs across multiple turns before producing a final answer.The trajectory is represented as τ_TIR = (q, t_1, c_1, o_1, …, t_N, c_N, o_N, a), a structure shared by recent TIR systems [6] [4].
  • 2 Preliminaries: SFT trains on teacher-distilled demonstration trajectories using next-token prediction, with trajectory selection shaping the learned tool-use behavior [4].This work applies the loss mask m_k = 1 to all tokens after finding no significant performance difference from masking tool-output tokens.
  • 2 Preliminaries: RL uses exact-match verifiable rewards, assigning r(τ) = 1 when the extracted answer equals the known groundtruth and avoiding a learned reward model.The reward is r(τ) = 1[a(τ) = a⋆(q)].
  • 2 Preliminaries: GRPO [16] trains without a critic by sampling groups of trajectories for each problem and computing advantages relative to rewards within each group.The sampled trajectories come from the current policy π_θ.
  • 2 Preliminaries: Following DAPO, the RL objective normalizes tokens across the group and uses asymmetric clipping with ε_low < ε_high to permit larger positive policy updates.The objective uses per-token importance ratios between the current and old policies.

3 THINC: Teaching Models to Think in Code

THINC teaches models to use code as the mathematical reasoner, with natural language limited to high-level planning and code blocks linked through execution outputs. It trains this format through teacher distillation, supervised fine-tuning, and multi-stage reinforcement learning.

  • 3.1 Code-centric trajectories: Code serves as the reasoner because programming-language variables, operations, and functions correspond directly to mathematical objects and can be executed precisely.The interpreter produces verified intermediate values, eliminating unverified numerical computation in NL.
  • 3.1 Code-centric trajectories: THINC replaces interleaved TIR with code-centric reasoning: one initial NL strategy is followed by code blocks that build on preceding execution outputs.Code directly expresses and executes derivation steps, while NL and code have specialized roles.
  • 3.2 Distillation and supervised fine-tuning: Teacher-model filtering produces THINC-SFT, a dataset of 12,200 correct, executable trajectories containing at least three code blocks.Problems are drawn from Skywork-OR1 [8] and OpenMathReasoning [11], and trajectories are sampled from Qwen3.5-27B.
  • 3.2 Distillation and supervised fine-tuning: THINC fine-tunes Qwen3-1.7B and Qwen3-4B-Thinking-2507 on THINC-SFT, yielding THINC-1.7B-SFT and THINC-4B-SFT.The models are trained for three epochs with a 32K-token context length.
  • 3.3 Reinforcement learning: Starting from the SFT checkpoints, multi-stage GRPO training on DAPO-Math-17k produces the final THINC-1.7B and THINC-4B checkpoints.Training uses increasing context budgets, reaching 32K tokens and up to 40 tool calls for harder problems.

4 Experiments

Across five competition-level mathematics benchmarks, THINC-4B achieves the strongest overall performance, outperforming larger tool-integrated and NL-only reasoners. Additional analyses show that its gains reflect code-centric reasoning, reinforcement learning, and robustness to execution failures.

  • Main results: THINC-4B averages 78.1%, leads on four of five benchmarks, outperforms every tool-integrated baseline, and exceeds Qwen3-235B-A22B-Thinking by 2.9 points.The evaluation covers AIME 2024, AIME 2025, AIME 2026, HMMT 2025 February, and BeyondAIME.
  • Reasoning format: Under matched conditions, THINC-4B exceeds the interleaved ASTER-4B baseline on every benchmark by 4.1 average points while requiring fewer tool calls.The comparison shares the base model, teacher capacity, and reinforcement-learning pipeline, isolating trajectory structure.
  • Training dynamics: Reinforcement learning adds 29.9 points at 4B and 24.6 points at 1.7B after supervised fine-tuning establishes the THINC format.THINC-4B-SFT averages 48.1%, while THINC-1.7B reaches 18.1%; both improve substantially with reinforcement learning.
  • Code-centric reasoning: THINC-4B uses 349 code lines per sample versus 102 for ASTER, 40 for CoRT, and 261 for ReTool, demonstrating substantially heavier code use.The comparison spans overall benchmark trajectories and includes baselines explicitly designed to strengthen tool use.
  • Code-centric reasoning: Final answers appear in execution output for 99.2% of THINC-4B trajectories, compared with 88.4% for ReTool and 74.3% for rStar2.This indicates that THINC’s answers are more directly grounded in interpreter execution than those of the compared baselines.
  • Failure recovery: Under initial execution failures, THINC-4B remains in a 64–69% Recovery@k band through k = 3 and reaches 33.3% at k = 5, nearly twice any interleaved baseline.ASTER falls from 52.1% at k = 1 to 18.5% at k = 5, while rStar2-Agent falls from 39.1% to 0%.

5 Discussion and Conclusion … A.3 Stage 2 — Code-Centric Reasoning

THINC makes code the primary reasoning medium after brief natural-language planning, achieving strong competition-math performance while grounding answers in interpreter outputs and recovering from code failures. A rollout illustrates this design through code-only correction, verification, and re-derivation, but evaluation remains limited to small models and competition mathematics.

  • 5 Discussion and Conclusion: 78.1% on five competition-level math benchmarks places THINC-4B above all baselines.The reported gains are associated with 99.2% of final answers coming from interpreter output and 64–69% recovery through three consecutive code failures.
  • 5 Discussion and Conclusion: THINC’s evaluation covers only 1.7B and 4B models and competition-level mathematics, leaving larger-scale and cross-domain applicability unresolved.The paper identifies scaling to larger models and extension to other tool-integrated reasoning domains as future directions.
  • A Case Study: A THINC-4B Trajectory on AIME 2026 Problem 3: A THINC-4B AIME 2026 Problem 3 rollout begins with one brief strategic plan, then conditions each code step on the previous interpreter output until execution yields the answer.The trajectory contains no NL-level computation after planning; code performs the reasoning and produces the final answer.
  • A.1 Problem: The case-study problem asks for the number of integers ≤100 representable as a+b+ab using distinct positive integers a and b, with reference answer 70.This establishes the target solved by the subsequent code-centric rollout.
  • A.2 Stage 1 — Strategic Planning (t1): The initial NL planning factors a+b+ab as (a+1)(b+1)−1 and specifies the constrained search, without performing arithmetic or enumerating cases.The plan fixes m,n′ ≥2, m≠n′, and m·n′ ≤101 before making a single transition to code.
  • A.3 Stage 2 — Code-Centric Reasoning: Five code/result exchanges perform enumeration, self-correction, structural checking, independent re-derivation, and complement auditing, with no NL reasoning between blocks.The code-only process corrects the loop bound using monotonicity, verifies the count by parity, and re-derives it with shifted variables.

A.4 What This Trajectory Illustrates

The rollout illustrates THINC’s design: natural language states strategy without arithmetic, while all subsequent reasoning proceeds through code conditioned on execution outputs. The final answer is code-grounded, with 70 appearing in interpreter outputs rather than being generated in natural language.

  • Strategy-only t1: The initial think channel expresses one algebraic insight and search constraints without performing arithmetic in NL, so it provides strategy rather than derivation.This realizes the THINC constraint that the first planning step expresses strategy only.
  • Code as the reasoner: Every reasoning step after the initial plan—including self-correction, structural verification, re-derivation, and complement auditing—occurs in code, with no intervening NL reasoning.NL appears only in inline code comments, while subsequent turns are connected through execution outputs.
  • Conditioning on execution outputs: Each subsequent turn conditions on the preceding execution output, as illustrated by Turn 2’s repair and formalized in Eq. 6.The rollout therefore instantiates the trajectory structure in which code outputs drive later reasoning.
  • Code-grounded final answer: 70 appears in interpreter outputs from Turns 2, 4, and 5, making the committed answer code-grounded rather than NL-generated.This rollout qualitatively reflects THINC-4B’s gap over interleaved TIR baselines and its 99.2% code-grounded answer rate.

B Few-Shot Prompt for Trajectory Distillation

The appendix specifies the 3-shot prompt used to elicit THINC trajectories from Qwen3.5-27B, requiring mathematical reasoning to proceed through Python execution. Its examples demonstrate code-based derivations, verification, recovery across fresh processes, and final boxed answers.

  • B.1 Few-Shot Prompt for Trajectory Distillation: The 3-shot prompt elicits THINC trajectories from the Qwen3.5-27B teacher model.The appendix presents the full prompt used for trajectory distillation.
  • B.1 Few-Shot Prompt for Trajectory Distillation: The solver must work exclusively through Python execution, placing code in <python> tags and receiving outputs in <result> tags.Final responses contain only Python blocks and one <answer> block, with concise reasoning embedded as code comments.
  • B.1 Few-Shot Prompt for Trajectory Distillation: Each code block runs in a fresh process, so the solver must re-import libraries and redefine or hardcode values from prior outputs.Variables do not persist between turns, making explicit state reconstruction part of the prompting protocol.
  • B.1 Few-Shot Prompt for Trajectory Distillation: The grid example derives k=12 by combining row bounds, block constructions, and exhaustive feasibility checks under the 104-cell color limit.A 10x10 block construction passes all 11-windows with 100 colors and at most 100 cells per color, while 11x11 exceeds 104.
  • B.1 Few-Shot Prompt for Trajectory Distillation: The geometric-area example computes the symmetric-difference region by evaluating two areas and their intersection, yielding the final answer 24.The code reports area_A = 24, area_B = 24, intersection area = 12, and area_K = 24.
  • B.1 Few-Shot Prompt for Trajectory Distillation: The triangle example uses symbolic coordinates, perpendicularity, and the shoelace formula to compute the area of triangle PQR.The trajectory first obtains t = 2/3 and R = (5a/6, sqrt(3)a/6), then simplifies the area symbolically.

C Tool Call and Response Length per Benchmark

This section reports average tool-call counts and response lengths per trajectory across each evaluation benchmark.

  • C Tool Call and Response Length per Benchmark: The section measures the average number of tool calls per trajectory.
  • C Tool Call and Response Length per Benchmark: It also reports average response length for each evaluation benchmark.
  • C Tool Call and Response Length per Benchmark: The analysis presents both metrics separately for every evaluation benchmark.

C.1 Tool Calls by Benchmark · C.2 Response Length by Benchmark · D OOD Generalization

THINC-4B generalizes to GPQA-Diamond, leading both reported accuracy metrics over ASTER-4B and the base model. The analysis also compares interpreter-call frequency and trajectory length across four mathematical benchmarks.

  • D OOD Generalization: THINC-4B leads GPQA-Diamond on avg@16 and best@16 accuracy, exceeding ASTER-4B by 3.1 points on avg@16 and the base model by 7.6 points on best@16.GPQA-Diamond serves as an OOD test spanning graduate-level physics, chemistry, and biology; Table 2 reports the accuracy metrics.
  • D OOD Generalization: The GPQA-Diamond evaluation tests whether THINC-4B’s code-centric reasoning format generalizes beyond the mathematical training domain.The supplied results describe this generalization as more effective than the interleaved alternative.
  • C.1 Tool Calls by Benchmark: Figure 6 compares average Python-interpreter tool calls across AIME 2024–2026, HMMT 2025 February, and BeyondAIME.The comparison measures how often models invoke the interpreter on each benchmark.
  • C.1 Tool Calls by Benchmark: The tool-call analysis covers AIME 2024–2026, HMMT 2025 February, and BeyondAIME as separate benchmark settings.Figure 6 reports average interpreter invocations for these benchmarks.
  • C.2 Response Length by Benchmark: Figure 7 reports mean trajectory length across AIME 2024–2026, HMMT 2025 February, and BeyondAIME.The figure compares response length by benchmark.
  • C.2 Response Length by Benchmark: The response-length analysis covers the same four mathematical benchmark settings and reports their mean trajectory lengths.Figure 7 presents the benchmark-level comparison.

E Training Details · E.1 Supervised Fine-Tuning · E.2 Reinforcement Learning

Training uses a shared single-node setup for both ThinC model sizes, with supervised fine-tuning followed by reinforcement learning. The SFT and RL configurations are documented in Tables 3 and 4, while RL follows several DAPO-inspired optimization choices.

  • E Training Details: Both THINC-1.7B and THINC-4B are trained on a single node with 8× NVIDIA H200 GPUs using identical hyperparameter configurations at each stage.SFT uses LLaMA-Factory [24], while RL uses verl [17].
  • E.1 Supervised Fine-Tuning: The supervised fine-tuning stage uses the LLaMA-Factory framework [24].
  • E.2 Reinforcement Learning: The reinforcement learning stage uses the verl framework [17].
  • E.2 Reinforcement Learning: RL applies token-level loss normalization across rollout groups, asymmetric clipping with ϵlow = 0.20 and ϵhigh = 0.28, and no KL penalty.These choices follow DAPO.
  • E.1 Supervised Fine-Tuning: SFT hyperparameters for THINC-1.7B-SFT and THINC-4B-SFT are reported in Table 3.
  • E.2 Reinforcement Learning: Table 4 reports the full RL hyperparameter configuration and three-stage curriculum for THINC-1.7B and THINC-4B.
Loading 2605.07237v2…