Source-linked AI summary
Scaling Code-Assisted Chain-of-Thoughts and Instructions for Model Reasoning
Honglin Lin, Qizhi Pei, Xin Gao, Zhuoshi Pan, Yu Li, Juntao Li, Conghui He, Lijun Wu
TL;DR
Natural-language reasoning traces are difficult to verify and scale, and existing code-assisted methods have limited generalization beyond predefined math problems. Caco generates code-based reasoning traces, filters them, and converts them into language instructions and CoTs; Caco-trained models outperform strong baselines on mathematical and out-of-domain benchmarks.
Problem
Natural-language CoTs are not executable and are costly to annotate at scale, while code-assisted methods remain constrained in generalizing beyond predefined mathematical problems.
Source: §1 · p. 2Method
Caco fine-tunes a code-based CoT generator, scales executable reasoning traces, filters them, and converts validated outputs into natural-language instructions and CoTs.
Source: §1 · p. 2, §3.2 · p. 6Results
Caco-trained models outperform strong baselines on both mathematical reasoning and out-of-domain benchmarks.
Source: §6 · p. 12Takeaways & Limitations
The authors highlight code-driven verification and instruction diversity as effective for improving reasoning generalization.
Source: §6 · p. 12Converting executable code back into natural-language instructions can lose details or require simplification, making outputs less rich or specific.
Source: E Limitations · p. 19
Abstract
from arXiv · showhide
Reasoning capability is pivotal for Large Language Models (LLMs) to solve complex tasks, yet achieving reliable and scalable reasoning remains challenging. While Chain-of-Thought (CoT) prompting has become a mainstream approach, existing methods often suffer from uncontrolled generation, insufficient quality, and limited diversity in reasoning paths. Recent efforts leverage code to enhance CoT by grounding reasoning in executable steps, but such methods are typically constrained to predefined mathematical problems, hindering scalability and generalizability. In this work, we propose Caco (Code-Assisted Chain-of-ThOught), a novel framework that automates the synthesis of high-quality, verifiable, and diverse instruction-CoT reasoning data through code-driven augmentation. Unlike prior work, Caco first fine-tunes a code-based CoT generator on existing math and programming solutions in a unified code format, then scales the data generation to a large amount of diverse reasoning traces. Crucially, we introduce automated validation via code execution and rule-based filtering to ensure logical correctness and structural diversity, followed by reverse-engineering filtered outputs into natural language instructions and language CoTs to enrich task adaptability. This closed-loop process enables fully automated, scalable synthesis of reasoning data with guaranteed executability. Experiments on our created Caco-1.3M dataset demonstrate that Caco-trained models achieve strong competitive performance on mathematical reasoning benchmarks, outperforming existing strong baselines. Further analysis reveals that Caco's code-anchored verification and instruction diversity contribute to superior generalization across unseen tasks. Our work establishes a paradigm for building self-sustaining, trustworthy reasoning systems without human intervention.
1 Introduction
Natural-language CoT can be unverifiable and costly to scale, while existing code-assisted approaches struggle beyond predefined math problems. Caco generates and validates code-anchored reasoning data before converting it into language instructions and CoTs; Caco-trained models perform strongly on mathematical benchmarks.
1 Introduction
- Natural-language CoT is not executable and can propagate intermediate errors, while manual annotation makes high-quality reasoning data difficult to scale.Source: §1 · p. 2
- Existing code-assisted methods enable execution-based verification but remain difficult to generalize beyond predefined mathematical problems.Source: §1 · p. 2
- Caco fine-tunes a code-reasoning generator, filters generated traces through execution and consistency checks, then converts validated code into instructions and language CoTs.The authors describe the resulting data as scalable, verifiable, and diverse.Source: §1 · p. 2
- 92.6% GSM8K and 82.4% MATH accuracy; Caco averages 67.7% across multiple benchmarks, surpassing comparable methods by a margin exceeding 7.9%.The paper reports these results for models fine-tuned using Caco-1.3M.Source: §1 · p. 2
2 Related Work
Prior mathematical-reasoning data work uses several augmentation and curation strategies, while code-integrated reasoning methods use executable steps for verification. Caco positions code generation as a way to scale diverse mathematical and algorithmic problems.
2.1 Data Augmentation for Mathematical Reasoning
- Math instruction datasets refine seed problems, generate new problems, curate web data, or filter solutions by difficulty.Examples include WizardMath, MetaMath, Orca-Math, MMIQC, MathFusion, KPMath, MathScale, ScaleQuest, MAmooTH2, Numina-Math, and DART-Math.Source: §2.1 · p. 3
2.2 Code Integration for Enhanced Reasoning
- Code-integrated methods use executable reasoning for computation and verification; Caco uses code generation for scalable, diverse coverage that also includes algorithmic problems.The cited approaches include PoT, PAL [9], CSV, OpenMathInstruct-1 [39], TORA, MathCoder [24], MegaMath [51], DotaMath [17], MAmmoTH, rStar-Math [11], CodeI/O [19], and MathGenie [23].Source: §2.2 · p. 3
3 Method
Caco unifies mathematical and algorithmic solutions as executable code, scales code reasoning patterns with CodeGen, and reverses generated code into language instructions and CoTs. Execution and consistency checks filter the resulting data.
3 Method
- Caco trains CodeGen on executable solution templates, samples code patterns, back-translates them into problems and solutions, and retains answer-matching instances.Source: §3 · p. 3, Figure 2 · p. 4
3.1 Unifying Code CoT
- Math and algorithmic solutions are standardized as executable Python templates, then filtered into 146K seed traces: 122K Math and 24K Code.The math templates support arithmetic, algebraic, geometric, and probabilistic reasoning; the algorithmic set includes sorting, searching, and dynamic programming.Source: §3.1 · pp. 4–5
3.2 Scaling Code CoT with CodeGen
- CodeGen is fine-tuned on seed Code CoTs and temperature-sampled to produce diverse reasoning paths; filtering retains about 4.6M of 5.3M generated programs.Source: §3.2 · pp. 5–6
- CodeGen supports problem-level augmentation through varied question formulations and pattern-level augmentation through novel reasoning structures.The paper illustrates a changed decomposition for a problem type seen during training.Source: §3.2 · p. 6
3.3 Instruction Reversal and Language CoT Generation
- A two-stage problem-then-CoT pipeline retains 1.3M pairs only after answer/code agreement and language/code reasoning consistency checks.The two stages generate a natural-language problem from code and then generate its language CoT.Source: §3.3 · pp. 6–7
4 Experiment
Experiments compare Caco-generated data with synthesized-dataset and model baselines across three base models and six mathematical benchmarks. Caco improves over prior methods, with gains as data scales and on challenging benchmarks.
4.1 Experimental Setup
- Evaluation fine-tunes three base models and measures zero-shot Pass@1 accuracy across six benchmarks using greedy decoding without tool integration.The comparison includes synthesized math instruction-tuning datasets and open-source instruction-tuned or RL-based models; the reported training configuration uses three epochs.Source: §4.1 · p. 7
4.2 Main Results
- Caco consistently outperforms existing methods across all three base models; on LLaMA3-8B, Caco-1.3M achieves an average score of 57.3, surpassing DartMath’s previous best of 39.7 by a relative improvement of 44.3%.Source: §4.2 · p. 7
- On Qwen2.5-Math-7B, Caco-1.3M scores 67.7, exceeding Caco-596K by 1.1.Source: §4.2 · p. 9
- On LLaMA-8B, Caco-596K raises OlympiadBench from 17.2 to 34.1 and TheoremQA from 20.0 to 31.0 versus MathFusion.Source: §4.2 · p. 9
- On Qwen2.5-Math-7B, Caco-1.3M scores 67.7, comparable to Qwen2.5-Math-7B-Instruct’s 63.6.Caco-1.3M trained models also significantly surpass DeepSeekMath-7B-RL and LLaMA-8B-Instruct, according to the authors.Source: §4.2 · p. 9
- On LLaMA3-8B, Caco-1.3M scores 57.3 versus 46.7 for the Caco-Seed-109K data.The authors report consistent improvements from Caco-596K and Caco-1.3M over seed data.Source: §4.2 · p. 9
5 Analysis
Caco’s analyses examine dataset diversity, scaling, verification, and generalization beyond mathematics. The reported findings include broader problem coverage, improved downstream performance with verification and more data, and gains on science reasoning benchmarks.
5.1 Analysis on Data Diversity
- Caco-generated problems broadly cover the seed-data embedding distributions, add a distinct region, and span 12 clusters of mathematical and algorithmic topics.The analysis sampled 5K Caco problems and used sentence embeddings, t-SNE, and KMeans clustering.Source: §5.1 · pp. 9–10, Figure 4 · p. 10
5.2 Analysis on Data Scalability
- Performance rises as training data grows from 109K to 1.3M for DeepSeekMath-7B and LLaMA3-8B, with larger early gains for LLaMA3-8B.Qwen2.5-Math also improves as data increases, but less markedly, which the authors attribute to its stronger base capabilities.Source: §5.2 · p. 10
5.3 Ablation on Verification
- 21.8 average accuracy with verified data versus 20.8 without; on Olympiad, the verified model scores 18.7, 1.1 points higher.The verification experiment also reports 97K solvable problems versus 91K and 93K correct answers versus 88K.Source: §5.3 · p. 11
5.4 Generality Beyond Mathematics
- Caco-1.3M models improve on mathematics, code, science, logic, and general-knowledge reasoning benchmarks beyond mathematics.The evaluation includes AIME24, HumanEval+, ARC-c, BBH, KorBench, and AGIEval.Source: §5.4 · p. 11
- On MegaScience science-reasoning seeds, Caco expands 5.2K examples to 37K valid QA samples; the LLaMA average rises from 59.0 to 63.4 across three benchmarks.The reported average covers AGIEval, ARC-c, and MMLU-STEM.Source: §5.4 · pp. 11–12, Table 3 · p. 11
6 Conclusion
The conclusion presents Caco as a code-assisted framework for synthesizing diverse, verifiable CoT data and reports model gains on mathematical and out-of-domain benchmarks.
6 Conclusion
- Caco uses code execution and automated filtering to synthesize logically grounded instruction data at scale without human supervision.Source: §6 · p. 12
- Models trained with Caco outperform strong baselines on mathematical reasoning and out-of-domain benchmarks.Source: §6 · p. 12
- The conclusion identifies code-driven verification and instruction diversity as effective for improving reasoning generalization.Source: §6 · p. 12
A Data Generation
The appendix lists prompts used for key generation and evaluation stages, and names the models assigned to each pipeline stage.
A.1 Prompts
- The appendix provides prompts for Code CoT unification, CodeGen training and sampling, question back-translation, answer generation, and solvability, correctness, and consistency evaluations.Source: §A.1 · p. 16
A.2 Model Usage
- Qwen2.5-72B-Instruct generated unified Code CoTs, while Qwen2.5-Coder-7B was fine-tuned for unconditional CodeGen.Source: §A.2 · p. 16
- Qwen3-8B handled question back-translation and answer synthesis, and Qwen3-32B conducted the evaluations.Source: §A.2 · p. 16
A.3 Implementation Details
The appendix specifies sampling choices and filtering checks for Code CoTs, including execution, semantic-use, and answer-consistency criteria.
A.3 Implementation Details
- BigMath Code CoTs were filtered to retain samples with solve-rate annotations below 0.3.Source: §A.3 · p. 16
- CodeGen training used LlamaFactory; inference sampling used temperature 0.9 and a 1024-token maximum sequence length.Source: §A.3 · p. 16
- Problem and solution generation used temperature 0.7, TopP 0.8, TopK 20, MinP 0, and disabled thinking.Source: §A.3 · p. 16
A.4 Filtering Mechanism for Code CoTs
- Filtering requires executable code, a runtime below 10 seconds, and at least six non-comment lines.Source: §A.4 · p. 16
- AST checks require input-dictionary variables to be used, and samples with known ground-truth answers must match the expected output exactly.Source: §A.4 · p. 17
B.1 Training Setup
The appendix specifies model-training and evaluation settings, then describes six mathematical benchmarks used for evaluation.
B.1 Training Setup
- Training ran for 3 epochs with batch size 128, AdamW at 5 × 10−6 learning rate, cosine decay, and 4096-token cutoff.The setup also used 8 NVIDIA A100 GPUs, a 0.03 warm-up ratio, and 0.1 weight decay.Source: §B.1 · p. 17
B.2 Evaluation Setup
- Evaluation used zero-shot greedy decoding with a maximum generation length of 2048 tokens.The evaluation prompt is provided in Prompt 11.Source: §B.2 · p. 17
B.3 Evaluation Benchmarks
- The evaluation benchmarks span competition and school mathematics, college mathematics, Olympiad problems, mathematical reasoning tasks, and theorem-driven cross-disciplinary questions.The set comprises MATH, GSM8K, CollegeMath, DeepMind-Mathematics, OlympiadBench-Math, and TheoremQA; the OlympiadBench evaluation uses its English text-only subset.Source: §B.3 · p. 17
C Additional Experiments
Control experiments compare Caco with teacher distillation and STaR-style self-improvement, while a cost analysis reports the resources needed to produce Caco-1.3M.
C.1 Distinguishing Caco from Teacher Knowledge Transfer and STaR-style Self-Improvement
- 66.2 vs. 65.5 AVG for Qwen-7B: Caco-300K outperformed the teacher-distilled baseline at matched data size; scaling Caco to 1.3M yielded up to 67.7 AVG.The authors interpret this comparison as evidence that Caco's prompt and reasoning diversity provides benefits beyond direct teacher distillation.Source: Table 4 · p. 18, §C.1 · p. 18
- 66.2 vs. 52.6 AVG for Qwen-7B and 53.2 vs. 48.9 AVG for DS-7B: Caco-300K outperformed the 300K SEED-SELF-IMPROVE dataset on both backbones.Caco trains a dedicated code generator to create executable CoTs and new problems, rather than iteratively refining answers to a fixed question set.Source: Table 4 · p. 18, §C.1 · p. 18
D Computational Cost and Efficiency
- Approximately 55 hours on one 8×A100 machine produced 1.3M verified samples, using open-source models rather than proprietary APIs.The four stages were unifying Code CoTs (339K samples, 2h), scaling Code CoTs (5.3M, 8h), question reversal (4.6M, 6h), and answer generation (4.6M, 38.5h).Source: D Computational Cost and Efficiency · p. 18, Table 5 · p. 19
- Caco generated over one million verified, diverse reasoning samples in under three days on a single 8-GPU node.Code CoT generation and question reversal together took about 16 hours; answer generation consumed most of the reported runtime.Source: D Computational Cost and Efficiency · pp. 18–19
E Limitations
Caco's limitations concern problem coverage, translation from code to natural language, and the role of generated code in training; future work proposes broader domains and training uses.
E Limitations
- Caco's performance remains bounded by predefined training problem types, with unconventional problems posing a challenge and potentially biasing generated-data distributions.The authors identify complex or uncommon problem types as particularly difficult.Source: E Limitations · p. 19
- Converting executable code into natural-language instructions may lose details or require simplification, making the final output less rich or specific than the original reasoning.Source: E Limitations · p. 19
- Generated code is used primarily to filter data rather than as material for final training; the authors identify using it to improve data quality and training as future work.Source: E Limitations · p. 19
- Caco currently focuses mainly on mathematical and algorithmic reasoning; extending its application to logical puzzles or STEM problem solving requires further work.Source: E Limitations · p. 19
F Future Works
- Future directions include harder seed problems with hardness-aware sampling, broader domains and formal languages, and reinforcement learning with execution-based rewards and a complexity curriculum.The proposed diversity expansion includes domains such as data reasoning, procedural planning, code debugging, diagrams, physics problems, and proofs.Source: F Future Works · p. 20
G More Cases
This section presents Caco dataset examples across mathematical problem types, including prompts for code generation, back-translation, reasoning, and checking. It also gives two worked examples pairing Python code with step-by-step explanations for an expression calculation and a line–parabola tangency problem.
Prompts
- Caco prompts require executable Python solutions in a standardized input/output format and can back-translate code into a self-contained math problem, with scenario variation allowed.The required script defines an input dictionary, calls a function, assigns output, and prints it.Source: G More Cases · pp. 21, 23
- Separate prompts check code–reasoning consistency, problem solvability, and chain-of-thought correctness using single-word Yes/No answers.Source: G More Cases · p. 24
Case 1: Subsequence Counting Problem
- 8 unique subsequences are reported for the list [1, 2, 3], including the empty subsequence.The solution counts all 2^3 subsequences and notes that they are unique because the list elements are distinct.Source: G More Cases · p. 26
Case 2: Geometric Sequence Problem
- The geometric-sequence example asks for b when −1, a, b, c, and −9 are consecutive terms and uses b^2 = ac in its code.The accompanying explanation frames the task as solving for the middle term from the given sequence.Source: G More Cases · p. 27
Case 3: Counting and Combinatorics
- 14 four-digit numbers use only 1 and 3 while containing both digits at least once.The example counts 16 total strings and excludes the two strings containing only one digit.Source: G More Cases · p. 28
Case 4: Mathematical Expression Calculation
- The expression-calculation example uses square roots, an absolute value, and a zero-power term, then adds the simplified components.The Python implementation computes these components, while the accompanying solution explains the simplifications and addition.Source: G More Cases · p. 29
Case 5: Tangency Condition of Line and Parabola
- For the line x − y − 1 = 0 and parabola y = ax^2, tangency is tested by substitution and setting the resulting quadratic’s discriminant to zero.The code uses symbolic computation to form the quadratic and solve the discriminant condition for a; the supplied passages do not show the resulting value.Source: G More Cases · p. 30