PaperlayerView exact revision on arXiv

Source-linked AI summary

SIaM: Self-Improving Code-Assisted Mathematical Reasoning of Large Language Models

Dian Yu, Baolin Peng, Ye Tian, Linfeng Song, Haitao Mi, Dong Yu

arXiv:2408.15565v1cs.CL

TL;DR

Code-assisted LLMs still face computation and reasoning challenges, especially out of domain, while training on augmented data from a few datasets offers limited diversity and large-scale expert-written web QA remains underused. The paper uses a code-based critic to guide iterative data construction, filtering, and evaluation, alongside preference learning. It reports gains across English and Chinese in-domain and out-of-domain benchmarks.

  • Problem

    Code-assisted LLM studies largely use augmented data from a few datasets, while the potential of diverse, large-scale expert-written web math QA for improving these models remains underexplored.

    Source: §1 · p. 1

  • Method

    A code-based critic assesses execution results against reference answers to guide iterative question-code data construction and filtering, complementary evaluation, and preference learning.

    Source: §1 · p. 2

  • Results

    The paradigm improves results across English and Chinese in-domain and out-of-domain benchmarks, with reported gains up to +5.7% in-domain and +4.4% out-of-domain.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The critic can also serve as a complementary evaluation scorer, reducing reliance on heuristic-rule design for new evaluation tasks.

    Source: §1 · p. 2

    Web-data augmentation is left for future work, and annotating large-scale web questions with closed-source LLMs is beyond the paper’s scope.

    Source: §4 · p. 10

Abstract

from arXiv · show

There is a growing trend of teaching large language models (LLMs) to solve mathematical problems through coding. Existing studies primarily focus on prompting powerful, closed-source models to generate seed training data followed by in-domain data augmentation, equipping LLMs with considerable capabilities for code-aided mathematical reasoning. However, continually training these models on augmented data derived from a few datasets such as GSM8K may impair their generalization abilities and restrict their effectiveness to a narrow range of question types. Conversely, the potential of improving such LLMs by leveraging large-scale, expert-written, diverse math question-answer pairs remains unexplored. To utilize these resources and tackle unique challenges such as code response assessment, we propose a novel paradigm that uses a code-based critic model to guide steps including question-code data construction, quality control, and complementary evaluation. We also explore different alignment algorithms with self-generated instruction/preference data to foster continuous improvement. Experiments across both in-domain (up to +5.7%) and out-of-domain (+4.4%) benchmarks in English and Chinese demonstrate the effectiveness of the proposed paradigm.

1 INTRODUCTION

The introduction motivates using diverse, expert-written web math QA to address the limited generalization of code-assisted models trained on a few datasets. It presents a critic-guided self-improvement paradigm and reports gains across English and Chinese in-domain and out-of-domain benchmarks.

1 INTRODUCTION

  • Continued training on augmented versions of a few datasets is described as less effective for generalization because those datasets have limited diversity.Prior work commonly starts with closed-source-model labels and pattern-based answer matching before in-domain augmentation.Source: §1 · p. 1
  • Large-scale, expert-written web QA spans primary school to college and varied formats, offering an under-studied resource beyond small datasets such as GSM8K and MATH.The formats include multiple-choice, application, proof, and cloze questions.Source: §1 · pp. 1–2
  • A code-based critic judges whether code execution matches reference answers, guiding iterative generation, filtering, and supervised fine-tuning, with DPO and ORPO explored using critic-labeled preferences.The critic predicts YES or NO for code responses; retained responses train the policy in subsequent iterations.Source: §1 · p. 2
  • Up to +5.7% on in-domain benchmarks, +4.4% on out-of-domain benchmarks, and 11.9% over state-of-the-art 70B code-assisted math LLMs in OOD scenarios are reported for the proposed paradigm.Experiments cover English and Chinese benchmarks and 7–8B models.Source: §1 · p. 2
  • The critic also serves as a complementary evaluation scorer, reducing reliance on heuristic-rule design for new evaluation tasks.The introduction also reports a strong correlation between critic judgments and traditional heuristic evaluation.Source: §1 · p. 2

2 METHOD

The method initializes a policy from seed and self-distilled code data, then uses a critic to assess code against reference answers and execution results. It generates code on new questions and iteratively improves the policy with filtered supervised data and preference learning.

2.1 TRAINING AN INITIAL MODEL

  • A strong initial policy is built by fine-tuning on high-quality seed data, generating code, retaining up to four correct predictions per question, and training on combined seed and self-distilled data.The resulting M0 is used as the initial model for later stages.Source: §2.1 · p. 3

2.2 BUILDING A MULTI-USE CODE-BASED CRITIC MODEL

  • Pattern matching struggles with diverse answer formats and code syntax, motivating a critic trained to judge code responses against reference answers and execution results as YES or NO.The critic objective takes the question, reference answer, code response, and execution result as inputs.Source: §2.2 · pp. 3–4

2.3 CODE DATA GENERATION

  • For new questions, the policy generates five code samples; web-data responses are scored by the critic, while scripts can assess well-formatted data with mostly numerical answers.The reference answers verify execution results rather than serving as direct training targets.Source: §2.3 · p. 4

2.4 SELF-IMPROVEMENT WITH UNSEEN DATA

  • Self-improvement uses critic-accepted code pairs for SFT, filters noisy judgments by confidence and retains each question’s highest-scoring positive, discarding questions whose instances are all positive.The method also forms preference pairs from the highest-scoring YES and NO responses and jointly trains with an SFT objective to help stabilize updates and alleviate overfitting to preference data.Source: §2.4 · pp. 4–5

3 EXPERIMENTS

Experiments evaluate the data pipeline, self-improvement stages, alignment choices, and critic-based evaluation across English and Chinese math benchmarks. The self-improved models gain 4.4% accuracy on both MathBench subsets, while results also show improvements on Chinese benchmarks and strong agreement between the critic and heuristic evaluation.

3.1 DATA

  • The data pipeline combines English seed questions, Chinese educational-web questions, value-style datasets, and diverse-format QA resources, with the critic retaining judged positive and negative code examples.In a 50-instance analysis of web data, only 14% of reference answers were single numerical values, while 50% involved format conversion; 8% of questions were multiple-choice or multi-part.Source: §3.1 · pp. 5–6

3.2 IMPLEMENTATION

  • Experiments use 7–8B backbones and BF16 fine-tuning on eight A100 GPUs; each stage trains for two epochs, with distinct learning rates for SFT and preference learning.The full self-improving training takes approximately 96 hours; CodeLlama-7B-Python serves as the critic backbone.Source: §3.2 · p. 6

3.3 THE PERFORMANCE OF THE INITIAL POLICY AND SELF-IMPROVED LLMS

  • After two critic-guided Chinese-data iterations, M2 beats M0 on Chinese benchmarks without hurting English accuracy; English seed data also raises Chinese M0 averages.For DeepSeekcode, Chinese-only self-improvement may benefit MATH and GSM8K by +1.5% on each.Source: §3.3 · p. 6, Table 3 · p. 7, Table 4 · p. 7

3.4 THE COMPARISON OF DIFFERENT CHOICES OF DATA AND ALIGNMENT METHODS

  • With D0 and D1, staged SFT underperformed concatenated training, while adding broader D2 made the two strategies comparably effective.Filtered D2,SFT,H matched D2,SFT average accuracy at 30.6% of its size. WebInstruct produced smaller gains than Chinese D2 on Llama3instruct (+0.9% vs. +1.8%) and QWEN2Mathinstruct (+0.6% vs. +2.5%).Source: §3.4 · pp. 7–8, Table 5 · p. 8
  • Joint DPO and SFT performed similarly at λ = 1.0 and λ = 2.0 (77.8%), whereas removing SFT loss increased response length and yielded worse results than the reference policy.Using DPO positive pairs for another SFT round reduced downstream accuracy by 1.8%; ORPO was less effective than joint DPO and SFT.Source: Table 6 · p. 8, §3.4 · p. 8

3.5 USING THE CRITIC MODEL AS AN EVALUATOR

  • The critic and heuristic exact-match evaluators have Kendall’s τ of 0.79, with strong associations also observed on English tasks despite the critic’s Chinese-only training.The authors suggest that the English-focused backbone and the lesser role of language-specific knowledge in comparing mathematical answers may explain the English-task association.Source: Table 7 · p. 8, §3.5 · p. 9

3.6 THE PERFORMANCE OF SELF-IMPROVED LLMS ON MORE OUT-OF-DOMAIN TASKS

  • The self-improved models show a 4.4% accuracy improvement across the MathBench application and theoretical subsets overall, outperforming the initial model across both subsets, educational levels, and subjects.The 8B model also outperforms ToRA 70B on the English MathBench subset by 14.5% on application (A) and 9.2% on theoretical (T).Source: §3.6 · p. 9

4 RELATED WORK

Prior work uses heuristics and external tools to evaluate well-formatted answers, while other studies use LLMs or critics to score natural-language responses [Li et al., 2023; Yu et al., 2023b; Lu et al., 2023; Yuan et al., 2024; Xu et al., 2024]. This paper focuses on tool-assisted assessment of math-code responses and leaves web-data augmentation for future work.

4 RELATED WORK

  • Heuristics and external tools work well for single-value answers, but diverse answer formats make task-specific matching infeasible for web data [Fourrier et al., 2023; Gao et al., 2023a; Cobbe et al., 2021; Miao et al., 2020; Patel et al., 2021].For MATH, closed-source LLM evaluation is also used but is not cost-effective for rating large-scale code samples.Source: §4 · p. 10
  • Prior scoring and filtering work targets natural-language responses, whereas this paper focuses on tool-assisted assessment of code responses to math questions [Li et al., 2023; Yu et al., 2023b; Lu et al., 2023; Yuan et al., 2024; Xu et al., 2024].Here, “self” contrasts with continuous improvement by distilling knowledge from larger models.Source: §4 · p. 10
  • The paper leaves web-data augmentation with CoT or code for future work and uses GPT-4 only to annotate seed and critic data [Luo et al., 2023; Yu et al., 2023a; An et al., 2023; Li et al., 2024].Annotating large-scale web questions with closed-source LLMs is outside the paper’s scope.Source: §4 · p. 10

5 CONCLUSIONS AND FUTURE WORK

The paper presents a code-based critic paradigm for improving LLMs and investigates alignment using self-generated data. It reports effectiveness and identifies code-only post-training and critic improvement as future research directions.

5 CONCLUSIONS AND FUTURE WORK

  • A code-based critic guides question-code data creation and filtering, as well as complementary evaluation.The paradigm is presented as a way to improve LLMs.Source: §5 · p. 10
  • The paper investigates alignment algorithms using self-generated instruction and preference data for further improvement.The conclusion reports that self-improving LLMs using the paradigm are effective.Source: §5 · p. 10
  • Future work includes post-training on code-only data to enhance computational capabilities and improving the critic model.Source: §5 · p. 10

A.1 BACKBONE COMPARISONS FOR INITIAL MODEL SELECTION

The appendix describes backbone-selection choices and a development-set accuracy comparison for initial policy models across backbones.

A.1 BACKBONE COMPARISONS FOR INITIAL MODEL SELECTION

  • QWEN2’s math-specific variant was selected to ensure diversity among the backbone models.Source: §A.1 · p. 13
  • Llama3instruct was used because its fine-tuned performance differed only marginally from Llama3base.The passage also states that QWEN2 demonstrates strong performance.Source: §A.1 · p. 13
  • The appendix compares initial-policy accuracy across development sets of math datasets and backbone models.Source: Table 9 · p. 14

A.2 IMPACTS OF STAGES AND DATA SELECTION

The appendix identifies development-set ablations of stages and data selection, with the table caption specifying how best results are indicated.

A.2 IMPACTS OF STAGES AND DATA SELECTION

  • Table 10 covers impacts of stages and data selection on development-set performance.Source: Table 10 · p. 14
  • Table 10 highlights the best performance separately for each model family.Source: Table 10 · p. 14
  • The appendix reports ablation studies of the stages and data selection on dataset development sets.Source: §A.2 · p. 14, Table 10 · p. 14

A.3 SUB-TYPE PERFORMANCE ON MATHBENCH

SIaM improves across MathBench educational levels and task types, and using a Chinese system prompt also improves English-question performance. The paper reports an OOD advantage over ToRA-70B, while noting limits in comparing interactive methods on unseen formats.

A.3 SUB-TYPE PERFORMANCE ON MATHBENCH

  • SIaM(Llama3instruct)2 consistently outperforms SIaM(Llama3instruct)0 on MathBench-A and MathBench-T, including 6.7% Middle and 7.0% High gains on MathBench-A.MathBench-T gains include 8.1% for Middle school and 10.5% for English.Source: §A.3 · p. 14, Table 11 · p. 15
  • On MathBench, the model performs better on English questions when given the Chinese system prompt.Source: §A.3 · p. 14, Table 12 · p. 15
  • Interactive methods do not support robust comparison on unseen diverse formats, and the performance implications of multiple tool rounds remain unclear.The study therefore focuses on direct code use and leaves interactive settings for future work.Source: §A.3 · p. 15
  • The self-improved 8B model outperforms ToRA-70B across all MathBench OOD subcategories.Source: §A.3 · p. 15

A.5 OTHER ALIGNMENT ALGORITHMS

The appendix reports DPO outperforming ORPO with SFT and describes self-generated preference data. It also notes boundaries in resource language, resource release, and model scale.

A.5 OTHER ALIGNMENT ALGORITHMS

  • DPO outperforms ORPO both with the SFT loss.Table 16 reports self-improving performance across stages on development sets of different datasets.Source: §A.5 · p. 15, Table 16 · p. 16

A.6 SELF-GENERATED PREFERENCE DATA

LIMITATIONS

  • The educational QA resources and critic training focus on Chinese, although the critic is reported to rate English responses as effectively as Chinese ones.Source: LIMITATIONS · p. 17
  • The full-scale educational QA resources will not be released; the authors instead plan to provide samples and accompanying research assets.These include scripts, seed data in English, policy models, self-generated preference data, and the critic model.Source: LIMITATIONS · p. 17
  • Experiments focus on 7–8B LLMs because of limited computational resources, leaving larger-model scaling outside the experimental scope.Source: LIMITATIONS · p. 17
Loading 2408.15565v1…