PaperlayerView exact revision on arXiv

Source-linked AI summary

Advancing Mathematical Reasoning in Language Models: The Impact of Problem-Solving Data, Data Synthesis Methods, and Training Stages

Zui Chen, Tianqiao Liu, Mi Tian, Qing Tong, Weiqi Luo, Zitao Liu

arXiv:2501.14002v3cs.CLcs.AI

TL;DR

The paper asks whether problem-solving data can improve mathematical reasoning during CPT, and how synthesis and training stage affect learning. It tests alternative pre-training data and synthesis methods, and compares CPT with SFT. Problem-solving data and tutorship amplification show strong results, while same-data SFT gains less mathematical reasoning than CPT.

  • Problem

    The paper examines whether problem-solving data during CPT can improve mathematical reasoning over general mathematical corpora, which synthesis methods are effective, and how CPT and SFT differ on the same data.

    Source: §1 · p. 2

  • Method

    The study compares problem-solving data with mathematical corpora during CPT, tests four synthesis methods, and analyzes capability learning across CPT and SFT.

    Source: §1 · p. 2, §4 · p. 5

  • Results

    Problem-solving data improves mathematical capabilities over general mathematical corpora, tutorship amplification is particularly effective, and same-data SFT gains about 60% of CPT’s mathematical-reasoning gain.

    Source: §1 · p. 2, §5.1 · p. 7

  • Takeaways & Limitations

    The authors conclude that problem-solving data, especially more challenging examples during CPT, can support mathematical capability learning and produce a competitive base model, MathGPT-8B.

    Source: §1 · p. 3, §7 · p. 10

    Future work should refine the relatively naive synthesis methods and investigate post-training alignment.

    Source: §7 · p. 10

Abstract

from arXiv · show

Mathematical reasoning remains a challenging area for large language models (LLMs), prompting the development of math-specific LLMs such as LLEMMA, DeepSeekMath, and Qwen2-Math, among others. These models typically follow a two-stage training paradigm: pre-training with math-related corpora and post-training with problem datasets for supervised fine-tuning (SFT). Despite these efforts, the improvements in mathematical reasoning achieved through continued pre-training (CPT) are often less significant compared to those obtained via SFT. This study addresses this discrepancy by exploring alternative strategies during the pre-training phase, focusing on the use of problem-solving data over general mathematical corpora. We investigate three primary research questions: (1) Can problem-solving data enhance the model's mathematical reasoning capabilities more effectively than general mathematical corpora during CPT? (2) Are synthetic data from the same source equally effective, and which synthesis methods are most efficient? (3) How do the capabilities developed from the same problem-solving data differ between the CPT and SFT stages, and what factors contribute to these differences? Our findings indicate that problem-solving data significantly enhances the model's mathematical capabilities compared to general mathematical corpora. We also identify effective data synthesis methods, demonstrating that the tutorship amplification synthesis method achieves the best performance. Furthermore, while SFT facilitates instruction-following abilities, it underperforms compared to CPT with the same data, which can be partially attributed to its poor learning capacity for more challenging problem-solving data. These insights provide valuable guidance for optimizing the mathematical reasoning capabilities of LLMs, culminating in our development of a powerful mathematical base model called MathGPT-8B.

1 INTRODUCTION

The introduction frames mathematical reasoning as a challenge of applying mathematical knowledge, while common training separates math-corpus pre-training from problem-data post-training. It proposes using problem-solving data during CPT and investigates data choice, synthesis, and differences between CPT and SFT.

1 INTRODUCTION

  • The common paradigm uses math-related corpora during pre-training and problem datasets for SFT, which supports instruction-following and desired output formats.Source: §1 · pp. 1–2
  • The authors motivate the approach by distinguishing mathematical reasoning and planning from memorizing and understanding knowledge, and lay out three questions on CPT data, synthesis methods, and CPT–SFT learning differences.Source: §1 · p. 2
  • Problem-solving data during CPT significantly improves mathematical capabilities over general mathematical corpora, and a higher proportion is more effective.Source: §1 · p. 2
  • Tutorship amplification is the strongest tested synthesis method, using a teacher model to detect and correct errors in student responses.Response diversification and query expansion are also effective.Source: §1 · p. 2
  • SFT facilitates some mathematical learning but has a clear disadvantage compared with CPT.Source: §1 · p. 2
  • Both stages mainly develop capabilities aligned with their data distributions, but SFT has weaker in-domain learning; its out-of-domain findings are less clear, and it is more susceptible to distribution disturbances.Source: §1 · p. 3
  • More challenging problem-solving data supports more effective learning, especially in CPT, and may be the primary source of the CPT–SFT learning gap.The authors recommend preparing more challenging problem-solving data for CPT.Source: §1 · p. 3
  • The study applies its optimal strategy to Llama3-8B to produce MathGPT-8B, which surpasses DeepSeekMath-Base-7B and Qwen2-Math-7B and is comparable to Qwen2-Math-72B and Qwen2.5-Math-7B.It uses 100B mathematical tokens, one-tenth of Qwen2.5-Math-7B’s amount, and undergoes no post-training.Source: §1 · p. 3

2 EXPERIMENTAL PREPARATION

The experimental preparation specifies the training-data groups, base model, evaluation sets, contamination controls, and scoring procedure used in the study.

2 EXPERIMENTAL PREPARATION

  • Training data comprises general, mathematical, and problem-solving groups, with 48.3B, 13.7B, and 7.2B tokens respectively, plus 30.54B synthetic tokens.The problem-solving sources include NuminaMath, Lila, and proprietary data; 14 million pieces were used for synthetic augmentation.Source: §2 · p. 3
  • Llama2 is the base model, selected because it predates OpenWebMath and thereby mitigates contamination risk from that newer corpus.Source: §2 · p. 3
  • Evaluation covers GAOKAO, ZHONGKAO, GSM8K, and MATH, with the first two included to broaden assessment and reduce contamination risk.Source: §2 · p. 3
  • MinHash deduplication removes documents with substantial duplicate content, while byte thresholds support deduplication and decontamination, particularly for OpenWebMath.Source: §2 · p. 3
  • For each dataset, scoring takes the higher accuracy from zero-shot and few-shot prompting, then reports the arithmetic mean across datasets.The evaluation process includes inference, answer comparison for irregular outputs, and statistical scoring.Source: §2 · pp. 3–4

3 IMPROVING REASONING ABILITY IN CPT WITH PROBLEM-SOLVING DATA

The section tests whether problem-solving data improves CPT reasoning over mathematical corpora and examines the effects of different data mixture ratios. Its reported results favor problem-solving data and a higher proportion of it.

3 IMPROVING REASONING ABILITY IN CPT WITH PROBLEM-SOLVING DATA

  • The experiments aim to validate problem-solving data during CPT and identify an effective proportion for mixing it with math-corpus data.Source: §3 · p. 4
  • The experiments hold total math-data quantity constant while varying the math-corpus/problem-solving-data mixture across three test groups.The groups use ratios of 5:5, 3:7, and 7:3, respectively.Source: §3 · p. 4
  • The section uses Llama2 for 25,000 CPT steps and selects the checkpoint with the lowest validation loss for evaluation.Training uses a global batch size of 1024 and a context length of 4096 tokens.Source: §3 · p. 4
  • The three test groups using problem-solving data improve more than the same-token Base1 group using a math corpus, and the green-line test group has the highest average accuracy among the three tests as steps increase.Test4, which uses only problem-solving data, performs comparably to or above the green-line group.Source: Figure 1 · p. 4, §3 · pp. 4–5

4 EXPLORATION OF EFFICIENT DATA SYNTHESIS METHODS

The section evaluates four ways to synthesize data from problem-solving examples. It reports that response diversification, query expansion, and tutorship amplification are effective, with tutorship amplification particularly strong.

4 EXPLORATION OF EFFICIENT DATA SYNTHESIS METHODS

  • Response diversification generates two distinct solutions from a question and original answer, then retains a solution with the correct final answer.The method does not alter answers and therefore requires no additional labeling.Source: §4 · p. 5
  • Query expansion reshapes existing questions and answers into new questions, then supplies and evaluates answers with reasoning.The process aims to ensure new questions are valid and solvable and their answers are assessed.Source: §4 · p. 5
  • Retrospective enhancement inserts later steps into earlier ones so models can retry after regret, using a special [back] token.The study tests this method on a more diverse question set and a mainstream model with existing mathematical capability.Source: §4 · p. 5
  • Tutorship amplification has a strong teacher model check a student model’s answer, identify erroneous steps, and continue solving from the error when needed.The process aims to create realistic errors and support self-evaluation, error identification, and timely correction.Source: §4 · pp. 5–6
  • A filtered seed set from the original problem-solving data is augmented using the four synthesis methods; the experiments add synthetic tokens to a control group and evaluate converged checkpoints.Source: §4 · p. 6
  • Tutorship amplification has particularly pronounced effects; response diversification and query expansion are also effective, while retrospective enhancement has minimal influence.The authors hypothesize that tutorship data teaches self-checking, error detection, correction, and knowledge application through error correction.Source: §4 · p. 6, Table 1 · p. 6

5 ABILITIES ACQUISITION COMPARISON OF CPT AND SFT STAGES

The section compares how CPT and SFT learn from problem-solving data, examining overall capability gains, data-distribution effects, and problem difficulty. CPT shows an advantage, especially for more challenging data, while both stages primarily learn capabilities aligned with their training data.

5.1 COMPARISON OF ABILITIES ACQUISITION

  • After accounting for instruction-following gains, SFT’s capability gain is about 60% of CPT’s, and Base1-SFT remains below Base2.A small amount of SFT data improves instruction following, partly by reducing inconsistent and repetitive outputs.Source: §5.1 · p. 7, Figure 2 · p. 7

5.2 IMPACT OF DIFFERENT DATA DISTRIBUTIONS

  • Both CPT and SFT primarily develop capabilities aligned with their data distributions, but SFT has weaker in-domain learning and is more susceptible to out-of-domain disruption.The out-of-domain findings are less clear overall; disruption is especially noticeable on GSM8K, which has the largest distributional difference.Source: §5.2 · p. 8, Table 2 · p. 8

5.3 IMPACT OF DIFFERENT DIFFICULTY LEVELS

  • CPT’s advantage over SFT is especially evident on challenging problem-solving data, and hard-data training improves results across datasets.The authors identify this stronger learning of challenging data as a possible primary source of the CPT–SFT difference.Source: §5.3 · p. 9, Table 3 · p. 9
  • For both training stages, gains are largest on easy problems and decrease as problem difficulty increases.This pattern holds for easy- and hard-data training groups, indicating that both stages primarily learn to solve simpler, fewer-step problems.Source: §5.3 · p. 9, Table 3 · p. 9

6 TRAINING A STRONG MATH-SPECIFIC MODEL

The authors apply findings from their research questions to train MathGPT-8B on Llama3-8B, using a data mixture that emphasizes synthesized problem-solving examples. The resulting model surpasses DeepSeekMath-Base-7B and Qwen2-Math-7B and is comparable to larger or more recently released math-specific models.

6 TRAINING A STRONG MATH-SPECIFIC MODEL

  • The model is trained on Llama3-8B using a 3:7 mathematical-corpus-to-problem-solving-data ratio and synthesized data, with emphasis on tutorship amplification.The authors also prioritize problems with more than five reasoning steps as seed data for synthesis.Source: §6 · pp. 9–10
  • MathGPT-8B scores 0.6222 on MMLU, compared with 0.6211 for Llama3-8B, maintaining similar general-knowledge benchmark performance.The authors also report improved foundational capabilities over the base model and results exceeding Llama3.1-70B and Qwen2-72B.Source: §6 · p. 10, Table 4 · p. 10
  • MathGPT-8B surpasses DeepSeekMath-Base-7B and Qwen2-Math-7B and is comparable to Qwen2-Math-72B and Qwen2.5-Math-7B.It uses 140 billion tokens, including 100 billion math-related tokens, versus 1 trillion tokens for Qwen2.5-Math-7B, and starts from a weaker base model.Source: §6 · p. 10, Table 4 · p. 10

7 CONCLUSION

The conclusion presents MathGPT-8B as a competitive model developed through alternative pre-training strategies and identifies data synthesis and post-training alignment as future research areas. The appendices describe the training-data groups and explain the choice of Llama2 as the experimental base model.

7 CONCLUSION

  • The authors identify refining data synthesis and examining post-training alignment as priorities for future work.They describe their current synthesis approaches as relatively naive.Source: §7 · p. 10

A.1 TRAINING DATA DETAILS

  • The mathematical corpus provides foundational mathematical content, while problem-solving data is intended to improve reasoning more efficiently.The mathematical corpus primarily includes general mathematical content from sources such as OpenWebMath; the general corpus also supports retention of broader knowledge during specialized training.Source: §A.1 · pp. 13–14
  • Problem-solving data comprises 25 million examples, including 14 million seed examples used to create synthetic data.The Llama2 experiments use 48.3B general-corpus tokens, 13.7B mathematical-corpus tokens, 7.2B problem-solving-data tokens, and 30.54B synthetic-data tokens.Source: §A.1 · p. 14

A.2 BASE MODEL SELECTION

  • Llama2 was chosen as the base model because it predates OpenWebMath, reducing concern that the corpus was already widely incorporated into the model.The authors say this choice aims to make their findings more robust.Source: §A.2 · p. 14

A.3 EVALUATION DATASETS

The appendix broadens evaluation beyond GSM8K and MATH with translated Chinese entrance-exam problems, removes overlapping training documents, and reports average accuracy across four datasets. These choices address evaluation coverage and some contamination from duplicated or paraphrased text.

A.3 EVALUATION DATASETS

  • GAOKAO and ZHONGKAO add translated, recent exam problems to complement GSM8K and MATH and broaden the evaluated ability dimensions.The retained sets contain 235 GAOKAO items and 658 ZHONGKAO items; both were created after Llama2 was released.Source: §A.3 · p. 14

A.4 DEDUPLICATION AND DECONTAMINATION

  • MinHash deduplication removes documents with more than 2048 bytes of duplicate text and filters overlaps exceeding 100 bytes with evaluation data.The authors report removing 2,594 contaminated OpenWebMath documents, which significantly affected initial experimental conclusions.Source: §A.4 · p. 14

A.5 EVALUATION METRICS

  • For each dataset, the evaluation reports the higher accuracy from zero-shot and few-shot prompting, then averages accuracy across the four datasets.Few-shot settings use eight examples for GSM8K and four for MATH, with the same settings applied to ZHONGKAO and GAOKAO.Source: §A.5 · pp. 14–15

B ABILITY DIMENSIONS OF THE FOUR EVALUATION SETS

The four evaluation sets differ in general-knowledge, math-knowledge, and reasoning-step demands, and their problem distributions also vary. MATH, ZHONGKAO, and GAOKAO are distributionally correlated, while GSM8K differs most.

B ABILITY DIMENSIONS OF THE FOUR EVALUATION SETS

  • The evaluation framework defines general knowledge, math knowledge, and reasoning steps, each divided into three progressively demanding levels.General knowledge covers common sense; math knowledge ranges from arithmetic through advanced mathematics; reasoning steps measure reasoning depth.Source: B ABILITY DIMENSIONS OF THE FOUR… · p. 15
  • GAOKAO and MATH represent similar capability dimensions, while GAOKAO may require general knowledge for some problems.ZHONGKAO and GSM8K both demand more general knowledge but differ in math-knowledge and reasoning-step requirements.Source: B ABILITY DIMENSIONS OF THE FOUR… · p. 15, Table 5 · p. 15
  • MATH, ZHONGKAO, and GAOKAO have certain distributional correlations, whereas GSM8K exhibits the largest distributional difference.The analysis sampled up to 1,000 problems per set, used t-SNE for visualization, and calculated cosine similarity.Source: B ABILITY DIMENSIONS OF THE FOUR… · p. 15

C DISCUSSION ON EXPERIMENTAL SETTINGS

The experiments use controlled comparisons to isolate the effects of problem-solving data, synthetic data, and training stage. Further comparisons examine data-distribution alignment and training-data difficulty.

C DISCUSSION ON EXPERIMENTAL SETTINGS

  • The Section 3 control and test groups use the same 14.7B math-token total, with the test groups varying the mixture of problem-solving data and math corpus.The tested mixture ratios include 5:5, 3:7, and 7:3; the 3:7 setting is intended to use the smaller math corpus fully within 10K steps.Source: C DISCUSSION ON EXPERIMENTAL SETTINGS · pp. 15–16
  • Section 4 adds synthetic data to a group already using the original problem-solving data, testing whether synthesis contributes new value rather than replacing it.Source: C DISCUSSION ON EXPERIMENTAL SETTINGS · p. 16
  • Section 5.1 compares CPT and SFT using the same problem-solving data, including 1% SFT groups intended to isolate instruction-following effects.Comparisons among Base1-SFT, Base1-1%SFT, and Base2-1%SFT assess reasoning gains associated with SFT and CPT.Source: C DISCUSSION ON EXPERIMENTAL SETTINGS · p. 17
  • Section 5.2 compares SFT and CPT on middle-school and high-school subsets, evaluating in-domain learning and out-of-domain transfer across evaluation sets.The high-school subset comparisons repeat the design to strengthen robustness.Source: C DISCUSSION ON EXPERIMENTAL SETTINGS · p. 17, Figure 4 · p. 17
  • Section 5.3 compares easy and hard training subsets within and across SFT and CPT, using evaluation-set distributions only as supplementary analysis.Source: C DISCUSSION ON EXPERIMENTAL SETTINGS · p. 17

D DETAILED RESULTS OF PROBLEM-SOLVING DATA EFFECTIVENESS EXPERIMENT

Problem-solving data can outperform adding more mathematical tokens, although removing the math corpus changes which evaluation sets benefit. The experiments compare mixtures and report accuracy across four evaluation sets.

D DETAILED RESULTS OF PROBLEM-SOLVING DATA EFFECTIVENESS EXPERIMENT

E DETAILED RESULTS OF COMPARISON OF ABILITIES ACQUISITION

The detailed results section points to evaluation metrics for SFT, while related work describes prior math-CPT approaches and the appendix supplies synthesis-prompt examples.

F RELATED WORK

  • Prior math-CPT work includes open-source mathematical corpora, data-quality scoring, and synthetic data, with Qwen2-Math and Qwen2.5-Math reported to improve using synthesis.LLEMMA introduced Proof-Pile-2; DeepSeekMath used fastText retrieval for OpenWebMath-like pages, while InternLM-Math trained a scoring model for dataset quality.Source: F RELATED WORK · p. 18

E DETAILED RESULTS OF COMPARISON OF ABILITIES ACQUISITION

G SYNTHETIC DATA PROMPT

  • The appendix prompt asks a model to produce two distinct additional solutions and refuse when it cannot do so reliably.Source: G SYNTHETIC DATA PROMPT · p. 20
  • Other prompts generate new word problems from a given problem and check proposed solutions, correcting errors when found.Source: G SYNTHETIC DATA PROMPT · p. 20

Related papers

Loading 2501.14002v3…