PaperlayerView exact revision on arXiv

Source-linked AI summary

MathCoder2: Better Math Reasoning from Continued Pretraining on Model-translated Mathematical Code

Zimu Lu, Aojun Zhou, Ke Wang, Houxing Ren, Weikang Shi, Junting Pan, Mingjie Zhan, Hongsheng Li

arXiv:2410.08196v1cs.CLcs.AIcs.CV

TL;DR

Existing mathematical pretraining code often lacks explanations and is not directly focused on mathematical reasoning. The paper constructs MathCode-Pile by pairing extracted reasoning steps with generated, checked code, and reports improved performance across four base models and five benchmarks.

  • Problem

    Code used in mathematical pretraining often comes from fields beyond mathematics and lacks accompanying natural-language explanations or context.

    Source: §1 · pp. 1–2

  • Method

    The authors curate mathematical data, extract expressions with their conditions and results, generate corresponding Python code, and combine verified pairs with the basic corpus.

    Source: §1 · p. 2, §2.2 · p. 5

  • Results

    MathCode-Pile improves four base models across five mathematical benchmarks; MathCoder2-Llama-3-8B beats basic-data-only training by 3.1% on MATH and 4.1% on GSM8K.

    Source: §1 · p. 2, §6 · p. 11

  • Takeaways & Limitations

    The authors release the data-processing pipeline and training code to promote transparency, reproducibility, and collaboration in mathematical reasoning research.

    Source: §1 · p. 2, §6 · p. 11

    The corpus is English-only and focused primarily on mathematics, and resource limits restricted training to models with 1.3B–8B parameters.

    Source: §5 · p. 10

Abstract

from arXiv · show

Code has been shown to be effective in enhancing the mathematical reasoning abilities of large language models due to its precision and accuracy. Previous works involving continued mathematical pretraining often include code that utilizes math-related packages, which are primarily designed for fields such as engineering, machine learning, signal processing, or module testing, rather than being directly focused on mathematical reasoning. In this paper, we introduce a novel method for generating mathematical code accompanied with corresponding reasoning steps for continued pretraining. Our approach begins with the construction of a high-quality mathematical continued pretraining dataset by incorporating math-related web data, code using mathematical packages, math textbooks, and synthetic data. Next, we construct reasoning steps by extracting LaTeX expressions, the conditions needed for the expressions, and the results of the expressions from the previously collected dataset. Based on this extracted information, we generate corresponding code to accurately capture the mathematical reasoning process. Appending the generated code to each reasoning step results in data consisting of paired natural language reasoning steps and their corresponding code. Combining this data with the original dataset results in a 19.2B-token high-performing mathematical pretraining corpus, which we name MathCode-Pile. Training several popular base models with this corpus significantly improves their mathematical abilities, leading to the creation of the MathCoder2 family of models. All of our data processing and training code is open-sourced, ensuring full transparency and easy reproducibility of the entire data collection and training pipeline. The code is released at https://github.com/mathllm/MathCoder2 .

1 INTRODUCTION

The paper addresses the lack of mathematical focus and explanatory context in code used for continued mathematical pretraining. It pairs extracted mathematical reasoning steps with generated, verified code, forming MathCode-Pile, and reports improved benchmark performance across four base models.

1 INTRODUCTION

  • Pairing code with natural-language reasoning is intended to address the missing explanations or context in existing pretraining code and make mathematical reasoning more comprehensible.The authors say this paired data may also support later fine-tuning for Tool-Integrated Reasoning.Source: §1 · pp. 1–2
  • The method extracts LaTeX expressions, conditions, and results, generates Python snippets that reflect each reasoning step, and retains snippets that execute correctly against expected results.The paper describes Llama-3.1-70B-Instruct as the model used for extraction and translation.Source: §1 · p. 2, Figure 1 · p. 3
  • The authors combine 2.7B tokens of paired mathematical reasoning and code with basic pretraining data to create the 19.2B-token MathCode-Pile corpus.The basic dataset contains 16.5B tokens.Source: §1 · p. 2
  • MathCode-Pile significantly improves four base models across five mathematical benchmarks; MathCoder2-Llama-3-8B beats basic-data-only training by 3.1% on MATH and 4.1% on GSM8K.Its 4-shot accuracies are 38.4% on MATH and 69.9% on GSM8K.Source: §1 · p. 2
  • The authors release data-processing and training code as an open-source framework, which they describe as performance-competitive with DeepSeekMath, InternLM-Math, and Qwen2.5-Math.They contrast this release with those works releasing model weights only.Source: §1 · p. 2

2 CURATION OF MATHCODE-PILE

MathCode-Pile is built from diverse mathematical data and an added corpus of model-generated code paired with reasoning steps. The pipeline filters source material, checks generated code against expected results, and combines retained examples with the basic data.

2 CURATION OF MATHCODE-PILE

  • MathCode-Pile is curated in two stages: collecting basic mathematical pretraining data, then generating paired reasoning steps and code from it.The basic data draws on mathematical web content, synthetic data, code using mathematical packages, and textbooks.Source: §2 · p. 3, §2.1 · p. 3

2.1 BASIC DATA

  • Filtered-OpenWebMath contains 4.8B tokens, down from 13.7B; adding 6.4B tokens of filtered Common Crawl math data yields 11.2B web-data tokens.Common Crawl documents undergo an initial fastText filter and a second classifier trained from annotations.Source: §2.1 · p. 3
  • Basic data also includes 2.2B synthetic math tokens, 1.7B tokens of code using mathematical packages, and 8K textbook PDFs converted to markdown.The code collection retains programs importing selected math-related packages; numpy is excluded as a filtering criterion because it is common in non-mathematical contexts.Source: §2.1 · p. 4

2.2 MODEL-TRANSLATED MATHEMATICAL CODE

  • The model extracts LaTeX expressions with their conditions and expected results, combines these into reasoning steps, and generates Python code to reflect the underlying reasoning.The extraction and translation model used in this paper is Llama-3.1-70B-Instruct.Source: §2.2 · pp. 4–5
  • Generated snippets are retained only when they execute successfully and match expected results; benchmark overlaps are removed using exact matching and 13-gram deduplication.The deduplication condition uses Jaccard similarity greater than 0.6.Source: §2.2 · p. 5
  • Filtering reduces the initially generated 3.1B tokens to 2.7B of paired code and reasoning, which combines with the basic data to form 19.2B-token MathCode-Pile.The authors say the dataset is tailored to mathematical and coding abilities.Source: §2.2 · p. 5

3 EXPERIMENTS

The experiments evaluate MathCoder2 across five mathematical benchmarks, test supervised fine-tuning, and ablate dataset components. Continued pretraining improves performance across benchmarks, while the analyses compare data sources and training configurations.

3 EXPERIMENTS

  • The experiments evaluate four continued-pretraining base models against similarly sized mathematical models, then examine post-training and dataset ablations.The benchmark suite spans GSM8K, MATH, SAT-Math, OCW, and MMLU-Math; GSM8K and MATH use 4-shot evaluation, while MMLU-Math uses zero-shot.Source: §3 · pp. 5–6, §3.1 · p. 6

3.1 MAIN RESULTS

  • Continued pretraining on MathCode-Pile consistently improves all five benchmark results, and MathCoder2 models rival leading models while remaining behind Qwen2-Math and Qwen2.5-Math.The authors suggest the gap may relate to those models’ greater resources and reported 700B-token mathematical dataset.Source: Table 2 · p. 6, §3.1 · pp. 6–7

3.2 POST-TRAINING

  • MathCoder2-Instruct-TIR reaches 69.7% on MATH and 86.5% on GSM8K after supervised fine-tuning.The authors describe this as validation of downstream potential; they report only simple supervised fine-tuning, without reinforcement learning or direct preference optimization.Source: §3.2 · p. 7, Table 3 · p. 7

3.3 ABLATION STUDIES

  • Mathematical code constitutes 14.1% of MathCode-Pile, and its addition improves accuracy across all five benchmarks relative to basic data.The reported share of total accuracy improvement attributed to code ranges from 21.8% to 66.2% across the five datasets.Source: §3.3 · p. 8, Table 4 · p. 8
  • Combining natural-language reasoning steps with translated code performs best among the tested configurations, and outperforms a text-rewriting prompt baseline.The comparisons include reasoning-step-only, translated-code-only, and concatenated reasoning-step-and-code data.Source: §3.3 · p. 8, Table 4 · p. 8
  • Filtered OpenWebMath and filtered CC-En-math significantly improve results, while smaller data sources add modest gains when combined with filtered OpenWebMath.Removing irrelevant content from OpenWebMath and initially filtered CC-En causes no obvious performance degradation.Source: §3.3 · p. 8, Table 5 · p. 9
  • On DeepSeekCoder-1.3B, MathCode-Pile outperforms OpenWebMath, Proof-Pile-2, MathPile, and the DeepSeekMath Corpus.The DeepSeekMath comparison uses a model trained for 150B tokens, versus approximately 60B tokens for MathCode-Pile.Source: §3.3 · pp. 8–9, Table 6 · p. 9
  • Fine-tuning MathCoder2-Llama-3-8B on NuminaMath-TIR yields higher results than fine-tuning either the original Llama-3-8B or the basic-data MathCoder2 model.The comparison tests the models after three epochs of fine-tuning.Source: §3.3 · p. 9

4 RELATED WORK

Related work includes mathematical web data, synthetic data, textbooks, and code in continued pretraining, but prior code often lacks natural-language reasoning context. This paper extracts reasoning steps and generates accompanying mathematical code, while also open-sourcing its data-processing and training code.

Continued mathematical pretraining

  • Prior continued-pretraining work uses mathematical web content, synthetic data, textbooks, and code, but existing math-related code mostly lacks natural-language explanations or context.MathCode2 describes its approach as building on and expanding these sources, with finer filtering of relevant data.Source: §4 · pp. 9–10

Synthetic data

  • Unlike template-based synthetic numerical data, model-generated textbook data, and entity-linked text, this work extracts reasoning steps and generates code that captures the reasoning process.The cited examples include MathGLM and InternLM-Math, phi, and EntiGraph.Source: §4 · p. 10

Post-training

  • The paper explores supervised fine-tuning with existing open-source data as one approach to downstream mathematical problem-solving.Related work also discusses Chain-of-Thought, Program-of-Thought, Tool-Integrated Reasoning, reinforcement learning, and Direct Preference Optimization.Source: §4 · p. 10

5 LIMITATIONS AND FUTURE WORK

The work’s scope is limited to English mathematical pretraining, models with 1.3B–8B parameters, and continued pretraining without preference optimization or reinforcement learning.

5 LIMITATIONS AND FUTURE WORK

  • The corpus focuses primarily on mathematics and does not intentionally include subjects such as physics or chemistry.The authors identify other STEM subjects as a possible expansion.Source: §5 · p. 10
  • The pretraining data is entirely English, and the trained models range from 1.3B to 8B parameters because of computational-resource limits.The authors suggest expanding to other languages and larger models as future work.Source: §5 · p. 10
  • The paper does not use PPO, GRPO, or Direct Preference Optimization in post-training, though the authors identify these methods as future work.The paper focuses on continued mathematical pretraining.Source: §5 · p. 10

6 CONCLUSION

The conclusion presents a curated mathematical dataset and a method for adding reasoning-linked code, combined as a 19.2B-token corpus. The appendix describes annotation prompts for filtering math web documents.

6 CONCLUSION

  • The basic dataset is curated from diverse math-related texts, including web content, synthetic data, package-using code, and textbooks.The paper describes the continued mathematical pretraining pipeline as open-source.Source: §6 · p. 10
  • The resulting 19.2B-token MathCode-Pile combines basic data with model-generated mathematical code paired with reasoning steps.The authors report improved performance for four base models across five representative mathematical benchmarks.Source: §6 · p. 11

A PROMPT FOR ANNOTATION OF MATH WEB DOCUMENTS

  • The annotation prompt classifies documents into seven types, which the authors say filters irrelevant text better than binary math-related classification.The prompt is used for documents in OpenWebMath and initially filtered CC-En.Source: A PROMPT FOR ANNOTATION OF MATH WEB… · p. 14
  • Figure 4’s prompt asks Llama-3.1-70B-Instruct to rewrite and improve a document’s text, with {TEXT} replaced by that document’s content.Source: Figure 4 · p. 15

B TRAINING DETAILS OF FASTTEXT CLASSIFIERS

The fastText classifier configuration uses fixed training settings and separate positive and negative samples for initial and finer filtering.

B TRAINING DETAILS OF FASTTEXT CLASSIFIERS

C EXAMPLES OF THE MODEL-GENERATED MATHEMATICAL CODE

The examples show a translation process that extracts mathematical expressions with context before generating code from the extracted information.

C EXAMPLES OF THE MODEL-GENERATED MATHEMATICAL CODE

D EXAMPLES OF REMOVED IRRELEVANT TEXTS

The appendix shows examples of OpenWebMath documents removed because they were not relevant to mathematical reasoning, and reports no obvious impact from their removal on mathematical continued pretraining performance.

D EXAMPLES OF REMOVED IRRELEVANT TEXTS

E PROMPT FOR SIMPLE REWRITING TO IMPROVE QUALITY FOR ABLATION STUDY

The ablation uses a rewrite prompt for Llama-3.1-70B-Instruct to check whether improvements could be explained solely by the quality of generated text.

E PROMPT FOR SIMPLE REWRITING TO IMPROVE QUALITY FOR ABLATION STUDY

F COMPARISON BETWEEN ADDING AND NOT ADDING MATHEMATICAL CODE

On Llama-3 8B, adding model-translated mathematical code improves accuracy across training steps on GSM8K and MATH.

F COMPARISON BETWEEN ADDING AND NOT ADDING MATHEMATICAL CODE

Loading 2410.08196v1…