PaperlayerView exact revision on arXiv

Source-linked AI summary

OpenMathInstruct-2: Accelerating AI for Math with Massive Open-Source Instruction Data

Shubham Toshniwal, Wei Du, Ivan Moshkov, Branislav Kisacanin, Alexan Ayrapetyan, Igor Gitman

arXiv:2410.01560v2cs.CLcs.AIcs.LG

TL;DR

Open-source math-reasoning research lacks access to leading training data and evidence about synthesis choices. This paper studies those choices through ablations, builds OpenMathInstruct-2, and reports a 15.9% absolute MATH gain over Llama3.1-8B-Instruct.

  • Problem

    Leading math-reasoning training datasets are largely closed-source, limiting access to progress and understanding of data-synthesis choices.

  • Method

    The paper conducts ablation studies on math-reasoning SFT data synthesis and releases the resulting OpenMathInstruct-2 dataset, code, and finetuned models.

  • Results

    15.9% absolute improvement on MATH over Llama3.1-8B-Instruct was achieved by finetuning Llama3.1-8B-Base with OpenMathInstruct-2.

  • Takeaways & Limitations

    Strong teacher data, concise solution formats, tolerance of up to 20% low-quality data, and greater question diversity are supported design choices for math-reasoning SFT.

  • Takeaways & Limitations

    OpenMathInstruct-1 has limited question diversity and underrepresents challenging problems because its questions come from MATH and GSM8K training sets and were synthesized by open-weight models.

Abstract

from arXiv · show

Mathematical reasoning continues to be a critical challenge in large language model (LLM) development with significant interest. However, most of the cutting-edge progress in mathematical reasoning with LLMs has become \emph{closed-source} due to lack of access to training data. This lack of data access limits researchers from understanding the impact of different choices for synthesizing and utilizing the data. With the goal of creating a high-quality finetuning (SFT) dataset for math reasoning, we conduct careful ablation experiments on data synthesis using the recently released \texttt{Llama3.1} family of models. Our experiments show that: (a) solution format matters, with excessively verbose solutions proving detrimental to SFT performance, (b) data generated by a strong teacher outperforms equally-sized data generated by a weak student model, (c) SFT is robust to low-quality solutions, allowing for imprecise data filtering, and (d) question diversity is crucial for achieving data scaling gains. Based on these insights, we create the OpenMathInstruct-2 dataset, which consists of 14M question-solution pairs ($\approx$ 600K unique questions), making it nearly eight times larger than the previous largest open-source math reasoning dataset. Finetuning the \texttt{Llama-3.1-8B-Base} using OpenMathInstruct-2 outperforms \texttt{Llama3.1-8B-Instruct} on MATH by an absolute 15.9\% (51.9\% $\rightarrow$ 67.8\%). Finally, to accelerate the open-source efforts, we release the code, the finetuned models, and the OpenMathInstruct-2 dataset under a commercially permissive license.

1. Introduction

OpenMathInstruct-2 addresses limited access to open, commercially permissive math-reasoning data by combining ablation-guided synthesis with a substantially larger dataset. Its finetuned models achieve strong MATH performance, including a 15.9% absolute gain over Llama3.1-8B-Instruct.

  • Motivation: Closed or restrictive math datasets limit researchers’ access to training data and their ability to study synthesis and data-composition choices.OpenMathInstruct-1 also had limited question diversity and underrepresented challenging problems relative to GPT-based counterparts.
  • Approach: Ablations examine solution format, teacher-model choice, low-quality solutions, and question diversity to guide construction of OpenMathInstruct-2.The dataset uses Llama3.1 models for synthetic math instruction-tuning data and evaluates quality through finetuning 8B and 70B base models.
  • Ablation findings: 3.9% higher performance with solutions 40% shorter shows that the proposed OpenMath CoT format outperforms Llama’s more verbose CoT format.Using the base-model template also increases adherence to the proposed format for instruct models following few-shot examples.
  • Ablation findings: 7.8% better performance from a strong teacher’s data, robustness to up to 20% low-quality data, and a 10.5% gain from 1K to 6.5K unique questions summarize the main ablation findings.These comparisons control for SFT data size where specified.
  • Dataset: 14M question-solution pairs and 600K unique questions make OpenMathInstruct-2 about eight times larger than the previous biggest standalone open-source dataset.The dataset includes 592K synthesized questions and is constructed using Llama-3.1-405B-Instruct with decontamination and manual inspection.
  • Results: 67.8% MATH accuracy lets OpenMath2-Llama3.1-8B outperform Llama3.1-8B-Instruct by 15.9% absolute with SFT.The 70B model reaches 71.9% accuracy and exceeds Llama3.1-70B-Instruct by 3.9%.

2. Data: Solution Augmentation

The solution augmentation pipeline synthesizes and filters mathematical solutions, then uses controlled ablations to study how format, teacher strength, data quality, and question diversity affect SFT.

  • Pipeline: OpenMathInstruct-2 synthesizes solutions for existing questions using few-shot prompting and adds high-quality question-solution pairs to the SFT dataset.Multiple candidate solutions are sampled, and usually those leading to the correct answer are retained.
  • Dataset comparison: Fair Downsampling represents questions as equally as possible, while Matching Coverage equalizes unique questions and per-question solution counts across datasets.Matching Coverage controls dataset size when comparing design choices.
  • Solution format: OpenMath CoT is 40% less verbose than Llama CoT and produces better SFT performance after datasets are matched at 260K question-solution pairs.All subsequent experiments use the OpenMath CoT format.
  • Teacher model: A strong teacher model outperforms a weaker student model when SFT data coverage is matched.The authors compare Llama3.1-405B-Instruct with Llama3.1-8B-Base and suggest weaker models produce noisier reasoning.
  • Low-quality solutions: SFT performance changes little with up to 20% incorrect solutions, and filtering strategies remove 6% to 12% of data without meaningful gains over the unfiltered baseline.Incorrect Pairing remains strong even with 40% incorrect solutions, while filtering accuracy is imperfect.
  • Question diversity: With 256K question-solution pairs, increasing unique questions from 1K to 6.5K improves SFT performance, while restricting diversity causes a drop of more than 10 points.The result supports generating new questions as part of the construction pipeline.

3. Data: Question-Solution Augmentation

OpenMathInstruct-2 augments seed math questions and solutions in two stages, while applying syntactic filtering and LLM-based decontamination against evaluation benchmarks.

  • Pipeline: Question-Solution Augmentation consists of question augmentation followed by solution augmentation.These are the two stages of the OpenMathInstruct-2 construction pipeline.
  • Question augmentation: New questions are generated from MATH and GSM8K training questions using few-shot examples and nucleus-sampling variance.The process does not explicitly request greater difficulty or additional conditions.
  • Question augmentation: Generated questions are filtered for syntactic validity before potential contamination with evaluation benchmark test sets is checked.The filtering and contamination checks are part of question augmentation.
  • Decontamination: The decontamination process targets GSM8K, MATH, AMC 2023, and AIME 2024 test sets to reduce contamination from paraphrased evaluation questions.The method was adopted because n-gram overlap and embedding similarity alone can miss simple variations such as paraphrasing or translation.
  • Decontamination: LLM-based decontamination compares each synthesized question with the top-k similar test examples and removes it when any pair is judged a paraphrase.Embedding similarity retrieves candidates, and an advanced LLM performs the paraphrase judgment.
  • Decontamination: Using k=5 produces 10 LLM inference calls per generated question, with Llama3.1-405B-Instruct used for paraphrase detection.A popular Sentence Transformer model supplies the embeddings.

4. Results

OpenMathInstruct-2 improves mathematical reasoning across benchmark evaluations, with gains scaling as more training data is used, while the 70B model improves on only a subset of benchmarks.

  • Evaluation Setup: The models are evaluated on GSM8K, MATH, AMC 2023, AIME 2024, and Omni-MATH across difficulty levels from grade-school to advanced competition mathematics.Evaluation uses zero-shot greedy decoding and majority voting over 256 sampled solutions unless otherwise noted.
  • Data Scaling: Increasing SFT data size yields consistent MATH gains, with no saturation observed even at 14M examples.Even the 1M fair-downsampled dataset outperforms Llama3.1-8B-Instruct and NuminaMath-7B-CoT.
  • Final Results: The OpenMath2-Llama3.1-8B model outperforms or matches Llama3.1-8B-Instruct on all evaluated math reasoning benchmarks and surpasses NuminaMath-7B-CoT on all benchmarks.The model is finetuned on the full OpenMathInstruct-2 dataset.
  • Evaluation Caveat: Approximately 1.4% of Omni-MATH test questions were found in the training data because Omni-MATH was released after model training and was excluded from decontamination.This affects the contamination status of the Omni-MATH evaluation.
  • Final Results: The 8B model ranks second among the presented models, behind Qwen2.5-Math-7B-Instruct, whose base model is about 35% better than Llama3.1-8B-Base.Qwen2.5-Math-7B-Instruct was trained on more than one trillion synthetic math-reasoning tokens.

5. Related Work

Prior open-source math datasets vary in scale, diversity, licensing, and reasoning format, while OpenMathInstruct-2 combines open-weight synthesis with expanded construction and decontamination procedures.

  • Open Math Datasets: NuminaMath contains 860K competition-level problem-solution pairs, while Skywork-MathQA contains 2.5M question-solution pairs using multiple augmentation techniques.These datasets represent major recent efforts to expand open mathematical reasoning data.
  • Tool-Integrated Methods: Tool-integrated approaches such as Program of Thoughts combine natural-language reasoning with programming-language statements to solve mathematics problems.Subsequent datasets build on related programmatic reasoning concepts.
  • Contamination: Qwen2.5-Math’s contamination protocol is uncertain because n-gram matching may be weak at detecting paraphrases.The passage characterizes this as an uncertainty rather than a confirmed contamination result.
  • Prior Datasets: OpenMathInstruct-1 introduced 1.8M examples using code-interpreter solutions, while InfinityMATH provided 100K examples for programmatic mathematical reasoning.Both datasets are earlier open-source efforts using computational or programmatic solution strategies.
  • OpenMathInstruct-2: OpenMathInstruct-2 uses open-weight models, studies low-quality data and solution format, and applies LLM-based decontamination that detects rephrased test questions.These choices distinguish its construction from prior work and support release under a permissive license.

6. Conclusion

The paper addresses closed or restricted math-reasoning data by releasing reproducible resources and analyzing how synthesis choices affect SFT outcomes. Its findings emphasize concise formats, strong teachers, tolerance of some incorrect data, diverse questions, and careful decontamination.

  • Conclusion: Not all chain-of-thought formats are equally effective, and longer solutions are not necessarily better.The conclusion frames solution length as an important design consideration for SFT data.
  • Conclusion: Data generated by a strong teacher model outperforms equally sized data produced by a weaker student model.The comparison controls for SFT data size.
  • Conclusion: SFT remains robust to incorrect solutions, limiting the utility of extensive data filtering for math-reasoning datasets.The reported robustness applies to the presence of incorrect solutions during SFT.
  • Conclusion: Training on diverse questions is crucial for obtaining data-scaling gains.The conclusion identifies question diversity as a key condition for scaling benefits.
  • Conclusion: Proper decontamination is required so benchmark evaluations accurately represent model strengths.The dataset construction uses a decontamination pipeline designed to detect rephrased test questions.
  • Solution Format: Removing Llama special role tokens improves adherence to the proposed OpenMath CoT format, reducing Llama-format generations from about 57% to 0.1%.This result comes from prompting Llama3.1-405B-Instruct with few-shot examples.

A.2. Post-Processing

Post-processing removes or edits solutions using structural, length, and arithmetic criteria before constructing the training data.

  • Filtering Rules: Solutions with multiple boxed entries are removed during post-processing.The procedure also removes the prefix “My Solution:” and truncates solutions after the first boxed answer.
  • Filtering Rules: Incorrect arithmetic calculations are removed from solutions.Complex arithmetic may instead be split into step-by-step calculations to ease generation.
  • Length Filtering: Solutions longer than 1024 Llama3.1 tokens or shorter than 200 characters are removed.These are explicit length thresholds in the post-processing procedure.

A.3. Composition of OpenMathInstruct-2

OpenMathInstruct-2 adds about 592K newly synthesized questions and about 11M new question-solution pairs.

  • 592K new synthetically generated questions contribute about 11M new question-solution pairs to OpenMathInstruct-2.

A.4. Checkpoint Averaging

Checkpoint averaging consistently improves performance in the reported setup. Averaging the last four checkpoints produced a gain exceeding 2% in one ablation run versus using only the final checkpoint.

  • Checkpoint averaging produced consistent gains in the reported setup.
  • More than 2% improvement occurred when averaging the last 4 checkpoints in one ablation run.
  • The comparison used a final checkpoint averaged over the last 4 checkpoints versus only the last checkpoint.

B. Performance Comparison between Different Teacher Models

The comparison examines low-quality data from 8B-Base and 405B-Instruct teacher models using LLM-as-a-judge and reward-model labeling. Low-quality data was more prevalent for the weaker teacher, while SFT performance remained relatively stable across filtering strategies and was consistently higher with 405B-Instruct.

  • The study identified low-quality solutions using LLM-as-a-judge and reward-model labeling.
  • For the 8B-Base teacher, 128K samples were generated from MATH with correct final answers and a 1024-token solution limit.
  • 45% to 67% of data from the Llama3.1-8B-Base teacher was classified as low quality across filtering strategies.
  • More advanced teachers such as Llama3.1-405B-Instruct generally produce higher-quality data than Llama3.1-8B-Base.
  • SFT performance remained relatively stable across filtering strategies for both teacher models.
  • SFT performance was consistently higher when Llama3.1-405B-Instruct served as the teacher.

C.1. Minimum Majority Vote Ablation

The ablation uses majority voting as a proxy for ground-truth answers and removes questions with too few majority-vote solutions. It evaluates this threshold using 32 generated solutions per question after decontamination with the MATH validation subset.

  • Majority voting serves as a proxy for the ground-truth answer to synthetically generated questions.
  • Questions with fewer majority-vote solutions than the selected minimum threshold are removed from SFT data.
  • The ablation generates 32 solutions per question for an initial synthesized set after decontamination with the MATH validation subset.

C.2. Contaminated Examples Detected by LLMs

The decontamination pipeline identifies paraphrased questions that n-gram matching misses, while the dataset still contains questions similar but not equivalent to MATH test items.

  • Approximately 50K of 569K newly synthesized questions were filtered, reducing the total to 519K.The pipeline catches contamination missed by a simple n-gram baseline.
  • OpenMathInstruct-2 includes questions similar but not equivalent to MATH test questions.Examples are shown in Table 11.
  • The pipeline detects paraphrases that n-gram matching would miss.Table 10 provides examples of these detected paraphrases.

D. LLM Prompts

The appendix presents prompts for augmenting solutions and generating new math problems, illustrated with few-shot examples spanning algebra, functions, geometry, vectors, and arithmetic.

  • D.1. Solution Augmentation Prompt: The solution-augmentation prompt asks models to solve a new question after reviewing examples and place only the answer inside boxed delimiters.This instruction is illustrated alongside multiple worked examples.
  • D.1. Solution Augmentation Prompt: A placeholder question example is paired with a solution scaffold and substitution steps for solving a quadratic-parameter problem.The displayed example introduces the question, solution heading, and equations from two points.
  • D.1. Solution Augmentation Prompt: For algebra examples, the prompt substitutes given points into a parabola equation and eliminates variables to solve for c.The example derives -b + c = -12 and 3b + c = 8, then solves the resulting system.
  • D.1. Solution Augmentation Prompt: For function questions, the prompt applies the odd-function identity f(-x) = -f(x) twice to classify f(f(x)) as odd.The worked conclusion is boxed as “odd.”
  • D.1. Solution Augmentation Prompt: For geometry, the prompt converts edge-sum and surface-area constraints into the box diagonal, obtaining r = 10.It uses the fact that an inscribed box’s diagonal equals the sphere’s diameter.
  • D.1. Solution Augmentation Prompt: For vector questions, the prompt combines dot- and cross-product constraints into a linear system and solves for b = (-1, 3, 2).The example expands the cross product into three scalar equations before substitution.
  • D.2. Question-Solution Augmentation Prompts: Question-solution augmentation prompts instruct an LLM to create reasonable, solvable problems similar to given examples.GSM8K-style demonstrations cover totals, repeated additions, purchases, losses, and multi-step scheduling.
  • D.2. Question-Solution Augmentation Prompts: The arithmetic augmentation examples preserve the original problem structure while changing entities, quantities, and contexts.Examples include shopping, collecting objects, and planning work with breaks.
Loading 2410.01560v2…