PaperlayerView exact revision on arXiv

Source-linked AI summary

OpenMathInstruct-1: A 1.8 Million Math Instruction Tuning Dataset

Shubham Toshniwal, Ivan Moshkov, Sean Narenthiran, Daria Gitman, Fei Jia, Igor Gitman

arXiv:2402.10176v2cs.CLcs.AIcs.LG

TL;DR

Existing math instruction-tuning pipelines rely heavily on closed-source models despite licensing and capability gaps that limit open-source data generation. This work uses Mixtral with targeted prompting and masked solutions to build OpenMathInstruct-1, a 1.8M-pair dataset, and reports competitive performance from models trained on it. The dataset, models, and code are released under a commercially permissive license.

  • Problem

    Math instruction-tuning datasets often rely on closed-source LLM outputs, creating licensing restrictions while open-source models lag in mathematical solution quality.

  • Method

    The authors synthesize code-interpreter solutions for GSM8K and MATH with Mixtral using few-shot, subject-specific, masked-text, and brute-force prompting strategies.

  • Results

    OpenMathInstruct-1 contains 1.8M problem-solution pairs, reaches 99.9% GSM8K and 93% MATH training coverage, and supports competitive fine-tuned models.

  • Takeaways & Limitations

    The released dataset, models, and code provide a commercially permissive alternative to publicly available GPT-distilled math instruction-tuning resources.

  • Takeaways & Limitations

    The generated corpus can contain semantically noisy solutions with correct answers but flawed reasoning, and benchmark gains may not transfer to perturbed or related tasks.

Abstract

from arXiv · show

Recent work has shown the immense potential of synthetically generated datasets for training large language models (LLMs), especially for acquiring targeted skills. Current large-scale math instruction tuning datasets such as MetaMathQA (Yu et al., 2024) and MAmmoTH (Yue et al., 2024) are constructed using outputs from closed-source LLMs with commercially restrictive licenses. A key reason limiting the use of open-source LLMs in these data generation pipelines has been the wide gap between the mathematical skills of the best closed-source LLMs, such as GPT-4, and the best open-source LLMs. Building on the recent progress in open-source LLMs, our proposed prompting novelty, and some brute-force scaling, we construct OpenMathInstruct-1, a math instruction tuning dataset with 1.8M problem-solution pairs. The dataset is constructed by synthesizing code-interpreter solutions for GSM8K and MATH, two popular math reasoning benchmarks, using the recently released and permissively licensed Mixtral model. Our best model, OpenMath-CodeLlama-70B, trained on a subset of OpenMathInstruct-1, achieves a score of 84.6% on GSM8K and 50.7% on MATH, which is competitive with the best gpt-distilled models. We release our code, models, and the OpenMathInstruct-1 dataset under a commercially permissive license.

1 Introduction

OpenMathInstruct-1 addresses the limitations of GPT-4-dependent math-data synthesis by using Mixtral, targeted prompting, and scaling to generate permissively licensed training data. The resulting dataset achieves high training coverage and supports models competitive with GPT-distilled systems.

  • Closed-source-model distillation is limited by legal restrictions and higher data-generation costs, motivating open-source alternatives.
  • Mixtral’s training-set coverage approaches GPT-4 on GSM8K with nearly 8x more samples but remains lower on MATH even with 12x more samples.Training-set coverage measures whether any sampled solution reaches the ground-truth answer.
  • Targeted subject-specific prompting and masked text solutions increase coverage by steering Mixtral toward dataset-specific mathematical content and discouraging copied answers.Masked solutions replace intermediate-computation numbers with symbols while retaining question-given amounts.
  • OpenMathInstruct-1 contains 1.8M problem-solution pairs, is at least four times larger than prior datasets, and uses a commercially permissive license.The collection includes synthesized solutions for GSM8K and MATH and was released with models and code.
  • The dataset reaches 99.9% training coverage on GSM8K and 93% on MATH, while OpenMath-CodeLlama-70B scores 84.6% on GSM8K and 50.7% on MATH.These scores are reported as competitive with current GPT-distilled models.
  • Code-preferential data selection improves performance on the MATH subset in ablation experiments.

3 Experimental Setup

The experiments train and evaluate math-reasoning models under specified optimization, checkpointing, and decoding settings, comparing OpenMath-finetuned models with gpt-distilled counterparts across several benchmarks.

  • Training Details: Models of size 34B or smaller are trained for four epochs using global batch size 128 and AdamW optimization.The training setup uses weight decay and dropout of 0.1.
  • Training Details: Final model runs average saved checkpoints, while ablation experiments save one checkpoint per epoch and final runs save two.
  • Evaluation: Evaluation covers GSM8K, MATH, GSM-Hard, SVAMP, TabMWP, ASDiv, and MAWPS in the zero-shot setting.A 1K-example validation set is created from GSM8K and MATH training data for ablations and hyperparameter selection.
  • Evaluation: The evaluation uses greedy decoding and self-consistency or majority voting, with Table 3 comparing OpenMath-finetuned and gpt-distilled models.Table 3 reports results across seven mathematical reasoning tasks and identifies best results for the relevant decoding settings.

4 Results

The experiments show that fair sampling, masked prompting, larger fine-tuning datasets, and diverse solution selection improve OpenMath model performance across GSM8K and MATH. OpenMath-finetuned models are competitive with or outperform several GPT-distilled alternatives.

  • OpenMath-Mistral-7B is competitive with GPT-distilled models, second-best to WizardMath on GSM8K and 0.1% behind ToRA on MATH.It outperforms MetaMath and MAmmoTH when controlling for the base fine-tuned model.
  • With self-consistency decoding, OpenMath models outperform ToRA on both MATH and GSM8K.With greedy decoding, they show a meaningful MATH drop but are equal or better on GSM8K.
  • Fair vs. Naive Downsampling: Fair downsampling outperforms naive downsampling, with the largest performance gap on MATH.The comparison uses a 128K-instance corpus combining equal-sized GSM8K and MATH subsets.
  • Default vs. Masked Prompting: Masked prompting outperforms default prompting on both GSM8K and MATH, translating higher training-set coverage into better fine-tuning performance.The comparison uses fair-sampled 128K-instance subsets.
  • Fine-tuning Dataset Size: Performance increases on both GSM8K and MATH as the fine-tuning dataset grows from 128K to 512K instances.The authors report no benefit from training for more steps, attributing the gain to increased data size.
  • Code-Preferential Subsets: Code-preferential selection improves greedy-decoding performance but can reduce self-consistency performance by lowering solution diversity.The authors therefore choose the Any-Code subset, which is smaller than the Majority-Code subset.

5 Analysis

Analysis finds strong variation across MATH difficulty levels and solution formats, with geometry and hardest problems remaining difficult. Code-based solutions improve performance overall, but most code-solution errors are still reasoning errors.

  • Performance by Subjects and Levels: The model scores 72.4% on Level 1 MATH problems but 16.3% on Level 5 problems.Performance decreases monotonically as problem hardness increases.
  • Performance by Subjects and Levels: Geometry is the model’s weakest MATH subject, which the authors attribute to the base models’ lack of multi-modality.
  • Error Analysis: Code-based answering performs 13.3% better than non-code answering in absolute terms.The analysis identifies arithmetic errors in text-based solutions that code preference could avoid.
  • Error Analysis: About 74% of errors in code-based solutions are reasoning errors, while 26% are execution-related.

7 Conclusion

OpenMathInstruct-1 is a large, permissively licensed math instruction-tuning dataset whose synthesized solutions support models with performance comparable to or better than gpt-distilled counterparts. The authors also identify quality, robustness, reproducibility, and statistical-significance limitations.

  • Conclusion: OpenMathInstruct-1 contains 1.8M problem-solution pairs, is at least four times larger than previous work, and achieves 99.9% GSM8K and 93% MATH training-set coverage.The dataset, models, code, and 6.6M incorrect sampled solutions are publicly released under a commercially permissive license.
  • Conclusion: Fine-tuning experiments show models achieving performance comparable to or better than their gpt-distilled counterparts.
  • Limitations: Manual analysis found solutions that reach correct answers through flawed reasoning, so the synthesized dataset has no guarantee of solution quality.
  • Limitations: Improved performance on in-domain math benchmarks may not transfer to related tasks, and the GSM-Hard drop suggests limited robustness to input perturbations.
  • Reproducibility and evaluation: Fine-tuning runs lacked multiple repetitions because of their cost, limiting the reported assessment of experimental variability.

10. Broader Impacts

The paper briefly addresses societal impacts by noting that imperfect datasets and models could pose risks in educational applications. It also frames the work as open-source mathematical reasoning research.

  • Discussion: The paper states that it briefly discusses societal impacts in its Limitations section.

11. Safeguards

The paper reports open access and licensing information for its assets, while stating that it does not provide dedicated safeguards because the dataset concerns mathematical reasoning.

  • Safeguards: The paper reports no safeguards for responsible release, justifying this by characterizing the dataset as related to mathematical reasoning.
  • Licensing: The paper states that Mixtral models are used under a permissive license and that original assets are credited.
  • Asset documentation: The paper reports that new assets are documented and provided alongside the release.It states that dataset-generation details and the commercially permissive license are communicated.

A Data Statistics

OpenMathInstruct-1 uses code-interpreter solutions combining natural-language reasoning with executable code, and its solution frequency differs substantially between GSM8K and MATH. Most solutions contain zero or one code block, while MATH problems generally receive fewer valid solutions.

  • Solution format: 16.4% of OpenMathInstruct-1 solutions have zero code blocks, 81.7% have one, and 2% have two or more.The code-interpreter format supports flexible text reasoning alongside precision from code-based reasoning.
  • Solution frequency: 57.4% of GSM8K problems have more than 128 valid solutions among 256 samples, compared with 3.1% of MATH problems having more than 448 among 896 samples.For MATH, 19% of problems have at most 10 valid solutions out of 896 samples, indicating a much sparser frequency distribution.

B.2 Sample Solutions

The section presents representative OpenMathInstruct-1 solutions, highlighting code-and-language reasoning, shortcut behavior from reference-conditioned prompting, extraneous text, and semantically flawed reasoning.

  • Figure 10 combines initial reasoning in code with final reasoning steps in natural language.
  • Reference-conditioned prompting can produce shortcut solutions that copy values from the reference solution into generated code.The example motivates masked text solutions in the prompt.
  • Some generated solutions continue with coherent but unrelated text after answering the problem.Post-processing trims text beyond the solution line containing the answer.
  • Some solutions reach the right answer through flawed reasoning, making them difficult to detect with simple syntactic filters.The paper leaves semantic filtering for future work, potentially using models such as GPT-4 to grade solutions.

B.3 Error Analysis of Solutions Generated by Fine-tuned Model

The error analysis catalogs failures in fine-tuned solutions, including answer-copying errors, timeouts, arithmetic mistakes, repeated incorrect code, and execution errors.

  • The model can reason correctly in code but generate a new answer when copying the code output.
  • Correct but inefficient code can exceed the stipulated execution time and produce a timeout error.
  • Arithmetic failures can occur after correct reasoning, supporting the preference for code-based solutions.The example specifically concerns multiplication.
  • When the model generates wrong code, it may repeat the same block until reaching the maximum execution constraint.The dataset lacks examples of recovering from such errors, unlike GPT-4.
  • An execution error can still be followed by an incorrect answer.

B.4 Instructions for Few-shot Data Generation

This section documents the few-shot generation instructions and examples, including solutions requiring trimming after the model generates text beyond the answer.

  • Table 13 organizes the instructions used for the different generation tasks.
  • The generation examples include a solution requiring trimming after the answer.The accompanying example uses a lassi-scaling problem.

B.5 Masked Text Solution Generation

Masked text solutions are generated through candidate sampling, filtering, and ranking designed to hide intermediate computation and the final answer.

  • The pipeline generates eight candidate masked solutions before filtering and ranking them.
  • Candidates with very different lengths or visible final answers are filtered out to preserve masking of the answer.
  • The ranking heuristic favors solutions with fewer numbers, masking more intermediate computation.

B.6 Few-shot prompts used for GSM8K and MATH

The few-shot prompts for GSM8K and MATH provide worked question-solution examples and require answers in boxed form. Examples illustrate code-interpreter reasoning, while figures document occasional reasoning, calculation, execution, and recovery failures.

  • Observed failure cases: Failure cases include flawed reasoning, disagreement with code output, inefficient recursion, calculation errors, execution errors, and failure to recover across repeated code blocks.
  • MATH examples: The examples include code-based solutions for a parabola, function parity, and fraction comparison, alongside additional geometry and sequence questions.
  • GSM8K examples: GSM8K examples use code-interpreter solutions to compute answers for arithmetic, scheduling, deposits, discounts, operating expenses, and related word problems.
  • Prompt format: Few-shot prompts present example questions and solutions before a new question, requiring the final answer inside \boxed{}.
  • MATH examples: MATH examples use symbolic computation and worked reasoning for algebra, functions, fractions, geometry, and combinatorics problems.
Loading 2402.10176v2…