Source-linked AI summary
MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuning
Xiang Yue, Xingwei Qu, Ge Zhang, Yao Fu, Wenhao Huang, Huan Sun, Yu Su, Wenhu Chen
TL;DR
Open-source LLMs lag closed-source systems on mathematical reasoning, while existing training approaches face computational or generalization limits. MAmmoTH addresses this with MathInstruct, a broad dataset combining CoT and PoT rationales, and reports strong gains across mathematical reasoning benchmarks. The models are particularly competitive on MATH, including a 35.2% vs 10.7% comparison against WizardMath for MAmmoTH-7B.
Problem
Open-source LLMs lag behind closed-source models on mathematical reasoning benchmarks, while existing approaches are costly or limited in generalization.
Method
MAmmoTH fine-tunes open-source LLMs on MathInstruct, a 260K-sample dataset spanning math fields and complexity levels with hybrid CoT and PoT rationales.
Results
MAmmoTH substantially improves open-source performance across in-domain and out-of-domain datasets, with MAmmoTH-7B achieving 35.2% on MATH versus WizardMath’s 10.7%.
Takeaways & Limitations
Broad math coverage and hybrid CoT-PoT training are associated with MAmmoTH’s strong mathematical reasoning performance across model scales and tasks.
Takeaways & Limitations
MAmmoTH may struggle with advanced mathematical fields outside its primary expertise and has limited theorem-proving capability because it was not trained on proof-type problems.
Abstract
from arXiv · showhide
We introduce MAmmoTH, a series of open-source large language models (LLMs) specifically tailored for general math problem-solving. The MAmmoTH models are trained on MathInstruct, our meticulously curated instruction tuning dataset. MathInstruct is compiled from 13 math datasets with intermediate rationales, six of which have rationales newly curated by us. It presents a unique hybrid of chain-of-thought (CoT) and program-of-thought (PoT) rationales, and also ensures extensive coverage of diverse fields in math. The hybrid of CoT and PoT not only unleashes the potential of tool use but also allows different thought processes for different math problems. As a result, the MAmmoTH series substantially outperform existing open-source models on nine mathematical reasoning datasets across all scales with an average accuracy gain between 16% and 32%. Remarkably, our MAmmoTH-7B model reaches 33% on MATH (a competition-level dataset), which exceeds the best open-source 7B model (WizardMath) by 23%, and the MAmmoTH-34B model achieves 44% accuracy on MATH, even surpassing GPT-4's CoT result. Our work underscores the importance of diverse problem coverage and the use of hybrid rationales in developing superior math generalist models.
1 INTRODUCTION
MAmmoTH addresses the gap between closed- and open-source mathematical reasoning by combining broad math coverage with hybrid CoT and PoT instruction tuning. Evaluations across in-domain and out-of-domain datasets show substantial gains over existing open-source models.
- Closed-source LLMs dominate GSM8K and MATH, while open-source models lag substantially across mathematical reasoning benchmarks.
- Existing approaches either incur high computation costs through continued pre-training or improve in-domain performance without generalizing broadly beyond their fine-tuning data.
- MAmmoTH introduces MathInstruct, combining broad coverage of math fields and complexity levels with hybrid CoT and PoT rationales.
- MathInstruct aggregates existing and newly curated math rationale datasets, while MAmmoTH models are fine-tuned across scales from 7B to 70B.
- 35.2% vs 10.7% on MATH: MAmmoTH-7B beats WizardMath by 3.5x, while MAmmoTH-Coder-34B surpasses GPT-4’s CoT result.
- MAmmoTH is evaluated on four in-domain and five out-of-domain datasets to assess mathematical reasoning and generalization.
2 OUR APPROACH
The approach builds MathInstruct as a diverse math instruction-tuning dataset with broad field and difficulty coverage and hybrid CoT and PoT rationales. Models can use either reasoning style, while preliminary evaluation identifies complementary strengths and limitations between them.
- The dataset targets broad coverage of mathematical fields and complexity levels to foster model versatility.Candidate source datasets include GSM8K, MATH, AQuA, Camel, and TheoremQA.
- MathInstruct combines chain-of-thought and program-of-thought rationales to support varied mathematical problem-solving approaches.GPT-4 is used to supplement program rationales because existing datasets provide limited program rationales.
- MathInstruct contains 260K instruction-response pairs spanning arithmetic, algebra, probability, calculus, geometry, language, and difficulty levels.
- All subsets are standardized into an Alpaca-like instruction format so fine-tuned models can process data consistently across original dataset formats.
- Models default to CoT solutions but switch to PoT when prompted with “Let’s write a program to solve the problem” after the question.
- PoT generally outperforms CoT on open-form GSM8K and MATH questions but struggles with commonsense reasoning, formal logic, and abstract algebra without built-in APIs.
3 EXPERIMENTS
MAmmoTH is evaluated across diverse in-domain and out-of-domain mathematical reasoning datasets, with results showing stronger generalization than existing open-source models. Ablations attribute these gains to diverse data sources and complementary CoT–PoT training and decoding.
- Evaluation Setup: Nine evaluation datasets span in-domain and out-of-domain problems across elementary, high-school, and college mathematics, including formal logic and commonsense reasoning.The evaluation includes four in-domain datasets and five out-of-domain datasets covering varied mathematical fields.
- Main Results: MAmmoTH and MAmmoTH-Coder outperform state-of-the-art models across scales, with larger gains on out-of-domain than in-domain datasets.MAmmoTH-Coder-34B and MAmmoTH-70B surpass closed-source LLMs on several datasets.
- Main Results: MAmmoTH’s gain over WizardMath on MATH can exceed 25% at different model scales, especially on more complex problems.WizardMath is highly competitive on GSM8K and MATH because its training is rooted in those datasets, whereas MAmmoTH improves broadly.
- Main Results: MAmmoTH-7B improves WizardMath-7B’s CoT performance on MMLU-Math by 9%, despite many questions extending beyond the training subjects.This result supports generalization beyond the subjects directly covered during training.
- Ablation Study on Data Source: CoT-only fine-tuning raises nine-dataset accuracy from 27% to 32%, PoT-only fine-tuning raises it to 41%, and hybrid training reaches 47.9%.CoT supports generic language-based reasoning, while PoT supports precise computation through Python APIs; together they cover complementary problem types.
- Ablation Study on Data Source: Removing the six newly curated datasets lowers overall performance by 9%, showing that the curated data materially improves results across many datasets.The comparison removes the newly curated rationales from MathInstruct and trains on the remaining existing data.
4 CONCLUSION
The paper presents math instruction tuning for open-source LLMs using broad problem coverage and hybrid CoT/PoT training. Its 260K-sample dataset supports affordable fine-tuning and strong performance across model scales.
- 260K samples make the instruction-tuning dataset highly affordable for fine-tuning, including for academic labs.
- Broad coverage across math fields and complexity levels is identified as a major benefit of the models.
- Hybrid CoT and PoT training is identified as another major source of model benefit.
- The approach is presented as a way to activate mathematical reasoning capabilities in open-source LLMs and support future specialized-domain studies.
A.1 MATHEMATICAL REASONING DATASETS
Mathematical reasoning datasets progressed from synthetic basic arithmetic problems toward more difficult math word problems addressing realistic scenarios.
- Early mathematical reasoning research focused mainly on synthetic basic math problems such as AddSub and related arithmetic datasets.
- Later datasets introduced more difficult math word problems aimed at addressing realistic mathematical scenarios.
A.2 REASONING WITH LARGE LANGUAGE MODELS
Large language models have demonstrated strong reasoning capabilities through Chain-of-Thought prompting and related methods. Research has developed multiple approaches for using LLMs to solve challenging tasks.
- Chain-of-Thought prompting has demonstrated strong reasoning capabilities in large language models.
- CoT has been reported to surpass human performance on challenging BIG-Bench tasks.
- Several studies propose different approaches for utilizing LLMs to solve reasoning tasks.
A.3 INSTRUCTION TUNING IN LANGUAGE MODELS
Instruction tuning is part of a broader effort to align language models with useful objectives and human preferences. It has been used to activate capabilities and improve general-purpose instruction following.
- Instruction tuning is designed to align language models with more useful objectives and human preferences.
- The instruction-tuning step is viewed as a major way to activate certain capabilities in LLMs.
- Earlier instruction-tuning work mainly targeted general-purpose instruction-following abilities.
B CASE STUDY
The case study compares CoT and PoT on mathematical problems, showing that both can solve some tasks while PoT can make solutions less tedious.
- PoT can simplify solutions by using external tools and Python packages.
- Formal logic questions may be better handled with CoT than PoT.
- CoT and PoT both solve the first example, but CoT requires a very tedious derivation.The paper characterizes the CoT solution as slow and unstable.
PoT
The PoT case studies illustrate both its advantages for computational problems and its limitations on abstract reasoning tasks, motivating comparison with CoT.
- PoT generates the correct solution while CoT fails in the second and third examples.
- The examples include a probability question involving products of integers selected from two sets.
- One worked expression example evaluates endpoint cases and concludes that none of the listed answer choices applies.
- Formal logic questions are difficult for PoT but can be handled by CoT.
C LIMITATIONS
The paper identifies limitations in MAmmoTH’s mathematical scope, theorem-proving capability, safety, and reasoning robustness.
- MAmmoTH may struggle with mathematical analysis, complex analysis, graph theory, and numerical analysis outside its primary expertise.The authors state that the models are not suitable for more complex problems in these fields.
- The models have limited theorem-proving capability because they were not trained on proof-type problems.
- MAmmoTH may generate harmful, offensive, or biased content when asked questions beyond mathematics.The paper recommends application-specific safety testing and tuning before deployment.
- Future work includes improving MAmmoTH’s robustness and generalizability in mathematical reasoning.The paper mentions synthetic data intervention as one possible direction.