MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuning

Introduction

• Prior continued pre-training is computationally expensive, while dataset-specific fine-tuning can fail to generalize, with RFT and WizardMath gaining 30%+ on GSM8K but losing up to 10% on OOD datasets.The introduction frames these limitations against a gap between closed-source models such as GPT-4, PaLM-2, and Claude 2 and open-source models such as Llama, Falcon, and OPT.Source: §1 · p. 2
• CoT offers broad subject coverage but struggles with precise computation and complex procedures, whereas PoT offloads computation to Python but struggles with abstract reasoning without built-in APIs.The introduction gives quadratic roots and matrix eigenvalues as computational examples, and commonsense reasoning, formal logic, and abstract algebra as abstract cases.Source: §1 · p. 2
• MathInstruct combines broad coverage of math fields and complexity levels with hybrid CoT and PoT rationales, drawing on seven existing and six newly curated datasets.The authors fine-tune Llama models ranging from 7B to 70B to create the MAmmoTH series.Source: §1 · p. 2
• 35.2% vs 10.7% on MATH: MAmmoTH-7B beats WizardMath by 3.5x, while 34B MAmmoTH-Coder exceeds GPT-4’s chain-of-thought result.The evaluation spans four in-domain and five out-of-domain datasets, and the authors report better out-of-domain generalization than existing methods.Source: §1 · p. 2
• The authors report training and evaluating over 50 models and baselines from 7B to 70B to study data sources and input-output formats, with substantial accuracy gains over existing open-source models.MathInstruct is presented as the data contribution, while MAmmoTH and MAmmoTH-Coder are the modeling contribution.Source: §1 · p. 2

2.1 BACKGROUND

• The approach aims to optimize both incremental CoT scratchpad reasoning and PoT programs executed by external tools, extending PoT beyond its prior proprietary-model focus (Zhou et al., 2023a).CoT supports stepwise solutions, while PoT offloads calculations to tools such as Python.Source: §2.1 · p. 3

2.2 CURATING A DIVERSE AND HYBRID INSTRUCTION TUNING DATASET

• To broaden field coverage, the authors select established datasets and use GPT-4 to synthesize CoT rationales for college-level topics underrepresented in existing data.The topics include abstract algebra and formal logic; new question-CoT pairs are created through Self-Instruct.Source: §2.2 · p. 3
• GPT-4 supplements PoT rationales for MATH, AQuA, GSM8K, and TheoremQA, and generated programs are retained only when execution matches human-annotated answers.This addresses the limited program rationales in existing datasets and balances them with CoT rationales.Source: §2.2 · p. 4
• MathInstruct contains 260K instruction-response pairs spanning core mathematical fields, language styles, and difficulty levels, with both CoT and PoT rationales.Its coverage includes arithmetic, algebra, probability, calculus, and geometry.Source: §2.2 · p. 4

2.3 TRAINING SETUP

• The authors fine-tune Llama-2 and Code Llama at 7B, 13B, 34B, and 70B scales using MathInstruct standardized to an Alpaca-like format.Training uses three epochs, batch size 128, and a cosine scheduler with 3% warm-up; DeepSpeed ZeRO-3 is used for 34B and 70B models.Source: §2.3 · p. 4

2.4 EVALUATION SETUP

• Models default to CoT, while the trigger phrase “Let’s write a program to solve the problem” switches them to PoT.The hybrid decoding strategy tries PoT first and falls back to CoT if the program is not executable.Source: §2.4 · p. 4
• Preliminary evaluation finds PoT generally stronger than CoT on open-form GSM8K and MATH questions, but weaker on abstract reasoning without built-in APIs.The authors report that falling back to CoT when PoT is not executable significantly enhances overall performance.Source: §2.4 · p. 4

3.1 EVALUATION DATASETS

• Evaluations span four in-domain and five out-of-domain datasets, covering elementary through college math, varied fields, formal logic, commonsense, and open-form and multiple-choice questions.The datasets are selected to assess generalization to unfamiliar situations and different mathematical fields.Source: §3.1 · p. 4

3.2 BASELINES

• Baselines include closed-source LLMs, Llama models, code models, STEM-pretrained models, instruction-tuned models, and dataset-specific models, with prompting and shot counts varying by dataset and model.Most baselines use CoT, code models use PoT, and MAmmoTH models are evaluated zero-shot; other datasets use the specified zero-, five-, or eight-shot setups.Source: §3.2 · p. 5

3.3 MAIN RESULTS

• MAmmoTH and MAmmoTH-Coder outperform state-of-the-art models at different scales, with larger gains on out-of-domain datasets and some results surpassing closed-source LLMs.The in-domain and out-of-domain results are reported in Tables 3 and 4, respectively.Source: §3.3 · p. 5, Table 3 · p. 6, Table 4 · p. 7
• MAmmoTH’s gains over WizardMath on MATH can exceed 25% across scales, while MAmmoTH-7B improves WizardMath-7B’s MMLU-Math CoT performance by 9%.MAmmoTH-70B ties Platypus on MMLU-Math.Source: §3.3 · pp. 7–8
• Code-Llama-based MAmmoTH-Coder is consistently stronger than Llama-2-based MAmmoTH, especially out of domain, and its 34B model exceeds MAmmoTH-70B’s average OOD performance.The gap between MAmmoTH and MAmmoTH-Coder can reach 5%; the authors attribute the advantage to Code-Llama’s continuous code training, which enhances PoT and general reasoning.Source: §3.3 · p. 8

3.4 ABLATION STUDY ON DATA SOURCE

• On the nine-dataset average, CoT fine-tuning raises accuracy from 27% to 32%, PoT from 27% to 41%, and hybrid training reaches 47.9%.CoT helps with language-based abstract multiple-choice reasoning, while PoT enables Python API use for complex computations; hybrid training combines these strengths. Figure 2 also reports hybrid training as the best-performing model among the investigated setups.Source: §3.4 · p. 8, Figure 2 · p. 8
• Adding diverse MathInstruct subsets improves generalization beyond GSM8K-only training, and the six newly curated datasets increase overall performance by 9%.The subset analysis examines GSM8K, MATH, Camel, AQuA, and NumGLUE; the authors recommend diverse data collection rather than focusing on specific types.Source: §3.4 · p. 9
• Hybrid decoding improves performance on every test set by trying PoT first and switching to CoT when PoT fails to generate an executable query.The experiment compares CoT, PoT, and hybrid decoding.Source: §3.4 · p. 9

• The proposed approach outperforms SoTA models across scales by a huge margin, according to the paper’s comprehensive study.Source: §4 · p. 9
• The authors attribute model benefits to broad coverage of math fields and complexity levels, combined with hybrid CoT and PoT training.Source: §4 · p. 9
• The 260K-sample instruction-tuning dataset makes fine-tuning highly affordable for academic labs.Source: §4 · p. 9
• The work aims to pave the way for future studies that activate LLMs’ core capabilities in specialized domains.Source: §4 · p. 9

A.1 MATHEMATICAL REASONING DATASETS

• Early mathematical-reasoning datasets focused on synthetic basic arithmetic, including AddSub and other arithmetic reasoning datasets (Koncel-Kedziorski et al., 2015; Roy & Roth, 2015; Patel et al., 2021).Source: §A.1 · p. 16
• Later datasets targeted more difficult, realistic math word problems (Cobbe et al., 2021; Amini et al., 2019; Ling et al., 2017; Hendrycks et al., 2021b).Source: §A.1 · p. 16
• NumGLUE (Mishra et al., 2022b) and LiLA (Mishra et al., 2022a) compiled existing work into more diversified dataset collections.Source: §A.1 · p. 16
• Despite this diversification, these datasets mostly focus on grade-school math problems.Source: §A.1 · p. 16
• MMLU (Hendrycks et al., 2021a) includes college math problems in its evaluation suite to test LLMs on more complex problems.Source: §A.1 · p. 16
• More recent work targets more challenging college-level science and math problems (Chen et al., 2023; Wang et al., 2023e).Source: §A.1 · p. 16
• MAmmoTH’s instruction-tuning dataset builds on existing work by collecting math problems from different subfields.Source: §A.1 · p. 16

A.2 REASONING WITH LARGE LANGUAGE MODELS

• CoT prompting enables LLMs to reason through intermediate steps, and it surpassed human performance on challenging BIG-Bench tasks (Suzgun et al., 2022).Source: §A.2 · p. 16
• ReAct uses external tools such as search engines to enhance LLM reasoning (Yao et al., 2023), while PoT uses programs as thought processes (Chen et al., 2022).Source: §A.2 · p. 16
• Self-critic (Gou et al., 2023) and self-eval use self-evaluation to make generated programs more robust.Source: §A.2 · p. 16
• Plan-and-solve provides more detailed planning instructions to help LLMs form high-level reasoning plans, and these methods improve on PoT (Wang et al., 2023c).Source: §A.2 · p. 16

A.3 INSTRUCTION TUNING IN LANGUAGE MODELS

• Instruction tuning is intended to align language models with useful objectives and human preferences and activate their ability to respond to instructions.Earlier work mainly targeted general-purpose instruction following.Source: §A.3 · p. 16
• CrossFit, NaturalInstruction (Wang et al., 2022b), FLAN (Wei et al., 2022a), and T0 (Sanh et al., 2022) were among early efforts to study instruction-tuned generalization.These approaches mainly used human-annotated datasets; FLAN-v2 (Chung et al., 2022; Longpre et al., 2023) examined scaling instruction datasets and its impact on performance.Source: §A.3 · p. 16
• Recent work uses synthetic instruction-following data distilled from GPT-3/4 to align open-source LLMs (Wang et al., 2023h; Xu et al., 2023; Peng et al., 2023; Zhou et al., 2023b; Wang et al., 2023g).The closest effort to this work is Platypus, which uses a domain-specialized dataset to build a small-scale instruction-following dataset for enhancing reasoning.Source: §A.3 · p. 16

B CASE STUDY

• PoT uses external tools and Python packages to greatly simplify solutions in the second and third examples.For the first example, both PoT and CoT solve the problem, but CoT’s solution is tedious, slow, and unstable.Source: B CASE STUDY · p. 17, Figure 3 · p. 17
• CoT is a better choice for questions, especially formal logic, that are not easily handled by programs.Source: B CASE STUDY · p. 17

PoT

• In Examples 2 and 3, PoT generates correct solutions while CoT fails.Source: Figure 4 · p. 17, Figure 5 · p. 18
• Some questions, such as formal logic, are hard for PoT to solve but can be handled by CoT.Source: Figure 6 · p. 18

C LIMITATIONS

• MAmmoTH may struggle with advanced fields such as mathematical analysis, complex analysis, graph theory, and numerical analysis, and lacks theorem-proving capability because it was not trained on proof-type problems.The authors propose expanding coverage to more fields and theorem-proving problems.Source: C LIMITATIONS · p. 19
• MAmmoTH may generate harmful, offensive, or biased content beyond math, could be misused, and may contain inappropriate questions that escaped data curation.Developers should conduct safety testing and tuning tailored to their applications before deployment.Source: C LIMITATIONS · p. 19
• Future work could improve mathematical-reasoning robustness against sycophancy and the Clever Hans effect, including through synthetic data interventions (Wang et al., 2023b; Wei et al., 2023).These effects describe models failing to maintain truthful solutions when users challenge them with absurdly invalid arguments and critiques.Source: C LIMITATIONS · p. 19