PaperlayerView exact revision on arXiv

Source-linked AI summary

MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Models

Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T. Kwok, Zhenguo Li, Adrian Weller, Weiyang Liu

arXiv:2309.12284v4cs.CLcs.AI

TL;DR

Open-source LLMs remain limited on mathematical problems requiring complex reasoning. MetaMath bootstraps diverse questions into MetaMathQA and fine-tunes LLaMA-2 models on it, achieving strong results on GSM8K and MATH while remaining challenged by longer questions.

  • Problem

    Open-source LLMs remain unsatisfactory on mathematical problem-solving because these tasks require complex reasoning procedures.

  • Method

    MetaMath rewrites mathematical questions from multiple perspectives to build MetaMathQA, then fine-tunes LLaMA-2 models on the resulting dataset.

  • Results

    MetaMath outperforms existing open-source LLMs across GSM8K and MATH, including 66.5% on GSM8K and 19.8% on MATH for MetaMath-7B.

  • Takeaways & Limitations

    Question diversity is an important factor in MetaMathQA, with bootstrapped data supporting continued accuracy gains beyond the saturation of answer augmentation.

  • Takeaways & Limitations

    MetaMath and related methods struggle with longer questions, although MetaMath remains superior across the examined question-length subsets.

Abstract

from arXiv · show

Large language models (LLMs) have pushed the limits of natural language understanding and exhibited excellent problem-solving ability. Despite the great success, most existing open-source LLMs (e.g., LLaMA-2) are still far away from satisfactory for solving mathematical problem due to the complex reasoning procedures. To bridge this gap, we propose MetaMath, a fine-tuned language model that specializes in mathematical reasoning. Specifically, we start by bootstrapping mathematical questions by rewriting the question from multiple perspectives without extra knowledge, which results in a new dataset called MetaMathQA. Then we fine-tune the LLaMA-2 models on MetaMathQA. Experimental results on two popular benchmarks (i.e., GSM8K and MATH) for mathematical reasoning demonstrate that MetaMath outperforms a suite of open-source LLMs by a significant margin. Our MetaMath-7B model achieves 66.4% on GSM8K and 19.4% on MATH, exceeding the state-of-the-art models of the same size by 11.5% and 8.7%. Particularly, MetaMath-70B achieves an accuracy of 82.3% on GSM8K, slightly better than GPT-3.5-Turbo. We release all the MetaMathQA dataset, the MetaMath models with different model sizes and the training code for public use.

1 INTRODUCTION

MetaMath addresses the difficulty of mathematical multi-step reasoning in open-source LLMs by bootstrapping diverse training questions. The resulting MetaMathQA dataset and models improve benchmark performance, with question diversity linked to continued gains.

  • Open-source LLMs remain difficult to train for mathematical problem-solving because these tasks require complex symbolic, multi-step reasoning.
  • MetaMath bootstraps available mathematical questions through forward, backward, and rephrased reasoning directions without requiring additional knowledge.
  • MetaMathQA is substantially more diverse than the original dataset, and its test accuracy continues increasing while non-bootstrapped data rapidly saturates.
  • Question bootstrapping uses reasoning diversity as a heuristic proxy for constructing training data that supports generalization, unlike methods optimizing empirical risk.
  • MetaMath-7B reaches 66.5% on GSM8K and 19.8% on MATH, improving over the previous best open-source model by 11.5% and 8.7%, respectively.

2 RELATED WORK

Prior work shows that large LLMs can solve complex tasks by generating multiple reasoning steps from few-shot demonstrations. This reasoning-oriented prompting provides context for mathematical problem-solving methods.

  • LLMs with more than 100B parameters have been shown to solve complex tasks by generating multiple reasoning steps from few-shot examples.

3 METHOD

MetaMath constructs a diverse mathematical dataset by generating answer augmentations, rephrased questions, and backward-reasoning questions, then fine-tunes LLMs on the merged data. The procedure preserves correct-answer examples while targeting broader reasoning patterns.

  • Method overview: MetaMathQA combines answer augmentation, question rephrasing, Self-Verification, and FOBAR backward-question data into one training dataset.
  • Answer augmentation: Answer augmentation samples multiple reasoning paths for each question using few-shot chain-of-thought prompting and temperature sampling, retaining paths with correct answers.
  • Question rephrasing: Rephrasing prompting generates alternative question texts, then produces reasoning paths and collects rephrased questions whose answers match the original answer.
  • Question rephrasing: Rephrased questions preserve high answer quality: GPT-3.5-Turbo obtains 76.30% with Complexity-based CoT versus 80.74% on original training questions.
  • Backward reasoning: Backward reasoning masks a number in a question and starts from the answer to generate reasoning steps that recover the masked value.
  • Backward reasoning: Self-Verification rewrites an answered question into a declarative statement before asking for the masked variable, while FOBAR appends the answer directly to the question.

4 EXPERIMENTS AND RESULTS

Experiments evaluate MetaMathQA construction and MetaMath across GSM8K and MATH, showing strong performance, benefits from diverse augmentations, and remaining difficulty with longer questions.

  • Experimental Setup: MetaMath is evaluated on GSM8K and MATH using LLaMA-2 base models at 7B, 13B, and 70B parameter scales.GPT-3.5-Turbo supplies rephrased questions and generated answers; 70B uses QLoRA for computational efficiency.
  • Results on GSM8K and MATH: MetaMath achieves state-of-the-art performance among open-source models with 1–10B parameters, improving testing accuracy by 11.6% on GSM8K and 9.1% on MATH.The reported comparison attributes the improvement to fine-tuning on MetaMathQA.
  • Results on GSM8K and MATH: MetaMath performs best among 11–50B and 51–70B models, exceeding SFT, RFT, and WizardMath by +7% and surpassing GPT-3.5-Turbo on GSM8K.The +7% comparison applies to both GSM8K and MATH for the 11–50B setting.
  • Augmentation Effects: Combining answer augmentation, rephrasing, FOBAR, and SV improves mathematical reasoning beyond individual augmentation strategies on GSM8K and MATH.Experiments report further gains when all augmented data are merged, including for the unseen GSM8K task after training on MATH data.
  • Perplexity Perspective: MetaMathQA has significantly lower perplexity than answer-only data and GSM8K chain-of-thought data, suggesting it is easier for LLaMA-2-7B to learn.The paper connects this simplicity with stronger problem-solving elicitation, while separately reporting that diversity improves generalization.
  • Diversity Perspective: Adding 20K Rephrasing, FOBAR, or SV samples yields 0.4%, 2.3%, and 2.6% accuracy gains, respectively, with diversity and accuracy positively correlated at 0.972.By contrast, adding 20K answer-augmentation samples after 80K produces only a 0.1% gain because of sample homogeneity.
  • Error Analysis: MetaMath and related methods struggle on longer GSM8K questions, although MetaMath consistently outperforms the compared methods.The paper identifies improving performance on longer questions as future work.

5 CONCLUDING REMARKS

MetaMath improves open-source LLM mathematical problem-solving by training on MetaMathQA, a dataset built through diverse question bootstrapping. The paper reports strong benchmark performance and releases the dataset, models, and training code.

  • 5 CONCLUDING REMARKS: MetaMathQA augments mathematical training data by rewriting questions through multiple perspectives, including rephrasing and declarative transformations.The supplied examples illustrate prompts for rephrasing questions and rewriting questions with provided answers into declarative statements.
  • 5 CONCLUDING REMARKS: MetaMath models are fine-tuned from open-source LLMs such as LLaMA-2 to improve elementary mathematical problem-solving.
  • 5 CONCLUDING REMARKS: 66.5% on GSM8K and 19.8% on MATH are reported for MetaMath-7B, surpassing previous open-source LLMs by a significant margin.
  • 5 CONCLUDING REMARKS: The paper emphasizes training-data characteristics as an important factor in improving LLM problem-solving capabilities.

B EXPERIMENTAL DETAILS

The experiments use standardized prompting and answer extraction procedures to evaluate fine-tuned models. Training configurations vary by model size and fine-tuning setting.

  • B EXPERIMENTAL DETAILS: Fully fine-tuned models are trained with AdamW for 3 epochs using batch size 128 and a learning rate of 2e-5 with 3% warmup.The 7B and 13B models use 8 NVIDIA A100 GPUs; the supplied passage also describes QLoRA settings for the 70B model.
  • B EXPERIMENTAL DETAILS: MetaMath evaluates GSM8K and MATH with zero-shot prompting because it performs better for fine-tuned LLMs and reduces inference costs.
  • B EXPERIMENTAL DETAILS: Evaluation uses temperature 0 for the fine-tuned LLaMA model.
  • B EXPERIMENTAL DETAILS: MetaMath extracts the final answer from the string following “The answer is:” and appends this format to MetaMathQA answers during training.

C HOW DO DIFFERENT LLMS SOLVE REVERSAL MATHEMATICAL PROBLEMS?

The reversal-mathematical case study compares how different language models solve a backward problem. The examples show that MetaMath produces the ground-truth answer while several baselines make errors.

  • C HOW DO DIFFERENT LLMS SOLVE REVERSAL MATHEMATICAL PROBLEMS?: The case study asks models to determine x in a robe problem whose ground-truth answer is 50.
  • C HOW DO DIFFERENT LLMS SOLVE REVERSAL MATHEMATICAL PROBLEMS?: MetaMath's reverse-problem reasoning reaches x = 50 and outputs 50 as the final answer.
  • C HOW DO DIFFERENT LLMS SOLVE REVERSAL MATHEMATICAL PROBLEMS?: The displayed comparisons include incorrect SFT and RFT answers of 60 and 2 before the correct result is shown.

D GENERALIZATION TO GAME OF 24

The paper extends question bootstrapping beyond GSM8K and MATH to Game of 24 and Game of n. Combining answer augmentation with bootstrapping improves accuracy over supervised fine-tuning in both settings.

  • D GENERALIZATION TO GAME OF 24: Question bootstrapping diversifies questions in forward and backward reasoning directions and is presented as extensible to other reasoning tasks.The Game of 24 extension uses arithmetic expressions to generate additional question-answer pairs.
  • D GENERALIZATION TO GAME OF 24: Answer augmentation enumerates multiple correct solutions for the same Game of 24 numbers, diversifying answer paths without altering the question.
  • D GENERALIZATION TO GAME OF 24: Game of 24 experiments use 1,362 questions, split into 681 training and 681 held-out testing questions.
  • D GENERALIZATION TO GAME OF 24: AnsAug and AnsAug+Bootstrapping achieve higher Game of 24 accuracy than SFT, with bootstrapping further improving AnsAug.
  • D GENERALIZATION TO GAME OF 24: On Game of n, both augmentation methods outperform SFT by a large margin, while AnsAug+Bootstrapping performs best.

E.1 METAMATHQA IS USEFUL FOR DIFFERENT BASE MODELS

MetaMathQA remains effective when used to fine-tune all four evaluated base models, including LLaMA-2, Mistral-7B, and Llemma-7B. The reported improvements are consistently large across the evaluated settings.

  • MetaMathQA is consistently useful for LLaMA-2-7B, LLaMA-2-13B, Mistral-7B, and Llemma-7B.The authors report large improvements across all four base models.
  • The evaluation compares MetaMathQA-fine-tuned models on GSM8K and MATH.Testing accuracy on both benchmarks is used to assess effectiveness across base models.
  • MetaMathQA produces large improvements across the evaluated base-model configurations.

E.2 TESTING ACCURACY UNDER DIFFERENT AUGMENTATION DATA SIZE

Question bootstrapping improves testing accuracy more effectively than answer augmentation alone as augmentation data increases. The comparison includes LLaMA-2-7B, LLaMA-2-13B, and Mistral-7B experiments.

  • Question bootstrapping boosts testing accuracy by a large margin, whereas AnsAug quickly reaches saturation.
  • The comparison uses 240K AnsAug examples against MetaMathQA-GSM8K data with question bootstrapping.
  • The data-size study evaluates LLaMA-2-7B and extends comparisons to LLaMA-2-13B and Mistral-7B.

E.3 ABLATION STUDY ON A LARGER MODEL LLAMA-2-13B

On LLaMA-2-13B, combining augmentation types slightly improves accuracy, while merging FOBAR and SV data yields further gains. Additional evaluation reports MetaMath's out-of-distribution performance on numerical DROP questions.

  • Ablation study on a larger model LLaMA-2-13B: Combining answer augmentation and rephrasing augmentation produces slightly higher accuracy on LLaMA-2-13B.
  • Ablation study on a larger model LLaMA-2-13B: Merging FOBAR and SV augmentation data further improves accuracy on LLaMA-2-13B.
  • Ablation study on a larger model LLaMA-2-13B: The augmentation study reports testing accuracy for LLaMA-2-13B fine-tuned on GSM8K.
  • MetaMath-7B and MetaMath-13B outperform baseline models in zero-shot evaluation on numerical-answer DROP questions.This result is presented as evidence that MetaMath does not suffer benchmark hacking on GSM8K and MATH.
Loading 2309.12284v4…