PaperlayerView exact revision on arXiv

Source-linked AI summary

MAmmoTH2: Scaling Instructions from the Web

Xiang Yue, Tuney Zheng, Ge Zhang, Wenhu Chen

arXiv:2405.03548v4cs.CL

TL;DR

LLM reasoning remains limited, while existing instruction data are constrained by annotation cost, distillation, or scale. The paper mines 10M web-based instruction-response pairs through recall, extraction, and refinement, then fine-tunes base models. MAmmoTH2 improves held-out reasoning performance, and MAmmoTH2-Plus reaches state-of-the-art results on several reasoning and chatbot benchmarks.

  • Problem

    LLMs remain limited on complex reasoning, while instruction-tuning data commonly depend on human annotation or GPT-4 distillation.

  • Method

    The paper recalls relevant web documents, extracts instruction-response pairs with open-source LLMs, refines them, and fine-tunes base models on the resulting WEBINSTRUCT dataset.

  • Results

    MAmmoTH2 significantly improves held-out reasoning performance, while MAmmoTH2-Plus achieves state-of-the-art results on several reasoning benchmarks and competitive MATH and GSM8K performance.

  • Takeaways & Limitations

    The results support harvesting large-scale instruction data from the web without human annotation or GPT-4 distillation for instruction tuning.

  • Takeaways & Limitations

    Extracted answers often lack intermediate rationales, while formatting issues and LLM modifications can introduce hallucinations during refinement.

Abstract

from arXiv · show

Instruction tuning improves the reasoning abilities of large language models (LLMs), with data quality and scalability being the crucial factors. Most instruction tuning data come from human crowd-sourcing or GPT-4 distillation. We propose a paradigm to efficiently harvest 10 million naturally existing instruction data from the pre-training web corpus to enhance LLM reasoning. Our approach involves (1) recalling relevant documents, (2) extracting instruction-response pairs, and (3) refining the extracted pairs using open-source LLMs. Fine-tuning base LLMs on this dataset, we build MAmmoTH2 models, which significantly boost performance on reasoning benchmarks. Notably, MAmmoTH2-7B's (Mistral) performance increases from 11% to 36.7% on MATH and from 36% to 68.4% on GSM8K without training on any in-domain data. Further training MAmmoTH2 on public instruction tuning datasets yields MAmmoTH2-Plus, achieving state-of-the-art performance on several reasoning and chatbot benchmarks. Our work demonstrates how to harvest large-scale, high-quality instruction data without costly human annotation or GPT-4 distillation, providing a new paradigm for building better instruction tuning data.

1 Introduction

The paper addresses limited complex reasoning in LLMs by mining dispersed, naturally occurring instruction data from the web. Its three-step pipeline produces WEBINSTRUCT, whose fine-tuning substantially improves reasoning and supports strong performance across reasoning, general, and chatbot tasks.

  • LLMs still have limited ability on complex mathematics, science, and engineering reasoning tasks.
  • Web pre-training corpora contain diverse, high-quality instruction data, but the data are highly dispersed and difficult to discover.
  • The pipeline recalls relevant Common Crawl documents, extracts question-answer pairs with open-source LLMs, and refines the resulting candidates.
  • 14 absolute points is the average improvement of MAmmoTH2-7B over Mistral-7B across seven held-out reasoning benchmarks.Mistral-7B’s MATH accuracy rises from 11.2% to 36.7% without in-domain evaluation data.
  • MAmmoTH2-Plus achieves state-of-the-art results on several reasoning benchmarks and competitive performance on MATH and GSM8K after additional public instruction tuning.It also performs well on code generation and general tasks, while the 10M-data comparison shows a six-point reasoning advantage over Llama-3-8B-Instruct with matched general-task performance.

2 WEBINSTRUCT

WEBINSTRUCT is constructed by recalling relevant web documents, extracting question-answer pairs, and refining them with language models. The resulting dataset targets a middle ground between smaller high-quality SFT datasets and larger, noisier continued-training corpora.

  • The construction pipeline has three stages: relevant-document recall, Q-A extraction, and Q-A refinement.
  • 100K positive seed examples and 100K negative Common Crawl documents train a fastText model to recall relevant web documents.
  • 18M recalled documents contain many natural Q-A pairs, but ads, markup, boilerplate, and other noise limit direct training gains.
  • Qwen-72B extracts question-answer pairs after rule-based HTML preprocessing removes site information, advertisements, and boilerplate.
  • Mixtral-22B×8 and Qwen-72B reformat extracted pairs and add intermediate reasoning when answers lack explanations, yielding 10M final Q-A pairs.
  • WEBINSTRUCT contains roughly 5B tokens, balancing the scalability of continued-training datasets with higher quality from its three-step construction pipeline.
  • Additional tuning uses OpenHermes 2.5, Code-Feedback, and Math-Plus to improve diversity, quality, reasoning, and chat ability.

3 Experimental Setup

The experiments fine-tune several open-source base models and evaluate reasoning across seven benchmarks, alongside code, general-language, and chat evaluations. The setup is designed to test performance across model scales, domains, and task types.

  • Mistral 7B, Mixtral 8×7B, Llama-3 8B, and Yi-34B serve as the base models for validation.
  • The reasoning evaluation covers GSM8K, MATH, TheoremQA, BBH, ARC-C, GPQA, and MMLU-STEM.
  • Table 2 compares released base and instruction models, using few-shot chain-of-thought with task-specific shot counts.
  • Additional evaluations cover HumanEval, MBPP, MMLU, MMLU-Pro, MT-Bench, AlpacaEval 2.0, and Arena Hard.

4 Main Results

MAmmoTH2 improves reasoning performance over base models and remains strong across code generation, language understanding, and instruction-following tasks. MAmmoTH2-Plus further reaches state-of-the-art or competitive results while outperforming comparable instruction models in several evaluations.

  • Reasoning Benchmarks: MAmmoTH2-7B boosts Mistral-7B performance by an average of 14 points, while MAmmoTH2-8B boosts Llama-3-8B-base by an average of 8.8 points on held-out reasoning benchmarks.WEBINSTRUCT contains no training data from the evaluation benchmarks.
  • Additional Tasks: MAmmoTH2-7B-Plus achieves average scores of 66.1 on HumanEval(+) and 58.2 on MBPP(+), outperforming Mistral-7B-Instruct-v0.2 on these code-generation metrics.These are the highest average scores reported for the two code-generation datasets in the comparison.
  • Additional Tasks: MAmmoTH2-8×7B-Plus achieves the highest scores on the AlpacaEval 2.0 and Arena Hard leaderboards, surpassing GPT-3.5-Turbo and Tulu-2-DPO-70B.The evaluations cover general language understanding and instruction-following capabilities.
  • Additional Tasks: Strong code-generation and general-language results indicate that MAmmoTH2 generalizes beyond reasoning benchmarks rather than overfitting to them.The paper characterizes the models as versatile and robust across a wide range of tasks.

5 Ablation Study

The ablations show that larger instruction datasets and supervised fine-tuning improve performance across representative reasoning tasks. Refinement generally improves extracted data, but missing rationales, formatting problems, and hallucinations remain documented quality concerns.

  • Scaling Effect of Instructions: SFT loss is more effective than LM loss for training on the synthetic instruction data.This comparison is part of the scaling study across MATH, TheoremQA, and ARC-C.
  • Data Refinement: Models trained on data refined by Mixtral-22B×8, Qwen-72B, or both are compared using the same 9000 steps and global batch size of 512.The supplied passages establish the controlled comparison but do not report its full outcome.
  • Case Study: Extracted question-answer pairs from well-formed exam and education websites are generally high quality, but many answers lack intermediate rationales, which could lead to worse generalization.The paper presents this as a dataset-quality issue observed in its case study.
  • Case Study: Formatting problems in extracted pairs challenge refinement, and refinement models can sometimes alter the original content’s intention, causing hallucinations.Mixtral and Qwen-72B were prompted to complete missing intermediate steps before refinement.
  • Case Study: Among 50 sampled refined examples, 78% improved after refinement and 10% introduced hallucinations.Human annotators judged correctness and improvement in format and intermediate solutions relative to extracted examples.

6 Conclusion

The paper mines naturally occurring instruction-response pairs from the web through recall, extraction, and refinement, producing the 10M-example WEBINSTRUCT dataset. Training on this dataset substantially improves reasoning performance, while additional public instruction data further strengthens the resulting models.

  • Conclusion: WEBINSTRUCT contains 10M diverse instruction-response pairs harvested from naturally occurring web data through recall, extraction, and refinement.The pipeline is presented as a way to mine broad-domain instruction data from the pre-training corpus.
  • Conclusion: The additional public datasets include OpenHermes 2.5, Code-Feedback, and Math-Plus for instruction tuning.OpenHermes 2.5 combines math, science, coding, synthetic, and chat-based data, while Code-Feedback focuses on multi-turn code generation and refinement.
  • Conclusion: Aligning the supplementary datasets with WEBINSTRUCT provides training resources spanning questions and explanations across a wide range of topics.The paper frames this alignment as supporting model versatility and applicability in real-world scenarios.

B Details of Evaluation Datasets

The evaluation spans reasoning benchmarks designed for different levels of mathematical, scientific, and general reasoning, alongside code-generation, general-language, and chat assessments.

  • Reasoning Benchmarks: The reasoning evaluation covers datasets that test different aspects of capability across domains and levels of complexity.The paper describes these datasets as collectively assessing reasoning across a spectrum of complexity and realism.
  • Reasoning Benchmarks: GSM8K contains 1.32K grade-school math problems for basic arithmetic and reasoning, while MATH contains 5000 competition-level problems for complex mathematical reasoning.TheoremQA evaluates applying mathematical theorems in fields including mathematics, physics, and engineering.
  • Additional Evaluations: Additional evaluations cover code generation, general language understanding, and chat quality through HumanEval, MBPP, MMLU, MT-Bench, AlpacaEval 2.0, and Arena Hard.The code-generation evaluation includes augmented versions of HumanEval and MBPP.

D Code Generation Results

The paper reports code-generation results in Table 6, whose baseline values are copied from the EvalPlus leaderboard.

  • Code Generation Results: Table 6 reports code-generation results for the paper’s models and baselines.The passage identifies the table as the location of the code-generation results but does not provide the individual scores.
  • Code Generation Results: The table compares different models on code-generation performance.The supplied table caption does not specify the individual metrics or model scores.
  • Code Generation Results: Baseline results in Table 6 are copied from the EvalPlus leaderboard.No baseline values are included in the supplied passage.

E Impact of Additional Public Instruction Tuning Datasets

The appendix examines how additional public instruction datasets affect model training, while the website-domain appendix describes WEBINSTRUCT’s source distribution.

  • E Impact of Additional Public Instruction Tuning Datasets: The appendix compares models trained on WEBINSTRUCT only, additional public datasets only, or WEBINSTRUCT followed by continued training on those datasets.The combined setup first trains on WEBINSTRUCT and then continues training on additional public datasets.
  • E Impact of Additional Public Instruction Tuning Datasets: The results of these three training setups are reported in Table 7.The passage frames Table 7 as the comparison of dataset-training configurations.
  • E Impact of Additional Public Instruction Tuning Datasets: The WEBINSTRUCT-only setup isolates the contribution of the mined dataset within the appendix comparison.This is one of the three explicitly listed training configurations.
  • E Impact of Additional Public Instruction Tuning Datasets: The additional-public-datasets-only setup provides a separate comparison condition in the appendix analysis.This is one of the three explicitly listed training configurations.
  • E Impact of Additional Public Instruction Tuning Datasets: The combined setup tests sequential training on WEBINSTRUCT and additional public instruction datasets.The stated order is WEBINSTRUCT first, followed by continued training on the additional datasets.
  • F Distributions of Website Domains in WEBINSTRUCT: Figure 7 shows the distribution of the top websites represented in WEBINSTRUCT.The supplied passages do not list the individual websites or their proportions.
  • F Distributions of Website Domains in WEBINSTRUCT: The website-domain analysis characterizes WEBINSTRUCT through the distribution of its most represented websites.Figure 7 is identified as the distribution of top websites in the instruction dataset.

G Domain Distribution of WEBINSTRUCT

Figure 8 breaks WEBINSTRUCT down by subject domains and data sources; the supplied description reports that science-related subjects predominate.

  • Domain Distribution: Figure 8 presents WEBINSTRUCT’s composition by subject domains and data sources.The figure is intended to show the composition and diversity of the mined instruction-response pairs.
  • Domain Distribution: Subject labels are automatically annotated with Llama-3-8B-Instruct.The annotation procedure applies to the subject-domain labels in the breakdown.
  • Domain Distribution: The education-versus-forum distribution is obtained by analyzing the samples’ source URLs.The data-source categorization is based on URL analysis rather than the subject-labeling model.
  • Domain Distribution: Science-related subjects constitute the predominant portion of WEBINSTRUCT, with 81.69% reported in the figure description.The supplied passage truncates the remainder of the sentence after this percentage.

H Limitations of WEBINSTRUCT

WEBINSTRUCT offers broad subject coverage but may contain noise, inaccuracies, biases, and underrepresented domains. The authors identify stronger filtering and broader seed data as future needs.

  • WEBINSTRUCT may still contain noise and inaccuracies despite its three-step construction pipeline.
  • The dataset spans subjects such as mathematics, science, and engineering, but some humanities and daily-chat topics may be underrepresented.
  • The authors suggest human-in-the-loop validation, data-selection filtering, and broader seed coverage as possible improvements.
  • Instruction-response quality may vary, so model outputs can reflect inconsistencies in the dataset.

J Case Study

The case study shows that refinement often improves clarity, structure, and explanatory detail, but it can also add verbosity, omit information, or introduce factual errors. Its value therefore depends on the quality and faithfulness of the transformation.

  • Extracted answers commonly lack intermediate rationale, while formatting damage can create unrecoverable errors.
  • Refinement can improve mathematical clarity by standardizing notation and presenting formulas explicitly.
  • Refined explanations can add concrete solution steps, such as factoring denominators and comparing leading coefficients for asymptotes.
  • Structured formatting and added explanations can make questions easier to evaluate and more educational.
  • Some refinements add words without improving comprehension when the original question is already clear.
  • Refinement can introduce inaccuracies, including an incorrect claim that tariffs reduce export spending.
Loading 2405.03548v4…