Source-linked AI summary
Infinity Instruct: Scaling Instruction Selection and Synthesis to Enhance Language Models
Jijie Li, Li Du, Hanyu Zhao, Bo-wen Zhang, Liangdong Wang, Boyan Gao, Guang Liu, Yonghua Lin
TL;DR
Instruction fine-tuning must improve task alignment without eroding foundational capabilities, while building comprehensive datasets at modern scale is costly. Infinity-Instruct addresses these challenges with a scalable, labeling-driven pipeline that constructs foundational and conversational datasets, and its models outperform official instruction-tuned counterparts across foundational and conversational benchmarks, including an 8.6% conversational gain over GPT-4-0314 for InfInstruct-Llama3.1-70B.
Problem
Instruction tuning can cause catastrophic forgetting of linguistic and reasoning capabilities, while constructing comprehensive high-quality datasets requires prohibitive human and computational resources.
Method
Infinity-Instruct uses a labeling-driven pipeline combining filtering, clustering, coverage-based selection, iterative synthesis, diagnosis, and quality filtering to build foundational and conversational instruction datasets.
Results
Infinity-Instruct models consistently outperform official instruction-tuned counterparts on foundational and conversational benchmarks; InfInstruct-Llama3.1-70B surpasses GPT-4-0314 by 8.6% in conversational ability and matches its foundational performance.
Takeaways & Limitations
A unified dataset construction pipeline can support both foundational and conversational tasks while yielding high-quality, diverse instruction data and strong benchmark performance.
Takeaways & Limitations
The pipeline relies on existing open-source models for labeling and selection, requires substantial computational resources, and leaves real-world robustness, safety, and long-term capability retention for further investigation.
Abstract
from arXiv · showhide
Large Language Models (LLMs) demonstrate strong performance in real-world applications, yet existing open-source instruction datasets often concentrate on narrow domains, such as mathematics or coding, limiting generalization and widening the gap with proprietary models. To bridge this gap, we introduce Infinity-Instruct, a high-quality instruction dataset designed to enhance both foundational and chat capabilities of LLMs through a two-phase pipeline. In Phase 1, we curate 7.4M high-quality foundational instructions (InfInstruct-F-7.4M) from over 100M samples using hybrid data selection techniques. In Phase 2, we synthesize 1.5M high-quality chat instructions (InfInstruct-G-1.5M) through a two-stage process involving instruction selection, evolution, and diagnostic filtering. We empirically evaluate Infinity-Instruct by fine-tuning several open-source models, including Mistral, LLaMA, Qwen, and Yi, and observe substantial performance gains across both foundational and instruction following benchmarks, consistently surpassing official instruction-tuned counterparts. Notably, InfInstruct-LLaMA3.1-70B outperforms GPT-4-0314 by 8.6\% on instruction following tasks while achieving comparable foundational performance. These results underscore the synergy between foundational and chat training and offer new insights into holistic LLM development. Our dataset\footnote{https://huggingface.co/datasets/BAAI/Infinity-Instruct} and codes\footnote{https://gitee.com/li-touch/infinity-instruct} have been publicly released.
1 Introduction
Infinity-Instruct addresses limitations in open-source instruction data with a scalable pipeline that selects and synthesizes high-quality foundational and conversational instructions. Its resulting models improve foundational and conversational benchmark performance, including a reported 8.6% conversational advantage over GPT-4-0314 for InfInstruct-Llama3.1-70B.
- Instruction fine-tuning must balance task alignment with preserving foundational linguistic and reasoning capabilities.Naive adaptation without sufficient regularization can cause catastrophic forgetting.
- 7.4M foundational instructions are selected from over 100M samples using filtering, clustering, and coverage-based strategies.
- 1.5M conversational instructions are synthesized from 1.2M high-quality seeds through labeling, iterative synthesis, and diagnosis.
- The pipeline applies de-duplication and contamination detection to support clean generalization across mathematics, coding, and knowledge Q&A.
- Infinity-Instruct consistently improves multiple open-source models on foundational and conversational benchmarks.Evaluated models include Mistral, Llama, Qwen, and Yi.
- InfInstruct-Llama3.1-70B surpasses GPT-4-0314 by 8.6% in conversational ability while achieving near parity in foundational tasks.
2 Methodology
Infinity-Instruct builds foundational and conversational instruction datasets through staged selection, labeling, synthesis, and diagnosis over a large open-source instruction pool.
- Pipeline overview: The pipeline collects over 100 million instructions, selects high-quality foundational data, and synthesizes conversational data from selected seeds.It produces InfInstruct-F-7.4M and InfInstruct-G-1.5M through separate selection and synthesis stages.
- Motivation: Initial experiments found that existing open-source datasets still lagged GPT-3.5 and GPT-4 across diversity, code comprehension, knowledge, and dialogue capabilities.These gaps motivate improving instruction quality and diversity through selection and synthesis.
- Foundational data selection: The foundational selection module uses source filtering, rule-based filtering, DSIR, incremental evaluation, and weak-domain supplementation.Selection rules are adapted to task characteristics such as mathematics, code, and knowledge.
- Conversational data synthesis: InfInstruct-G-1.5M is synthesized from 9M conversational instructions by labeling, selecting 1.2M diverse and difficult seeds, evolving them, and diagnosing model weaknesses.The process uses Qwen1.5-72B for labeling, Qwen1.5-7B for difficulty evaluation, evolutionary rewriting, and GPT-4-based response diagnosis.
3 Experiments
Experiments evaluate Infinity-Instruct across models, datasets, conversational and foundational benchmarks, and data scales, showing broad gains and increasing performance with more selected instruction data.
- Dataset comparison: Infinity-Instruct improves Llama3.1-8B more than MAGPIE-Pro across all three reported conversational review sets.MAGPIE-Pro remains the strongest comparison dataset overall in the cited evaluation, while Infinity-Instruct provides the larger Llama3.1-8B improvement.
- Foundational evaluation: Infinity-Instruct enables Llama3.1-8B to surpass the official Llama3.1-8B-Instruct model across the reported foundational evaluations.The cited comparison states that Infinity-Instruct improves foundational capabilities across the evaluation set.
- Scaling curves: Increasing instruction-data quantity continuously raises overall performance in both foundational and conversational scenarios.The strongest growth appears on reasoning tasks such as MATH and GSM-8K, using reward-ranked sampling across labeling categories.
4 Ablation Study
The ablation study compares single-dataset, mixed one-stage, and curriculum-style two-stage training, finding that staged training best balances foundational and conversational ability.
- Single-dataset training: 93.3% of full two-stage performance is achieved by training on the conversational dataset alone.Training on the foundational dataset alone does not maximize conversational ability, but it further improves conversation in two-stage training.
- Training strategy: Two-stage training yields the highest conversational ability and higher foundational ability than simply mixing foundational and conversational datasets in one stage.The staged setup trains on foundational data first and conversational data second, following a curriculum-learning design.
- Training strategy: The results support an association between stronger foundational capabilities and stronger conversational capabilities under the tested training settings.The study reports that one-stage dataset merging does not produce the highest score on either capability axis.
5 Conclusion
Infinity Instruct addresses limitations in open-source instruction datasets by supporting both foundational and conversational tasks. Fine-tuning open-source models on the resulting data yields strong gains, including an 8.6% advantage over GPT-4-0314 in chat ability while maintaining near-parity foundational performance.
- Infinity Instruct targets the limitations of narrow-domain open-source datasets through high-quality curation and synthesis for foundational and conversational tasks.
- 8.6%: InfInstruct-Llama3.1-70B surpassed GPT-4-0314 in chat ability while maintaining near parity in foundational tasks.
- Fine-tuning multiple open-source models on Infinity Instruct produced state-of-the-art results across foundational and conversational benchmarks.
A Evaluation Setup
The evaluation uses separate benchmark groups for chat ability and foundational ability. Chat benchmarks assess instruction following with coding, multi-turn, and open-ended comparisons, while foundational benchmarks cover reasoning, knowledge, language, and mathematics.
- Chat ability evaluation: Chat ability is evaluated with instruction-following benchmarks including HumanEval, MBPP, Mt-Bench, AlpacaEval2.0, and Arena-Hard.These benchmarks cover programming, multi-turn questions, open-ended prompts, and comparisons judged against GPT-4 or GPT-4-Turbo.
- Chat ability evaluation: Mt-Bench evaluates multi-turn questions across eight categories and uses GPT-4 as referee and baseline for response win rate.
- Chat ability evaluation: AlpacaEval2.0 uses 805 prompts, GPT-4 judging, and length-controlled win rates intended to reduce length bias.
- Chat ability evaluation: Arena-Hard compares responses to GPT-4 using GPT-4-Turbo as judge across 500 challenging user queries.
- Foundational ability evaluation: Foundational ability is evaluated with OpenCompass across MMLU, C-EVAL, MATH, and GSM8K.The datasets assess broad reasoning and knowledge, Chinese-language tasks, multi-step mathematics, and elementary arithmetic or logical reasoning.
B Related works
Related work frames Infinity Instruct as a response to the quality and curation challenges of open-source instruction datasets. Its distinguishing focus is combining large-scale foundational data with evolved chat data, systematic labeling, and model-ability diagnosis.
- Prior work highlights instruction-data filtering and curation as important for efficient instruction tuning and complex-instruction following.
- Open-source projects such as OpenHermes-2.5 and CodeBagel provide broad instruction collections, but maintaining quality and relevance remains challenging.
- Infinity Instruct combines a multi-million-instruction foundational dataset with a chat dataset evolved from high-quality seed data for real conversation scenarios.
- Figure 5 visualizes first-level label-type distributions for InfInstruct-F-7.4M and InfInstruct-G-1.5M using up to 2,000 instructions per label type.
- The project systematically tags instructions and uses AI assistants for validation and generation to produce data that is large, diverse, and informative.
C Training Details
Training proceeds in two phases: foundational training first, followed by dialogue fine-tuning for chat ability. The same training hyperparameters are applied across both phases, with their relevance linked to pretrained-model attributes.
- The first phase applies Infinity-Instruct-7M to improve pretrained models’ foundational abilities, including mathematics and coding.
- The second phase fine-tunes a dialogue dataset to improve the models’ chat ability.
- The same training hyperparameters are used in both phases, with relevance associated with pretrained-model attributes such as parameter count and end learning rate.
D Data Analysis
InfInstruct-F-7.4M and InfInstruct-G-1.5M exhibit complementary data distributions, with challenging reasoning tasks more dispersed and chat instructions balanced across task types. Both datasets are predominantly single-turn while retaining multi-turn examples to support dialogue adaptability.
- Semantic Distribution: Logic and Reasoning instructions show a more dispersed semantic distribution, reflecting their challenging and complex task types.The distributions were visualized with t-SNE for both datasets.
- Semantic Distribution: InfInstruct-G-1.5M emphasizes balanced allocation across task types for robust instruction following across downstream applications.
- Dialogue Turns: 92.6% of InfInstruct-F-7.4M instructions are single-turn dialogues.Approximately 7.4% are multi-turn instructions.
- Dialogue Turns: 97.7% of InfInstruct-G-1.5M instructions are single-turn dialogues.Approximately 2.3% are multi-turn instructions, supplementing the dataset for multi-turn scenarios.
E Limitation
Infinity-Instruct’s labeling and selection process may inherit biases from existing open-source models and miss nuanced instruction types. Its scalability still requires substantial computation, while evaluation remains focused mainly on benchmarks rather than real-world robustness, safety, and long-term capability retention.
- Limitations: The labeling and selection process relies on existing open-source models, which may introduce bias or miss nuanced instruction types.
- Limitations: The pipeline requires substantial computational resources for clustering, synthesis, and fine-tuning despite being scalable.
- Limitations: Evaluation focuses primarily on benchmark performance, leaving real-world robustness, safety, and long-term cross-domain retention for further investigation.
F Broader Impacts
Infinity-Instruct provides an open, scalable framework intended to lower barriers to building high-quality instruction-tuned models and broaden access to capable LLMs. Because it relies on existing data and models, careful auditing and responsible deployment remain necessary.
- Broader Impacts: The open, scalable framework lowers barriers to building high-quality instruction-tuned models.
- Broader Impacts: Infinity-Instruct may democratize access to capable LLMs for researchers and organizations with limited resources.
- Broader Impacts: Reliance on existing data and models may inherit their biases or limitations, requiring careful dataset auditing and responsible deployment.