Source-linked AI summary
AgentInstruct: Toward Generative Teaching with Agentic Flows
Arindam Mitra, Luciano Del Corro, Guoqing Zheng, Shweti Mahajan, Dany Rouhana, Andres Codas, Yadong Lu, Wei-ge Chen, Olga Vrousgos, Corby Rosset, Fillipe Silva, Hamed Khanpour, Yash Lara, Ahmed Awadallah
TL;DR
Synthetic data varies in quality and diversity, and its effective use often requires substantial human curation; model imitation and collapse are additional concerns. AgentInstruct uses agentic flows to generate diverse prompts and responses for post-training from raw documents and code, without requiring seed prompts. Compared with Mistral-Instruct-7B, the model improves AGIEval by 40%, MMLU by 19%, GSM8K by 54%, BBH by 38%, and AlpacaEval by 45%.
Problem
Synthetic data varies in quality and diversity, and its effective use often requires substantial human curation; model imitation and collapse are additional concerns.
Method
AgentInstruct uses agentic flows to generate diverse prompts and responses for post-training from raw documents and code, without requiring seed prompts.
Results
Compared with Mistral-Instruct-7B, the model improves AGIEval by 40%, MMLU by 19%, GSM8K by 54%, BBH by 38%, and AlpacaEval by 45%.
Takeaways & Limitations
AgentInstruct supports scalable, skill-targeted post-training data generation using raw materials and can produce datasets containing both prompts and responses.
Takeaways & Limitations
Without suitable safeguards, models generated through this approach could be misused to produce disinformation or harmful content.
Abstract
from arXiv · showhide
Synthetic data is becoming increasingly important for accelerating the development of language models, both large and small. Despite several successful use cases, researchers also raised concerns around model collapse and drawbacks of imitating other models. This discrepancy can be attributed to the fact that synthetic data varies in quality and diversity. Effective use of synthetic data usually requires significant human effort in curating the data. We focus on using synthetic data for post-training, specifically creating data by powerful models to teach a new skill or behavior to another model, we refer to this setting as Generative Teaching. We introduce AgentInstruct, an extensible agentic framework for automatically creating large amounts of diverse and high-quality synthetic data. AgentInstruct can create both the prompts and responses, using only raw data sources like text documents and code files as seeds. We demonstrate the utility of AgentInstruct by creating a post training dataset of 25M pairs to teach language models different skills, such as text editing, creative writing, tool usage, coding, reading comprehension, etc. The dataset can be used for instruction tuning of any base model. We post-train Mistral-7b with the data. When comparing the resulting model Orca-3 to Mistral-7b-Instruct (which uses the same base model), we observe significant improvements across many benchmarks. For example, 40% improvement on AGIEval, 19% improvement on MMLU, 54% improvement on GSM8K, 38% improvement on BBH and 45% improvement on AlpacaEval. Additionally, it consistently outperforms other models such as LLAMA-8B-instruct and GPT-3.5-turbo.
1 Introduction
AgentInstruct frames Generative Teaching as an agentic approach to producing abundant, diverse, high-quality post-training data from raw documents and code. Its 25-million-pair dataset improves a Mistral-7B model across multiple benchmarks, while supporting prompt and response generation without task-specific seed data.
- Motivation: The approach responds to concerns that synthetic-data training can cause model collapse or teach stylistic imitation because high-quality, diverse data is difficult to curate.Raw seeds are abundant and may reduce reliance on existing prompts or benchmark-specific data.
- Generative Teaching: Generative Teaching broadens synthetic post-training beyond reusing prompt sets by generating abundant, diverse, challenging data to teach particular skills.AgentInstruct creates demonstration and feedback data using raw documents as input, while domain-specific seeds can target specialization.
- AgentInstruct: AgentInstruct combines powerful models, tools, reflection, many agents, taxonomies, verification, and filtering to generate high-quality prompts and responses at scale.It does not require seed prompts and can use search and code interpreters within agentic flows.
- Dataset: AgentInstruct creates a 25-million-pair post-training dataset covering diverse skills from unstructured text and code seeds.The dataset is intended for instruction tuning different base models and includes skills such as creative writing, reasoning, math, retrieval-augmented generation, and tool use.
- Evaluation: 40% on AGIEval, 19% on MMLU, 54% on GSM8K, 38% on BBH, and 45% on AlpacaEval improve over Mistral-Instruct-7B after fine-tuning with AgentInstruct data.The resulting Orca-3 also outperforms LLAMA-8B-instruct and GPT-3.5 on multiple benchmarks.
2 Generative Teaching: AgentInstruct
AgentInstruct addresses the challenge of generating large, diverse, and complex synthetic teaching data through automated agentic flows seeded by raw content. Its pipeline transforms seeds, generates taxonomy-guided instructions, and iteratively refines them across many capabilities.
- AgentInstruct uses agentic flows with raw articles as seeds to automate scalable generation of diverse data with varying complexity.Automation supports scale, broad seeds support diversity, and iterative refinement supports increasing complexity.
- AgentInstruct Flows: Content Transformation Flow converts raw seeds into intermediate representations tailored to instruction-generation objectives.Multiple agents also introduce additional diversity and support higher-quality data generation.
- AgentInstruct Flows: Seed Instruction Generation Flow uses transformed seeds and an extensible taxonomy to produce diverse instructions for target tasks.The reading-comprehension implementation uses 43 question types spanning literal, critical, evaluative, reasoning, and argument-analysis questions.
- AgentInstruct Flows: Instruction Refinement Flow uses Suggester-Editor agents to increase the complexity and quality of initial instructions through iterative modification.Suggesters propose ways to make instructions more intricate, unsolvable, or tricky before Editors apply those changes.
- Coverage and Reading Comprehension: The framework was implemented for 17 capabilities, including reading comprehension, question answering, coding, retrieval-augmented generation, creative writing, and tool/API use.Reading-comprehension flows transform web-crawled text into structured passages and questions because raw sources do not consistently support diverse question types.
3 Orca-3
The authors construct the Orca-2.5-dataset from approximately 22 million AgentInstruct instructions plus 3.8 million publicly sourced instruction pairs, then use the combined 25.8 million pairs to fine-tune Mistral-7B into Orca-3.
- Dataset Description: The AgentInstruct portion contains approximately 22 million instructions covering 17 capabilities and is built from unstructured text and code files sampled from several public sources.Using unstructured content avoids benchmark-specific seeds and targets capabilities rather than a particular benchmark.
- Dataset Description: Approximately 25.8 million paired instructions are incorporated into Orca-3 training, combining 22 million AgentInstruct instructions with 3.8 million public instruction pairs.The additional pairs come from Orca-1, Orca-2, Orca-Math, and other publicly available sources.
- Dataset Description: The authors separately train Orca-2.5 on the 3.8 million additional instructions to evaluate the impact of the 22 million AgentInstruct instructions.This comparison isolates the contribution of the AgentInstruct-curated data within the broader training mixture.
- Training Details: Orca-3 is Mistral-7b-v0.1 fine-tuned on the AgentInstruct dataset and the added instruction pairs.The model uses the publicly available base model, whose permissive license supports redistribution.
- Training Details: Training tokenizes examples with the Mistral tokenizer, packs sequences to a maximum length of 8192, and masks labels so loss is computed only on responses.Fine-tuning uses three epochs, AdamW, an initial learning rate of 8e-6, cosine decay, and 500 warm-up steps across 152 NVIDIA A100 GPUs.
4 Evaluation Results
AgentInstruct is evaluated through Orca-Bench, broad benchmark suites, and skill-specific tests, with Orca-3 compared against multiple instruction-tuned baselines. The results report broad post-training gains, including improvements in aggregate performance, reading comprehension, mathematics, format following, summarization, and retrieval-augmented generation.
- Orca-Bench: 33.94% average improvement over Orca 2.5 and 14.92% over Mistral-Instruct-7B was measured across Orca-Bench dimensions with AgentInstruct data.Table 2 reports macro scores averaged across assessed dimensions, using GPT-4 as the score-10 reference.
- Reading Comprehension: 18% improvement over Orca 2.5 and 21% over Mistral-Instruct-7B was observed for reading comprehension, with Orca-3 matching GPT-4 on LSAT reading-comprehension sections.The reported reading-comprehension results include difficult LSAT sections intended for human test-takers.
- Math: 44% to 168% improvements were reported across mathematical benchmarks, supporting skill gains beyond optimization for a single benchmark.The authors state that Generative Teaching targets teaching a skill rather than generating data for a specific benchmark.
- Summarization and RAG: 31.34% lower hallucination rates were achieved while maintaining summarization quality comparable to GPT-4, and RAG performance improved by an average of 38.30%.On PubMedQA, Orca-3 achieved a 92.71% relative improvement over Mistral-Instruct; format following also improved by 11.5% and surpassed Gemini Pro.
5 Limitations
AgentInstruct reduces human expertise needed for large-scale synthetic-data generation, but remains constrained by data, model, validation, safety, cost, and extensibility limitations.
- Human expertise is reduced, but AgentInstruct remains an early approach with limitations inherited from synthetic-data generation.
- Flow construction for different skills still depends on human effort, limiting automatic extensibility.Future work is needed to automate construction from user specifications.
- Synthetic-data quality can be limited by inaccurate replication of real-world complexity, difficult validation, biased seeds, and dependence on seed quality.Poor-quality input data can produce poor-quality synthetic data, while seed biases may be reflected or amplified.
- Generating synthetic data with multiple agents and language models can be resource-intensive.
- The approach retains Mistral-family and broader language-model limitations, including transparency, content harms, hallucination, misuse, and bias risks.The authors specifically caution that models can be black boxes, hallucinate, generate harmful content, or be misused without safeguards.
- Orca-3 may be less accurate on areas underrepresented in its tuning-data distribution.
6 Conclusions
AgentInstruct uses agentic flows to generate diverse, high-quality post-training data from unstructured sources, and the resulting 25M-pair dataset substantially improves Orca-3 across benchmarks.
- AgentInstruct-generated data produced substantial performance gains across multiple benchmarks when used to post-train Orca-3.The dataset contained 25M pairs and was generated for model post-training.
- The approach targets large amounts of diverse, high-quality data while reducing intensive human curation during creation.
- Using unstructured content as seeds could support semi-automated pipelines for model customization and continual improvement.The conclusion identifies domain-specific seeds and agentic flows as bases for these applications.
A.1 Reading Comprehension Flow
The reading-comprehension flow uses specialized transformation agents to convert seed articles into varied passages that support comprehension-material generation.
- Reading Comprehension Flow: Nine transformation agents generate varied passages including arguments, debates, conversations, long texts, meeting transcripts, poems, and satirical content.
Instruction Taxonomy for Seed Instruction Generation Flow
The seed-instruction taxonomy organizes reading-comprehension tasks into distinct question types covering factual recall, numerical reasoning, and critical assessment.
- Literal comprehension questions ask for specific details or facts explicitly stated in the text.
- Numerical discrete-reasoning questions require combining and reasoning over multiple facts from the text.
- Critical comprehension questions ask readers to judge paired statements about a text’s purpose or point of view as true or false.
Instruction Taxonomy for Seed Instruction Generation Flow
The seed-instruction flow spans diverse text transformations, from rewriting and translation to formatting, sentiment, annotation, and constrained rewriting. These task types vary the requested operation while preserving or deliberately changing selected properties of the source text.
- The taxonomy includes paraphrasing, simplification, expansion, translation, formatting, sentiment modification, annotation, keyword replacement, and content removal.
- Additional tasks alter capitalization, styling, perspective, normalization, originality, language mixing, obfuscation, and textual entailment.
- The flow also generates rewrites under vocabulary or positional constraints, such as requiring words or sentences to begin with specified letters or vowels.
B Evaluation Details
Evaluation uses benchmark-specific extraction and judging procedures to convert open-ended model responses into scores. GPT-4-based evaluators handle many extraction, correctness, format, and quality judgments, while some benchmarks use author-provided code or prompts.
- Multiple-choice responses are generated openly, then GPT-4 extracts selected options for matching against ground truth, replacing regex-based extraction.
- Math and span-extraction tasks use GPT-4 to extract answers and compare them with supplied ground truth as Correct or Incorrect.
- EQBench uses GPT-4 to extract emotion scores, with scoring calibrated so 0 represents random answering and 100 represents perfect alignment with the benchmark’s target.
- Open-ended benchmarks use judge-based evaluation, including GPT-4 format scoring for FOFO and turn-level 1-to-10 judgments for MT-Bench.
- AlpacaEval measures the win-rate of evaluated outputs preferred by GPT-4-turbo over a reference answer, while InfoBench uses GPT-4 to assess decomposed-instruction adherence.
B.1 Summarization Quality and Hallucination Evaluation
The summarization evaluation uses GPT-4 prompts to judge response quality and hallucination. Dedicated prompt templates support separate assessment of summary quality and factual reliability.
- GPT-4 is used as an impartial judge for evaluating the quality of an AI assistant’s response to a user instruction.
- The evaluation includes a dedicated prompt template for detecting hallucinations in text summarization.
- A separate prompt template evaluates summary quality.