PaperlayerView exact revision on arXiv

Source-linked AI summary

Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothing

Zhangchen Xu, Fengqing Jiang, Luyao Niu, Yuntian Deng, Radha Poovendran, Yejin Choi, Bill Yuchen Lin

arXiv:2406.08464v2cs.CLcs.AI

TL;DR

Open-weight aligned LLMs often lack public alignment data, while existing synthetic-data methods face scalability challenges. MAGPIE extracts instructions directly from aligned models’ templates, and its aligned models achieve comparable or higher performance than official aligned models on instruction-following benchmarks.

  • Problem

    Open-weight aligned LLMs often keep alignment data private, while existing synthetic-data methods depend on prompt engineering and seed questions, limiting scalable high-quality diverse instruction-data creation.

  • Method

    MAGPIE prompts aligned LLMs with only pre-query templates to self-synthesize diverse instructions and responses, without prompt engineering or seed questions.

  • Results

    MAGPIE-aligned models achieve comparable or higher performance than official aligned models on instruction-following benchmarks; on AlpacaEval 2, LC exceeds 50% against Llama-3-8B-Instruct.

  • Takeaways & Limitations

    MAGPIE shows that aligned LLMs can self-synthesize useful instruction data directly from predefined templates, supporting scalable alignment-data construction without prompt engineering or seed questions.

  • Takeaways & Limitations

    MAGPIE-aligned LLMs show degraded performance on math and reasoning benchmarks and remain behind official models despite specialized reasoning datasets.

Abstract

from arXiv · show

High-quality instruction data is critical for aligning large language models (LLMs). Although some models, such as Llama-3-Instruct, have open weights, their alignment data remain private, which hinders the democratization of AI. High human labor costs and a limited, predefined scope for prompting prevent existing open-source data creation methods from scaling effectively, potentially limiting the diversity and quality of public alignment datasets. Is it possible to synthesize high-quality instruction data at scale by extracting it directly from an aligned LLM? We present a self-synthesis method for generating large-scale alignment data named Magpie. Our key observation is that aligned LLMs like Llama-3-Instruct can generate a user query when we input only the left-side templates up to the position reserved for user messages, thanks to their auto-regressive nature. We use this method to prompt Llama-3-Instruct and generate 4 million instructions along with their corresponding responses. We perform a comprehensive analysis of the extracted data and select 300K high-quality instances. To compare Magpie data with other public instruction datasets, we fine-tune Llama-3-8B-Base with each dataset and evaluate the performance of the fine-tuned models. Our results indicate that in some tasks, models fine-tuned with Magpie perform comparably to the official Llama-3-8B-Instruct, despite the latter being enhanced with 10 million data points through supervised fine-tuning (SFT) and subsequent feedback learning. We also show that using Magpie solely for SFT can surpass the performance of previous public datasets utilized for both SFT and preference optimization, such as direct preference optimization with UltraFeedback. This advantage is evident on alignment benchmarks such as AlpacaEval, ArenaHard, and WildBench.

1 INTRODUCTION

MAGPIE synthesizes diverse, high-quality instruction data directly from aligned LLMs by prompting only their pre-query templates, avoiding prompt engineering, seed questions, human intervention, and production-LLM API access. MAGPIE-based models outperform prior public-data approaches and achieve strong AlpacaEval 2 results, including surpassing official Llama-3-8B-Instruct and, with preference optimization extensions, GPT-4-Turbo(1106).

  • Motivation: Instruction-tuning success depends on high-quality data because it enables LLMs to follow instructions across diverse tasks, including tasks absent from training.Human-curated data is time-consuming and labor-intensive, while synthetic-data methods still face prompt-engineering, seed-selection, and diversity challenges.
  • Method: MAGPIE generates high-quality, diverse user queries when aligned LLMs receive only a pre-query template, using their autoregressive generation capabilities.The method directly extracts instructions from aligned models such as Llama-3-Instruct rather than relying on manually written prompts or seed questions.
  • Method: Unlike existing synthetic-data methods, MAGPIE does not rely on prompt engineering or seed questions and is extensible to multi-turn, preference-optimization, and domain-specific data.Existing approaches can suffer from reduced diversity as dataset size grows, whereas MAGPIE is designed for scalable self-synthesis from aligned LLMs.
  • Resources: MAGPIE-Air and MAGPIE-Pro were created using 206 and 614 GPU hours, respectively, without human intervention or access to production LLM APIs.The resulting datasets are comprehensively analyzed so practitioners can filter instances for their fine-tuning needs.
  • Empirical results: MAGPIE-aligned models outperform official Llama-3-8B-Instruct on AlpacaEval 2, while MAGPIE extensions combined with preference optimization can outperform GPT-4-Turbo(1106).MAGPIE also delivers the best results when combined with preference optimization methods such as DPO.

2 MAGPIE: A SCALABLE METHOD TO SYNTHESIZE ALIGNMENT DATA

MAGPIE synthesizes alignment data by prompting open-weight aligned LLMs to generate instructions from their pre-query templates, then generating corresponding responses. The method supports model-specific, filtered, multi-turn, preference-optimization, domain-specific, and multilingual datasets.

  • Overview of MAGPIE: MAGPIE generates alignment data in two steps: it elicits instructions from a crafted pre-query template, then prompts the aligned LLM to produce corresponding responses.The resulting dataset combines instruction-provider and follower roles with the generated instructions and responses.
  • Applicability of MAGPIE: MAGPIE applies to open-weight models including Llama-3, Llama-3.1, Qwen2, Gemma-2, and Phi-3.The method is described as readily deployable across these state-of-the-art model families.
  • Dataset Filtering: Users can filter MAGPIE’s raw generations with customizable metrics and off-the-shelf configurations to select instruction data for their needs.The method provides eight available metrics and six off-the-shelf filter configurations.
  • Generating Multi-Turn Instruction Datasets: MAGPIE extends to multi-turn instruction datasets by appending the pre-query template after each prior conversation, although 8B models may occasionally lose the user role.The first turn follows the standard instruction and response-generation steps, while subsequent turns continue from the previous prompt.
  • Extensions: MAGPIE also supports preference-optimization data and controlled domain-specific or multilingual generation through repeated response sampling, task selection, and tailored system prompts.Specialized models such as DeepSeek-Coder-V2 and Qwen2-Math-7B-Instruct can further provide domain-specific capabilities.

3 DATASET ANALYSIS

MAGPIE produces broad, mostly high-quality and safe instruction datasets with diverse task coverage. Its fully automated pipeline scales to millions of examples without human intervention or access to advanced commercial APIs.

  • Attribute: Task Categories of Instructions: MAGPIE-Pro covers diverse instruction topics, with information seeking as the predominant category followed by creative writing, advice seeking, planning, and math.The category distribution is produced by annotating MAGPIE-Pro with Llama-3-8B-Instruct.
  • Attribute: Quality of Instructions: Both MAGPIE-Air and MAGPIE-Pro are high quality, with most instructions rated average or higher.Quality is assessed using Llama-3-8B-Instruct across five categories from very poor to excellent; the distributions are shown in Figure 3-(a).
  • Attribute: Difficulty of Instructions: The two datasets have similar difficulty distributions, although MAGPIE-Pro contains some more challenging instructions.Difficulty is rated from very easy to very hard, with the distributions presented in Figure 3-(b).
  • Safety Analysis: Both MAGPIE-Air and MAGPIE-Pro are predominantly safe, with fewer than 1% of examples potentially containing harmful instructions or responses.Safety is analyzed using Llama-Guard-2.
  • Generation Efficiency: MAGPIE’s fully automated pipeline generated 3M MAGPIE-Air and 1M MAGPIE-Pro examples without human intervention or access to advanced commercial APIs.Generating instructions and responses took 1.55 and 50 hours for MAGPIE-Air, and 3.5 and 150 hours for MAGPIE-Pro, respectively.

4 PERFORMANCE ANALYSIS

MAGPIE-generated instruction data consistently outperforms public baselines across instruction-following benchmarks and can match or exceed official aligned models using substantially less data. The gains transfer across Llama and Qwen backbones, while reasoning performance requires supplementary data because reasoning instructions are underrepresented.

  • Evaluation Benchmarks: The evaluation uses AlpacaEval 2 and Arena-Hard, whose GPT-based assessments include 805 representative instructions and 500 challenging queries, respectively.Both benchmarks assess instruction-following responses using a GPT evaluator; win rate and length-controlled win rate are the reported metrics.
  • MAGPIE datasets outperform baselines with SFT only: MAGPIE fine-tuning outperforms baseline SFT datasets on both AlpacaEval 2 metrics and consistently across WildBench task categories.Table 1 reports the Llama-3 comparison, while Figure 5 breaks down MAGPIE-Pro and baselines by WildBench category.
  • Models aligned with data generated by MAGPIE achieve comparable or even higher performance to the official aligned model: MAGPIE-trained Llama-3-8B surpasses the official Llama-3-8B-Instruct model on AlpacaEval 2 and Arena-Hard, and even exceeds GPT-4-Turbo(1106) on AlpacaEval 2.The MAGPIE alignment process uses no more than 400K data, compared with more than 10M samples for official aligned models.
  • MAGPIE can enhance the performance of other backbone models: On Qwen2-1.5B, Qwen1.5-4B, and Qwen1.5-7B, MAGPIE fine-tuning outperforms official models trained with both supervised fine-tuning and preference tuning.Table 2 evaluates Qwen models using MAGPIE-Pro-300K-Filtered against official instruction-tuned models.
  • Performance of MAGPIE on More Benchmarks: MAGPIE-Air and MAGPIE-Pro are evaluated on MMLU, ARC Challenge, HellaSwag, TruthfulQA, WinoGrande, and GSM8K through the Huggingface Open LLM Leaderboard tasks.The evaluation covers both general knowledge and reasoning-oriented benchmarks.
  • Performance of MAGPIE on More Benchmarks: MAGPIE performance may degrade on reasoning tasks because reasoning instructions are scarce, so a supplementary 150K math, code, and reasoning booster dataset is added.The booster is combined with MAGPIE-Pro-300K-Filtered to create MAGPIE-Pro-Mix-Filtered, with results reported in Table 3.

5 RELATED WORK

Prior alignment work uses instruction and preference tuning, with performance strongly dependent on instruction-data quality. Existing alignment-data construction methods rely on human interactions, synthetic generation, data transformation, or mixtures, while related work shows that language models can also leak memorized training data.

  • LLM Alignment: Instruction tuning and preference tuning are widely used to align LLM responses with human values, and instruction-tuning performance depends heavily on instruction-data quality.Instruction tuning fine-tunes models on single- or multi-turn instructions and desired responses, while preference tuning uses RLHF or preference optimization.
  • Alignment Dataset Construction: Existing alignment datasets are constructed through human interactions with LLMs or synthetic instruction generation.Prior work collected human–LLM interactions, while manually crafting instructions is time-consuming and labor-intensive and may introduce toxic content.
  • Alignment Dataset Construction: Transforming existing data and combining or selecting high-quality instruction data provide additional construction strategies, but limited task variety can impede real-world generalization.Mixture datasets combine or select instruction data, whereas transformed datasets may remain constrained by their NLP-task coverage.
  • Training Data Extraction: Training-data extraction studies show that language models can memorize training examples, potentially allowing malicious users to extract private information.Prior work demonstrated private-data extraction from BERT, GPT-2, and ChatGPT.

6 LIMITATIONS, DISCUSSIONS, AND ETHICAL CONSIDERATIONS

MAGPIE-aligned LLMs perform strongly on instruction-following benchmarks but lag official models on math and reasoning. Raw MAGPIE data may contain harmful instructions or responses, though empirical evaluations find such instances constitute less than 1%.

  • Limitations and Discussions: MAGPIE-aligned LLMs show strong instruction-following performance but degrade on math and reasoning benchmarks relative to official Llama-3-8B-Instruct.Specialized booster reasoning datasets reduce but do not eliminate the performance gap.
  • Limitations and Discussions: Even with specialized booster reasoning datasets, MAGPIE-aligned models retain a performance gap on reasoning tasks compared with official models.
  • Societal Impact and Potential Harmful Consequences: Raw MAGPIE data may contain harmful instructions or responses that could produce unsafe behaviors when used for instruction tuning.Empirical evaluations indicate that harmful data instances constitute less than 1%.

7 CONCLUSION

MAGPIE is a scalable method for synthesizing instruction data to fine-tune large language models. It uses open-weight LLM instruction templates and autoregressive generation to create instructions and corresponding responses.

  • MAGPIE provides a scalable method for synthesizing instruction data for fine-tuning large language models.
  • MAGPIE leverages predefined instruction templates from open-weight LLMs and prompts the model only to act as an instruction provider.
  • Because of autoregressive generation, the prompted LLM produces detailed instructions from the crafted prompt.
  • MAGPIE then sends the generated instructions back to the LLM to produce corresponding responses.

A STATISTICS OF INSTRUCTION DATASETS GENERATED BY MAGPIE COMPARED TO OTHER INSTRUCTION DATASETS.

MAGPIE can generate diverse, high-quality instruction–response datasets across several state-of-the-art open-weight model families. Its dataset family exceeds 11.4 million instances, making it the largest alignment dataset described here without human-written questions or complex multi-stage pipelines.

  • Model coverage: MAGPIE datasets can be generated from Llama-3, Llama-3.1, Qwen2, Gemma-2, and Phi-3 open-weight model families.The method is presented as readily deployable across these state-of-the-art model families.
  • Dataset scale: The MAGPIE dataset family contains over 11.4 million diverse, high-quality instructions and corresponding responses.These datasets are generated from state-of-the-art open-source models.
  • Comparison scope: MAGPIE is described as the largest alignment dataset that avoids human-written questions and complex multi-stage pipelines.Table 4 compares MAGPIE-family statistics with other instruction datasets, counting tokens with the tiktoken library.

B MAGPIELM … D.2 ADDITIONAL SAFETY ANALYSIS

The paper extends MAGPIE into MAGPIELM, provides configurable filtering and dataset analyses, and finds broad coverage with minimal potentially harmful queries. MAGPIELM significantly outperforms similarly sized baselines across multiple alignment benchmarks.

  • B MAGPIELM: MAGPIELM significantly outperforms similarly sized baselines across Alpaca Eval 2, Arena Hard, and Wildbench.Figure 6 reports results for MAGPIELM-4B-Chat and MAGPIELM-8B-Chat against baselines of similar sizes.
  • B MAGPIELM: MAGPIELM uses 550K instructions for SFT and 200K for DPO, with responses generated by Gemma-2-9b-it across diverse categories.Both datasets select high-quality instructions from the MAGPIE family.
  • C FILTER SETUPS: MAGPIE filtering exposes input length, output length, task category, input quality, and input difficulty for dataset customization.The paper also provides off-the-shelf filter configurations, with output length applied last to select the k longest responses; experiments set τ1 = −12 and τ2 = 0.
  • D MORE DATASET ANALYSIS: Additional dataset analysis reports instruction and response lengths, while MAGPIE-Air’s common root verbs and noun objects indicate diverse topic coverage.These statistics are illustrated in Figure 7, and the root-verb visualization uses the MAGPIE-Air dataset.
  • D.1 ADDITIONAL ANALYSIS ON DATASET COVERAGE AND ATTRIBUTES.: MAGPIE-Pro’s t-SNE coverage encompasses Alpaca, Evol Instruct, and UltraChat, suggesting broader or more diverse topic coverage.The comparison randomly samples 10,000 instructions from each dataset.
  • D.1 ADDITIONAL ANALYSIS ON DATASET COVERAGE AND ATTRIBUTES.: MAGPIE-Pro and MAGPIE-Air have largely similar task-category distributions, but MAGPIE-Pro contains more creative-writing tasks.The distributions are labeled by Llama-3-Instruct.
  • D.2 ADDITIONAL SAFETY ANALYSIS: Potentially harmful queries comprise less than 1% of both MAGPIE-Air and MAGPIE-Pro, while unsafe responses mainly involve specialized advice.The specialized-advice category includes potentially risky financial, medical, or legal guidance and claims that dangerous activities or objects are safe.

D.3 ABLATION ANALYSIS ON GENERATION CONFIGURATIONS · D.4 IMPACT OF ANNOTATING MODELS

The ablations show that decoding choices trade instruction quality against difficulty and diversity, while system prompts generally reduce quality. MAGPIE-Air’s quality and difficulty remain high across annotators, exceeding Llama-3-8B-Instruct’s original ratings when assessed by Qwen-2-7B-Instruct.

  • D.3 ABLATION ANALYSIS ON GENERATION CONFIGURATIONS: The decoding ablation evaluates instruction quality, difficulty, and diversity using Average Quality Score, Average Difficulty Score, and Average Minimum Neighbor Distance.Quality and difficulty ratings use scales from 1 to 5, ranging from very poor to excellent and very easy to very hard, respectively.
  • D.3 ABLATION ANALYSIS ON GENERATION CONFIGURATIONS: Higher temperature and top-p may slightly reduce instruction quality while increasing difficulty and markedly enhancing diversity.The study varies temperature across 1, 1.1, and 1.2 and top-p across 1, 0.995, and 0.99 during MAGPIE Step 1.
  • D.3 ABLATION ANALYSIS ON GENERATION CONFIGURATIONS: The preferred decoding hyperparameters should be tailored to requirements by balancing quality, difficulty, and diversity.The observed trade-off means configurations favoring greater difficulty and diversity can incur lower overall quality.
  • D.3 ABLATION ANALYSIS ON GENERATION CONFIGURATIONS: Using a system prompt in MAGPIE Step 1 generally decreases the overall quality of generated instructions compared with omitting one.The comparison uses Vicuna’s default prompt because Llama-3 has no official system prompt.
  • D.4 IMPACT OF ANNOTATING MODELS: Qwen-2-7B-Instruct was used to annotate MAGPIE-Air quality and difficulty to test whether annotator-model identity affects evaluation.The experiment addresses the possibility that LLMs may occasionally favor their own responses.
  • D.4 IMPACT OF ANNOTATING MODELS: Qwen-2-7B-Instruct ratings show MAGPIE-Air retains high quality and difficulty, exceeding the original Llama-3-8B-Instruct annotations and indicating robustness across annotators.The annotator was selected outside the Llama-3 family to evaluate possible self-preference effects.

E DETAILED EXPERIMENTAL SETUPS … F.4 ABLATION ANALYSIS ON FILTER DESIGNS

The experiments show that MAGPIE benefits from multi-turn data, larger and higher-quality filtered datasets, and seed-free diversity, while performance varies across filter designs and decoding choices. Evaluation uses specified greedy-decoding settings for generation, fine-tuning, preference tuning, and benchmarks.

  • E DETAILED EXPERIMENTAL SETUPS: Figure 13 compares different annotators for evaluating instruction quality and difficulty.
  • E.1 EXPERIMENTAL SETUPS FOR GENERATING MAGPIE-AIR AND MAGPIE-PRO: Step 1 decoding parameters significantly affect instruction quality, difficulty, and diversity; MAGPIE-Air and MAGPIE-Pro therefore use diverse configurations to balance these attributes.The configurations are presented in Table 7.
  • E.1 EXPERIMENTAL SETUPS FOR GENERATING MAGPIE-AIR AND MAGPIE-PRO: Responses for MAGPIE-Air and MAGPIE-Pro are generated with greedy decoding because the highest-probability word is more likely to originate from the model’s training data.
  • E.2 EXPERIMENTAL SETUPS FOR INSTRUCTION TUNING AND PREFERENCE TUNING: The experiments specify supervised fine-tuning and DPO hyper-parameters, using Axolotl for SFT and Alignment Handbook for preference tuning.Evaluation uses greedy decoding for Arena-Hard and WildBench, and greedy decoding with RP = 1.2 for AlpacaEval 2.
  • F.1 PERFORMANCE OF MAGPIE-MT: Multi-turn MAGPIE-Air-MT and MAGPIE-Pro-MT improve performance over their single-turn counterparts, particularly on Arena-Hard.All models are instruction-tuned on Llama-8B base models.
  • F.2 COMPARE MAGPIE AND SELF-INSTRUCT USING LLAMA-3-8B-INSTRUCT: MAGPIE significantly outperforms Self-Instruct across all benchmarks, demonstrating diverse, high-quality instruction generation without seed questions.The comparison uses 100K datasets generated by Llama-3-8B-Instruct and fine-tunes Llama-8B base models.
  • F.3 ABLATION ANALYSIS ON DATA QUANTITY AND QUALITY: Increasing dataset size improves instruction-following performance, while MAGPIE-Pro-300K-Filtered outperforms datasets of equal or larger size.MAGPIE-Pro is generated by Llama-3-70B-Instruct.
  • F.4 ABLATION ANALYSIS ON FILTER DESIGNS: Different MAGPIE-Pro filtering strategies are optimal on different benchmarks, so no single filter consistently produces the best supervised-fine-tuning performance.The filter-design comparison is reported in Table 13 using Llama-8B base models.

F.5 ABLATION ANALYSIS ON RESPONSE GENERATOR … H MAGPIE EXAMPLES

MAGPIE remains effective when its response generator is replaced, while using only 300K examples yields competitive trustworthiness against Llama-3-8B-Instruct trained with over 10M samples. The paper also specifies prompt templates for extending and evaluating MAGPIE and illustrates broad, multilingual, and domain-specific instruction generation.

  • F.5 ABLATION ANALYSIS ON RESPONSE GENERATOR: Despite slight degradation, Qwen-2-7B-Instruct as MAGPIE’s response generator outperforms all baselines, including GPT-4-based variants.This suggests MAGPIE depends little on the specific response generator and that its instructions are high-quality and diverse.
  • F.6 TRUSTWORTHINESS OF MAGPIE-ALIGNED MODELS: With 300K samples versus over 10M, MAGPIE’s model slightly trails Llama-3-8B-Instruct on safety and fairness but surpasses it on ethics, privacy, and robustness.The comparison uses the TrustLLM benchmark across safety, fairness, ethics, privacy, and robustness.
  • G.1 PROMPT TEMPLATES FOR MAGPIE EXTENSION: MAGPIE-MT and controlled instruction tasks are generated with prompt templates that use first-turn instruction and response placeholders and domain-specific system prompts.The templates illustrate control for mathematics, coding, translation, and multilingual tasks.
  • G.2 PROMPT TEMPLATES FOR EVALUATION: Evaluation prompt templates generate task categories, instruction quality, and difficulty from an input instruction.These templates are presented in Figures 16–18.
  • H MAGPIE EXAMPLES: MAGPIE also produces domain-specific instructions with DeepSeek-Coder-V2 (Zhu et al., 2024) and Qwen2-Math-7B-Instruct (Yang et al., 2024).The examples include a scaled-rectangle geometry problem and area comparison.
  • H MAGPIE EXAMPLES: MAGPIE extends beyond English to multilingual datasets using Qwen2-72B-Instruct, with examples in Chinese, Spanish, Italian, and Portuguese.The examples cover bitwise programming, customer-satisfaction analysis, phrase interpretation, and creative dialogue.
Loading 2406.08464v2…