PaperlayerView exact revision on arXiv

Source-linked AI summary

Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothing

Zhangchen Xu, Fengqing Jiang, Luyao Niu, Yuntian Deng, Radha Poovendran, Yejin Choi, Bill Yuchen Lin

arXiv:2406.08464v2cs.CLcs.AI

TL;DR

High-quality alignment datasets often remain private, and synthesizing diverse instruction data at scale remains challenging. MAGPIE generates instruction–response pairs by prompting aligned LLMs with only pre-query templates, then requesting responses. Its data outperform prior public datasets in supervised fine-tuning and preference optimization, while supporting performance comparable to official Llama-3-8B-Instruct with less data.

  • Problem

    Instruction tuning needs high-quality data, but alignment datasets often remain private and scalable synthesis of diverse instruction data remains challenging.

    Source: §1 · pp. 1–2

  • Method

    MAGPIE elicits instructions from aligned LLMs using only pre-query templates, then asks the models to generate corresponding responses.

    Source: §1 · p. 2, §7 · p. 11

  • Results

    MAGPIE outperforms prior public instruction datasets with supervised fine-tuning alone and achieves the best results when combined with preference optimization.

    Source: §1 · p. 3

  • Takeaways & Limitations

    MAGPIE-generated data support alignment comparable to or better than official Llama-3-8B-Instruct using less data.

    Source: §4.2 · p. 8

    MAGPIE-aligned models underperform official models on math and reasoning benchmarks, even after adding specialized reasoning data.

    Source: §6 · p. 11

Abstract

from arXiv · show

High-quality instruction data is critical for aligning large language models (LLMs). Although some models, such as Llama-3-Instruct, have open weights, their alignment data remain private, which hinders the democratization of AI. High human labor costs and a limited, predefined scope for prompting prevent existing open-source data creation methods from scaling effectively, potentially limiting the diversity and quality of public alignment datasets. Is it possible to synthesize high-quality instruction data at scale by extracting it directly from an aligned LLM? We present a self-synthesis method for generating large-scale alignment data named Magpie. Our key observation is that aligned LLMs like Llama-3-Instruct can generate a user query when we input only the left-side templates up to the position reserved for user messages, thanks to their auto-regressive nature. We use this method to prompt Llama-3-Instruct and generate 4 million instructions along with their corresponding responses. We perform a comprehensive analysis of the extracted data and select 300K high-quality instances. To compare Magpie data with other public instruction datasets, we fine-tune Llama-3-8B-Base with each dataset and evaluate the performance of the fine-tuned models. Our results indicate that in some tasks, models fine-tuned with Magpie perform comparably to the official Llama-3-8B-Instruct, despite the latter being enhanced with 10 million data points through supervised fine-tuning (SFT) and subsequent feedback learning. We also show that using Magpie solely for SFT can surpass the performance of previous public datasets utilized for both SFT and preference optimization, such as direct preference optimization with UltraFeedback. This advantage is evident on alignment benchmarks such as AlpacaEval, ArenaHard, and WildBench.

1 INTRODUCTION

MAGPIE generates instruction data from aligned LLMs using only pre-query templates, without seed questions or prompt engineering. MAGPIE-aligned models outperform Llama-3-8B-Instruct with supervised fine-tuning and GPT-4-Turbo(1106) with preference optimization on AlpacaEval 2.

1 INTRODUCTION

  • Private alignment datasets limit open-model research, while existing synthetic methods depend on prompt engineering and seed questions and lose diversity as datasets grow.High-quality instruction data is crucial for instruction tuning, but human data creation is time-consuming and labor-intensive (Liu et al., 2024a).Source: §1 · pp. 1–2
  • MAGPIE prompts aligned LLMs with only pre-query templates to generate instructions, eliminating prompt engineering and seed questions.Autoregressive continuation produces high-quality, diverse user queries. The method extends to multi-turn, preference-optimization, domain-specific, and multilingual datasets.Source: §1 · p. 2
  • 206 GPU hours produced MAGPIE-Air, while 614 GPU hours produced MAGPIE-Pro, without human intervention or production-LLM API access.MAGPIE-Air and MAGPIE-Pro use Llama-3-8B-Instruct and Llama-3-70B-Instruct, respectively. Dataset analysis supports filtering and selecting instances for practitioners’ fine-tuning needs.Source: §1 · p. 2
  • MAGPIE SFT surpasses prior public instruction datasets and the official Llama-3-8B-Instruct model on AlpacaEval 2.It also surpasses baselines combining SFT and DPO (Rafailov et al., 2023) with UltraFeedback (Cui et al., 2023). The official model used over 10 million data points for SFT and subsequent preference optimization; evaluation also covered Arena-Hard and WildBench.Source: §1 · pp. 2–3
  • MAGPIE-generated preference data enables aligned Llama-3 models to outperform GPT-4-Turbo(1106) on AlpacaEval 2.MAGPIE also delivers the best reported results when combined with preference optimization methods such as DPO.Source: §1 · p. 3

2 MAGPIE: A SCALABLE METHOD TO SYNTHESIZE ALIGNMENT DATA

MAGPIE automatically synthesizes instruction-response pairs by prompting an aligned LLM with its pre-query chat template, then requesting responses to the generated instructions. Extensions support dataset filtering, multi-turn conversations, preference optimization, and domain-specific or multilingual generation.

2.1 MAGPIE PIPELINE

  • MAGPIE generates instructions from an aligned LLM’s pre-query chat template, then feeds those instructions back to the model to produce responses without human intervention.Instruction generation requires neither seed questions nor specialized prompt engineering and stops at an end-of-sequence token. Pairing instructions and responses with user and assistant roles forms the dataset.Source: §2.1 · p. 3, §2 · p. 3
  • MAGPIE supports open-weight model families including Llama-3, Qwen2 (Yang et al., 2024), Gemma-2 (Team et al., 2024), and Phi-3 (Abdin et al., 2024).It generates high-quality instructions even when instruction loss is masked during alignment. The authors hypothesize that models retain implicit memory of instruction distributions, leaving this explanation for future research.Source: §2.1 · p. 3

2.2 MAGPIE EXTENSIONS

  • Eight available metrics support customized filtering of raw MAGPIE data, alongside six off-the-shelf filter configurations.Practitioners can select data according to their needs; the paper discusses the preset configurations’ performance in Appendix F.4.Source: §2.2 · pp. 3–4
  • MAGPIE-MT generates subsequent conversation turns by appending the pre-query template to the full preceding conversation and generating another instruction-response pair.Models occasionally forget their user role, especially the 8B model. A system prompt reinforces that role and the multi-round conversation context.Source: §2.2 · p. 4
  • Preference-data generation samples multiple responses per selected instruction and labels the highest reward-model score as chosen and the lowest as rejected.The method selects a small, high-quality instruction subset spanning diverse task categories, then samples k responses per instruction at temperature T < 1.Source: §2.2 · p. 4
  • Tailored system prompts control the task category and language of generated instructions, while specialized models offer another route to domain-specific and multilingual data.System prompts describe the chatbot’s domain and expected queries. Specialized sources include DeepSeek-Coder-V2 (Zhu et al., 2024) for code and Qwen2-Math-7B-Instruct (Yang et al., 2024) for math.Source: §2.2 · p. 4

3 DATASET ANALYSIS

MAGPIE-Pro shows higher instruction quality than MAGPIE-Air and broad topic coverage, while both datasets contain less than 1% potentially harmful data. Generation is fully automated, with estimated cloud costs of $0.12 and $1.1 per 1,000 instances for MAGPIE-Air and MAGPIE-Pro, respectively.

3.1 DATASET COVERAGE

  • MAGPIE-Pro’s embedding projection encompasses all three synthetic baselines, suggesting broader topic coverage; common verb–object patterns also indicate diverse topics.The baselines are Alpaca, Evol Instruct (Xu et al., 2023a), and UltraChat (Ding et al., 2023). Coverage is examined using all-mpnet-base-v2 embeddings projected into two dimensions with t-SNE (Van der Maaten & Hinton, 2008).Source: §3.1 · p. 5

3.2 DATASET ATTRIBUTES

  • Over half of MAGPIE-Pro tasks involve information seeking, followed by creative writing, advice seeking, planning, and math.Llama-3-8B-Instruct supplies the task classifications; the reported distribution aligns with practical human requests.Source: §3.2 · p. 5
  • Most instructions in both datasets are rated average or higher by Llama-3-8B-Instruct, with MAGPIE-Pro surpassing MAGPIE-Air in overall quality.Instruction quality is assessed on a five-level scale from very poor to excellent.Source: Figure 3 · p. 5, §3.2 · p. 5

3.2 DATASET ATTRIBUTES / 3.5 ADDITIONAL ANALYSIS

  • Instruction difficulty distributions are similar across datasets, with some harder MAGPIE-Pro examples; additional analyses examine generation configurations and annotation-model choices.Llama-3-8B-Instruct rates difficulty from very easy to very hard. Appendices D.3 and D.4 examine configuration effects on instruction quality and difficulty, and ablate models used to assess those attributes.Source: Figure 3 · p. 5, §3.2 · p. 5, §3.5 · p. 6

3.2 DATASET ATTRIBUTES

  • Repetitive instructions are removed using nearest-neighbor distances in embedding space.Instructions are embedded with all-mpnet-base-v2, and FAISS (Douze et al., 2024) calculates minimum neighbor distances; the analysis summarizes MAGPIE-Air distances after repetition removal.Source: §3.2 · p. 6
  • Response quality is assessed through reward differences against base-model responses, with positive differences indicating higher-quality dataset responses.FsfairX-LLaMA3-RM-v0.1 scores both responses to the same instruction, while URIAL (Lin et al., 2023) elicits the base-model response. Positive differences could indicate potential benefits for instruction tuning; the supplied passage does not report their distribution.Source: §3.2 · p. 6

3.3 SAFETY ANALYSIS

  • Less than 1% of either dataset potentially contains harmful instructions or responses, according to analysis using Llama-Guard-2 (Team, 2024).Both MAGPIE-Air and MAGPIE-Pro are therefore characterized as predominantly safe.Source: §3.3 · p. 6

3.4 COST ANALYSIS

  • $0.12 for MAGPIE-Air and $1.1 for MAGPIE-Pro per 1,000 instances are estimated cloud costs for the fully automated pipeline.On four NVIDIA A100-SXM4-80GB GPUs, instruction and response generation took 1.55 and 50 hours for 3M MAGPIE-Air instances, versus 3.5 and 150 hours for 1M MAGPIE-Pro instances. The pipeline requires neither human intervention nor access to advanced commercial-model APIs.Source: §3.4 · p. 6

4 PERFORMANCE ANALYSIS

MAGPIE-generated data produces Llama-3 models that outperform public alignment baselines and, with SFT+DPO, official Llama-3-8B-Instruct using at most 400K alignment examples. Gains extend to Qwen backbones, while targeted supplementary data addresses reasoning weaknesses.

4.1 EXPERIMENTAL SETUPS

  • 300K-conversation raw and filtered MAGPIE-Air/Pro datasets underpin SFT experiments, with separate 100K-conversation datasets for DPO.Air and Pro use Llama-3-8B-Instruct and Llama-3-70B-Instruct, respectively; DPO data generation uses k = 5 and T = 0.8. Evaluation uses GPT-judged AlpacaEval 2 and Arena-Hard, reporting WR on both and length-controlled LC only on AlpacaEval 2.Source: §4.1 · p. 7

4.2 EXPERIMENTAL RESULTS

  • 400K or fewer alignment examples yield an SFT+DPO model outperforming Llama-3-8B-Instruct on AlpacaEval 2 and Arena-Hard, and GPT-4-Turbo(1106) on AlpacaEval 2.Official alignment uses more than 10M samples. Even SFT-only MAGPIE models exceed 50% LC on AlpacaEval 2 when evaluated against Llama-3-8B-Instruct.Source: Table 1 · p. 8, §4.2 · pp. 8–9
  • MAGPIE SFT models surpass all compared SFT baselines on AlpacaEval 2 LC and WR, show similar Arena-Hard gains, and outperform preference-optimized baselines.These comparisons demonstrate that MAGPIE’s gains do not require preference optimization.Source: Table 1 · p. 8, §4.2 · p. 8
  • MAGPIE-Pro consistently outperforms baseline datasets across WildBench task categories.WildBench contains 1024 tasks selected from real-world human–LLM conversation logs.Source: Figure 5 · p. 8, §4.2 · p. 8
  • MAGPIE-tuned Qwen2-1.5B, Qwen1.5-4B, and Qwen1.5-7B outperform their officially aligned counterparts, extending the improvements beyond Llama-3.SFT+DPO also produces MagpieLM using Llama-3.1-Minitron-4B-Width-Base (Sreenivas et al., 2024) and Llama-3.1-8B-Instruct (Dubey et al., 2024). MagpieLM ranks first among popular open-source instruction models with fewer than 10 billion parameters.Source: Table 2 · p. 9, §4.2 · p. 9
  • 150K supplementary math, code, and reasoning instructions address MAGPIE’s reasoning weakness when mixed with MAGPIE-Pro-300K-Filtered, placing the SFT model among the top three checkpoints.MAGPIE-Air/Pro otherwise perform comparably to official instruct and baseline models across Open LLM Leaderboard tasks and zero-shot MMLU-Redux (Gema et al., 2024). The mixed model performs slightly below OpenHermes 2.5 (1M conversations) and Llama-3-8B-Instruct (>10M conversations).Source: §4.2 · p. 9

5 RELATED WORK

Magpie generates synthetic alignment data using prompt templates without seed questions or prompt engineering. Unlike private training-data extraction, it aims to create publicly available alignment datasets with minimal human effort.

LLM Alignment

Alignment Dataset Construction

  • Magpie belongs to synthetic alignment-data generation but requires neither seed questions nor prompt engineering, relying instead on the prompt template.This distinguishes it from approaches that expand human-annotated seed instructions through few-shot prompting.Source: §5 · p. 10
  • Seed-based synthesis risks instructions resembling seed questions (Li et al., 2024a), while collecting human interactions is time-consuming and labor-intensive.Manually crafted instructions may also contain toxic content. Knowledge-based synthesis seeks broader coverage (Ding et al., 2023; Li et al., 2024a).Source: §5 · p. 10
  • Transforming existing datasets provides alignment data, but limited NLP-task variety may hinder real-world generalization (Li et al., 2024a).Examples of this construction approach include Wang et al. (2022), Sanh et al. (2022), and Gandhi et al. (2024).Source: §5 · p. 10
  • Mixture datasets combine or select existing instruction data to improve coverage (Ivison et al., 2023) or performance (Liu et al., 2024b).Reasoning- and math-focused data construction methods (Yue et al., 2023; 2024) could also be combined with Magpie.Source: §5 · p. 10

Training Data Extraction

6 LIMITATIONS, DISCUSSIONS, AND ETHICAL CONSIDERATIONS

MAGPIE-aligned models show strong instruction-following performance but retain math and reasoning gaps relative to official models. Generated data may contain harmful content, which filtering can identify and remove to mitigate safety risks.

Limitations and Discussions

  • MAGPIE-aligned models show strong instruction-following performance against Llama-3-8B-Instruct but exhibit degraded math and reasoning performance.Specialized booster reasoning datasets do not eliminate the performance gap with official models; improving reasoning remains a future research direction.Source: §6 · p. 11

Societal Impact and Potential Harmful Consequences

  • Less than 1% of the dataset contains harmful instructions or responses, which may induce unsafe behavior when used raw for instruction tuning.This proportion comes from the authors’ empirical evaluations.Source: §6 · p. 11
  • The Appendix C filtering technique can identify and remove harmful data instances, mitigating risks associated with MAGPIE-generated instruction data.The method’s stated objective is scalable instruction-data synthesis to improve instruction following and alignment with human values.Source: §6 · p. 11

7 CONCLUSION

MAGPIE synthesizes instruction–response pairs from role-only prompts and filters them for instruction tuning. Fine-tuning Llama-3-8B on the selected data outperformed all tested baselines and the official Llama-3-8B-Instruct model.

7 CONCLUSION

  • Llama-3-8B fine-tuned on selected MAGPIE data outperformed all tested baselines and the official Llama-3-8B-Instruct model.The official model underwent instruction tuning and preference optimization using more than 10M data instances.Source: §7 · p. 11
  • MAGPIE uses open-weight LLM instruction templates specifying only the instruction-provider role to generate instructions autoregressively, then requests corresponding responses.The generated instruction–response pairs constitute the instruction dataset.Source: §7 · p. 11
  • MAGPIE uses Llama-3-8B-Instruct to label synthesized data and a filtering technique to select effective instruction-tuning examples.The selected examples are used to fine-tune the Llama-3-8B base model.Source: §7 · p. 11

A STATISTICS OF INSTRUCTION DATASETS GENERATED BY MAGPIE COMPARED TO OTHER INSTRUCTION DATASETS.

MAGPIE’s dataset family contains over 11.4 million instruction–response pairs generated from state-of-the-art open-source models. It spans multiple model families without relying on human-written questions or complex multi-stage pipelines.

Model coverage

Dataset scale

Dataset comparison

B MAGPIELM … D.2 ADDITIONAL SAFETY ANALYSIS

MAGPIELM outperforms similarly sized baselines across multiple benchmarks, while additional analyses characterize MAGPIE’s filtering options, topical coverage, length statistics, and safety. Both MAGPIE-Air and MAGPIE-Pro contain fewer than 1% potentially harmful queries.

B MAGPIELM

  • MAGPIELM-4B-Chat and MAGPIELM-8B-Chat significantly outperform similarly sized baselines across multiple benchmarks.Benchmarks include Alpaca Eval 2, Arena Hard, and Wildbench.Source: Figure 6 · p. 18, B MAGPIELM · p. 19
  • 550K SFT instructions and 200K DPO instructions are selected from MAGPIE, with corresponding responses generated using Gemma-2-9b-it (Team et al., 2024).The selected high-quality instructions cover diverse categories.Source: B MAGPIELM · p. 19

C FILTER SETUPS

  • Customizable filters use input length, output length, task category, input quality, input difficulty, minimum neighbor distance, reward, and reward difference.Neighbor distance helps remove repetitive instances, while reward can exclude low-quality responses such as repetitions or refusals. Provided configurations apply output-length filtering last, retaining the k longest responses, with empirical thresholds τ1 = −12 and τ2 = 0.Source: C FILTER SETUPS · p. 19, Table 5 · p. 19

D MORE DATASET ANALYSIS

  • Additional dataset statistics visualize instruction and response lengths for MAGPIE-Air and MAGPIE-Pro.Figure 7 presents these length statistics, complementing the dataset analysis in Section 3.Source: D MORE DATASET ANALYSIS · p. 19, Figure 7 · p. 20

D.1 ADDITIONAL ANALYSIS ON DATASET COVERAGE AND ATTRIBUTES.

  • MAGPIE-Pro’s t-SNE plot encompasses the areas covered by Alpaca, Evol Instruct, and UltraChat, suggesting broader topical coverage.Each dataset’s visualization uses 10,000 randomly sampled instructions.Source: §D.1 · p. 19, Figure 8 · p. 20
  • MAGPIE-Pro and MAGPIE-Air have largely similar task distributions, but MAGPIE-Pro contains more creative writing; MAGPIE-Air’s root-verb/object analysis also indicates diverse topics.Task categories were labeled by Llama-3-Instruct.Source: §D.1 · p. 20, Figure 9 · p. 20

D.2 ADDITIONAL SAFETY ANALYSIS

  • Less than 1% of instances in MAGPIE-Air and MAGPIE-Pro contain potentially harmful queries, while most unsafe responses involve specialized advice.Safety categories were labeled by Llama-Guard-2 (Team, 2024). Specialized advice includes financial, medical, or legal guidance and suggestions that dangerous activities or objects are safe.Source: §D.2 · pp. 20–21

D.3 ABLATION ANALYSIS ON GENERATION CONFIGURATIONS · D.4 IMPACT OF ANNOTATING MODELS

MAGPIE’s decoding settings trade instruction quality against difficulty and diversity, while adding a system prompt generally lowers quality and difficulty. MAGPIE-Air retains high quality and difficulty when evaluated by an annotator outside the Llama-3 family.

D.3 ABLATION ANALYSIS ON GENERATION CONFIGURATIONS

  • Higher temperature and top-p values may slightly reduce instruction quality while increasing difficulty and markedly enhancing diversity.The authors recommend selecting these parameters according to users’ requirements for quality, difficulty, and diversity.Source: Figure 11 · p. 21, §D.3 · p. 22
  • 3 temperatures—1, 1.1, and 1.2—and three top-p values—1, 0.995, and 0.99—were tested during MAGPIE’s Step 1 instruction generation.Source: §D.3 · p. 21
  • Three metrics characterized each decoding configuration: Average Quality Score, Average Difficulty Score, and Average Minimum Neighbor Distance.Quality ratings range from 1 (‘very poor’) to 5 (‘excellent’), and difficulty ratings from 1 (‘very easy’) to 5 (‘very hard’). Diversity averages the minimum neighbor distances defined in Section 3 across instructions generated with the same parameters.Source: §D.3 · p. 22
  • Adding a system prompt during instruction generation generally reduced quality and produced easier instructions, motivating its omission by default.Because Llama-3 lacks an official system prompt, the experiment used Vicuna’s default prompt.Source: Figure 12 · p. 22, §D.3 · p. 22

D.4 IMPACT OF ANNOTATING MODELS

  • Qwen-2-7B-Instruct, outside the Llama-3 family, independently annotated MAGPIE-Air’s quality and difficulty.The experiment addressed concerns that LLMs may favor their own responses (Deutsch et al., 2022).Source: §D.4 · p. 22
  • Qwen-2-7B-Instruct rated MAGPIE-Air’s quality and difficulty higher than Llama-3-8B-Instruct did, suggesting robustness across annotators.The dataset maintained high quality and difficulty under this alternative evaluator.Source: §D.4 · p. 22

E DETAILED EXPERIMENTAL SETUPS … F.4 ABLATION ANALYSIS ON FILTER DESIGNS

Matched 100K-example comparisons reveal a significant performance gap between MAGPIE and Self-Instruct (Wang et al., 2023), whose seed-defined patterns constrain instruction diversity. Multi-turn data, larger datasets, filtering, and stronger generators improve performance, but no filter performs best across every benchmark.

E DETAILED EXPERIMENTAL SETUPS

  • Experimental analyses examine annotator effects on instruction-quality and difficulty evaluations, alongside decoding effects on generated instruction quality, difficulty, and diversity.Figure 13 compares annotators but supplies no specific outcomes in the provided caption.Source: Figure 13 · p. 22, §E.1 · p. 22

E.1 EXPERIMENTAL SETUPS FOR GENERATING MAGPIE-AIR AND MAGPIE-PRO

  • MAGPIE-Air and MAGPIE-Pro use varied decoding parameters for instruction generation and greedy decoding for response generation.Instruction decoding balances quality, difficulty, and diversity; greedy response decoding reflects the intuition that the highest-probability word more likely originates from training data.Source: §E.1 · p. 22

E.2 EXPERIMENTAL SETUPS FOR INSTRUCTION TUNING AND PREFERENCE TUNING

  • Supervised fine-tuning uses Axolotl, preference tuning uses Alignment Handbook, and benchmark evaluations use greedy decoding.AlpacaEval 2 additionally uses RP = 1.2 to mitigate repetition. Arena-Hard and WildBench retain their default settings.Source: §E.2 · p. 23

F.1 PERFORMANCE OF MAGPIE-MT

  • MAGPIE-Air-MT and MAGPIE-Pro-MT improve performance over their respective single-turn counterparts, particularly on Arena-Hard.All compared models are instruction-tuned on Llama-8B base models.Source: Table 10 · p. 24, §F.1 · p. 24

F.2 COMPARE MAGPIE AND SELF-INSTRUCT USING LLAMA-3-8B-INSTRUCT

  • 100K-example datasets generated using Llama-3-8B-Instruct reveal a significant performance gap between MAGPIE and Self-Instruct (Wang et al., 2023).The comparison uses the first 100K MAGPIE-Air examples. Self-Instruct instructions follow seed-defined patterns, whereas MAGPIE generates diverse, high-quality instructions without seed questions.Source: Table 11 · p. 24, §F.2 · pp. 24–25

F.3 ABLATION ANALYSIS ON DATA QUANTITY AND QUALITY

  • MAGPIE-Pro-300K-Filtered outperforms models trained on equal or larger raw datasets; increasing dataset size improves performance, and MAGPIE-Pro consistently outperforms MAGPIE-Air.The authors attribute MAGPIE-Pro’s advantage to its stronger generator, Llama-3-70B-Instruct.Source: Table 12 · p. 25, §F.3 · p. 25

F.4 ABLATION ANALYSIS ON FILTER DESIGNS

  • Different MAGPIE-Pro filtering strategies perform best on different benchmarks; no single filter consistently achieves the best performance across all benchmarks.Selecting instructional data to improve supervised fine-tuning remains an open research question.Source: Table 13 · p. 25, §F.4 · p. 25

F.5 ABLATION ANALYSIS ON RESPONSE GENERATOR … H MAGPIE EXAMPLES

MAGPIE retains its advantage over baselines after changing response generators, while its trustworthiness results exceed Llama-3-8B-Instruct on ethics, privacy, and robustness but trail on safety and fairness. Templates and examples demonstrate multi-turn generation, task control, instruction evaluation, and domain-specific and multilingual outputs.

F.5 ABLATION ANALYSIS ON RESPONSE GENERATOR

  • Using Qwen-2-7B-Instruct (Yang et al., 2024) as response generator slightly reduces performance but still beats all baselines, including those using GPT-4 responses.The ablation uses MAGPIE-Air-300K-Filtered. The authors interpret the results as evidence of limited dependence on response-generator choice and high instruction quality and diversity.Source: §F.5 · pp. 25–26, Table 14 · p. 26

F.6 TRUSTWORTHINESS OF MAGPIE-ALIGNED MODELS

  • MAGPIE-Pro-300K-Filtered outperforms Llama-3-8B-Instruct on TrustLLM ethics, privacy, and robustness, but slightly underperforms on safety and fairness.The comparison uses 300K training samples for MAGPIE versus over 10M for the official instruct model, evaluated on TrustLLM (Huang et al., 2024).Source: Table 15 · p. 26, §F.6 · p. 26

G.1 PROMPT TEMPLATES FOR MAGPIE EXTENSION

  • The MAGPIE-MT generation template incorporates first-turn instruction and response placeholders, illustrated using Llama-3-8B-Instruct.Figure 14 presents this multi-turn generation prompt.Source: §G.1 · p. 26, Figure 14 · p. 27
  • Domain-specific system prompts steer Llama-3-8B-Instruct toward mathematics, coding, translation, and multilingual instruction generation.Other instruction tasks can be targeted by replacing the System Prompt placeholder with an appropriate domain-specific prompt.Source: Figure 15 · p. 27

G.2 PROMPT TEMPLATES FOR EVALUATION

  • Evaluation prompt templates assess instruction task category, quality, and difficulty, using the instruction being evaluated as the input placeholder.Figures 16, 17, and 18 present the respective templates.Source: §G.2 · p. 28, Figure 16 · p. 28, Figure 17 · p. 29

H MAGPIE EXAMPLES

  • MAGPIE examples range from everyday observations and interactive games to domain-specific and multilingual instructions.Llama-3-70B-Instruct supplies general examples. Specialized examples use DeepSeek-Coder-V2 (Zhu et al., 2024) and Qwen2-Math-7B-Instruct (Yang et al., 2024), while multilingual examples use Qwen2-72B-Instruct.Source: H MAGPIE EXAMPLES · pp. 30–31

Related papers

Loading 2406.08464v2…