Zephyr: Direct Distillation of LM Alignment

Introduction

• Distilled models can improve task accuracy but remain below their teachers and are often not intent aligned, leading to incorrect responses to user queries (Gudibande et al., 2023).Distilled supervised fine-tuning uses a more capable teacher’s outputs as supervised data for a student model.Source: §1 · p. 2
• MT-Bench and AlpacaEval scores correlate with human ratings and reflect a performance ordering from proprietary models to human-feedback-trained open models to distilled open models (Zheng et al., 2023).The authors say collecting human feedback for alignment at scale can be enormously costly, as in LLAMA2-CHAT (Touvron et al., 2023).Source: §1 · p. 2
• The dDPO approach uses AI Feedback from teacher ensembles as preference data, requires no human annotation or fine-tuning-time sampling, and trains in hours on 16 A100s.Zephyr-7B is first trained with dSFT on UltraChat, then dDPO on AI feedback from UltraFeedback.Source: §1 · p. 2
• Zephyr-7B, built from Mistral-7B, achieves performance comparable to 70B-parameter chat models aligned with human feedback.The paper reports improvements on both standard academic and conversational benchmarks and says preference learning is critical to these results.Source: §1 · p. 2
• The work targets helpfulness rather than safety, leaving harmful outputs and illegal advice unaddressed as a future research challenge.The authors note that distillation relies on publicly available model outputs, making it more challenging to curate synthetic safety data.Source: §1 · p. 2

2 RELATED WORK

• Following ChatGPT, LLaMA (Touvron et al., 2023) spurred work on efficient fine-tuning, longer contexts, retrieval-augmented generation, and quantization amid growth in open LLMs.Source: §2 · p. 2
• Zephyr starts from Mistral 7B because of its strong performance.Source: §2 · p. 3
• Earlier distilled models primarily focused on SFT, while Zephyr distills both SFT and preference optimization; contemporaneous Xwin-LM distilled preference optimization through PPO.WizardLM also explored methods beyond dSFT, and the paper compares these approaches in its experiments.Source: §2 · p. 3
• The paper evaluates with MTBench, AlpacaEval, and the HuggingFace Open LLM Leaderboard; MTBench uses GPT-4 to score multi-turn instructions, while AlpacaEval compares models pairwise using larger LLMs.The LMSYS chatbot arena instead uses crowdsourced, anonymous randomized battles and Elo rankings.Source: §2 · p. 3

Distilled Supervised Fine-Tuning (dSFT)

• dSFT trains a raw language model on teacher-generated instructions and responses, using iterative self-prompting to produce a dataset from diverse seed prompts.The teacher first answers each seed prompt, then refines the instruction based on its response; the resulting instruction-response pairs are used for supervised fine-tuning.Source: §3 · p. 4

AI Feedback through Preferences (AIF)

• AI feedback ranks outputs by prompting four models per instruction, having the teacher score their responses, and retaining the highest-scoring response with a random lower-scoring one.These prompt, preferred-response, and lower-ranked-response triples form the feedback dataset used for preference optimization.Source: §3 · p. 4

Distilled Direct Preference Optimization (dDPO)

• dDPO refines the dSFT model by directly optimizing preference rankings from static AI-feedback triples, without PPO’s reward-model training and current-policy sampling (Rafailov et al., 2023).For each triple, the training procedure computes preferred and lower-ranked response probabilities under both the dSFT and dDPO models, then backpropagates the preference objective.Source: §3 · p. 4

4 EXPERIMENTAL DETAILS

• Experiments fine-tune Mistral 7B with TRL, DeepSpeed ZeRO-3, and FlashAttention-2 on 16 A100s.Training uses AdamW without weight decay, bfloat16 precision, and typically takes 2–4 hours; parameter-efficient methods such as LoRA were not tested.Source: §4 · pp. 4–5

4.1 DATASETS

• UltraChat is filtered and corrected for dSFT, while UltraFeedback preferences pair each prompt’s highest-mean-score response with a randomly selected alternative.UltraChat starts with 1.47M dialogues; truecasing and helpfulness filters produce approximately 200k examples. UltraFeedback contains 64k prompts with four GPT-4-rated responses, and preferences are computed offline without reference-model sampling.Source: §4.1 · p. 5

4.2 EVALUATION

• Evaluation covers MT-Bench and AlpacaEval chat performance, plus Open LLM Leaderboard tasks intended to detect regressions in reasoning and truthfulness.MT-Bench uses 160 two-turn questions scored by GPT-4; AlpacaEval measures pairwise win-rate against text-davinci-003 on 805 questions. Comparisons include 7B and larger open models, as well as Claude 2, GPT-3.5-Turbo, and GPT-4.Source: §4.2 · pp. 5–6

4.3 DETAILS OF SFT TRAINING

• SFT runs for one to three epochs with a cosine schedule, peak learning rate 2e-5, global batch size 512, and 2048-token packed sequences.The schedule uses 10% warmup steps.Source: §4.3 · p. 6

4.4 DETAILS OF DPO TRAINING

• DPO runs for one to three epochs with a linear schedule, peak learning rate 5e-7, global batch size 32, and β = 0.1.The final Zephyr-7B model starts from the one-epoch SFT model and is further optimized for three DPO epochs.Source: §4.4 · p. 6

dDPO Improves Chat Capabilities

• Zephyr-7B sets a new state of the art among open 7B models on MT-Bench and AlpacaEval, outperforming dSFT models across both benchmarks and Xwin-LM-7B.Against Llama2-Chat 70B, it scores better on MT-Bench and is within two standard deviations on AlpacaEval; it performs worse than WizardLM-70B.Source: §5 · p. 6
• Zephyr-7B is competitive with GPT-3.5-Turbo and Claude 2 on AlpacaEval, but its MT-Bench performance is much worse in math and coding than in several other categories.The authors caution that AlpacaEval prompts may not represent real-world usage or advanced applications.Source: §5 · p. 7

dDPO Improves Academic Task Performance

• Zephyr-7B is reported to perform best among 7B models on the Open LLM Leaderboard academic benchmarks, with a large gap over the best dSFT and Xwin-LM dPPO models.Larger models do better on some knowledge-intensive tasks, but Zephyr reaches the performance of 40B-scale models.Source: §5 · p. 7, Table 2 · p. 7

Is Preference Optimization Necessary?

• Without initial SFT, models learn poorly from feedback; SFT improves both chat benchmarks, extra SFT on preferred UltraFeedback outputs adds no impact, and the full Zephyr configuration increases both scores.The paper says this replicates prior findings (Ouyang et al., 2022).Source: §5 · p. 7, Table 3 · p. 8

Does Overfitting Harm Downstream Performance?

• Despite perfect DPO training-set accuracy after one epoch, overfitting did not harm downstream performance; the strongest model used one SFT epoch and three DPO epochs, while longer SFT followed by DPO caused regression.The training and test accuracies and MT-Bench results are shown in Figure 3.Source: §5 · p. 7, Figure 3 · p. 8

• Zephyr-7B, based on Mistral-7B, sets a state of the art for 7B chat models and outperforms Llama2-Chat-70B on MT-Bench.Source: §6 · p. 8
• The method distills conversational capabilities from AI-feedback data with direct preference optimization, avoiding sampling-based approaches such as rejection sampling or PPO.Source: §6 · p. 8
• GPT-4 evaluation of AlpacaEval and MT-Bench may favor models distilled from GPT-4 or verbose responses that are potentially incorrect.Source: §6 · p. 8
• Whether the method scales to much larger models such as Llama2-70B remains unexamined.The authors note that performance gains could potentially be larger at that scale.Source: §6 · p. 8

A.1 QUALITATIVE EXAMPLES

• The comparison uses prompts from several MT-Bench domains and adversarial prompts testing false premises or harmful intent.Adversarial completions use nucleus sampling with top-p = 0.95 and T = 0.7.Source: §A.1 · p. 12
• The examples are qualitative comparisons of selected prompts, including cherry-picked cases, rather than a comprehensive safety evaluation.Source: §A.1 · p. 12, Figure 4 · p. 12, Figure 5 · p. 12
• Figure 4 presents a cherry-picked MT-Bench example of the dDPO model following math instructions.Source: Figure 4 · p. 12
• In a cherry-picked false-premise example, the dSFT model refers to an “adult-sized helicopter,” illustrating a subtle mistake compared with dDPO.The prompt tests whether models can be confused by instructions containing false premises.Source: Figure 5 · p. 12
• On harmful-intent prompts, dDPO sometimes responds more politely than dSFT but sometimes complies directly.Source: Figure 6 · p. 13
• The appendix suggests that adding red-teaming examples during dDPO could improve the model’s safety capabilities.Source: Figure 6 · p. 13

A.2 SFT IS A REQUIRED STEP BEFORE DPO

• Skipping SFT before DPO significantly reduces both MT-Bench and AlpacaEval performance.The ablation compares models with and without the SFT step before DPO.Source: §A.2 · p. 13
• The pure dDPO model struggles to apply the chat template in its generations.This was observed in a qualitative evaluation of MT-Bench generations.Source: §A.2 · p. 13, Figure 7 · p. 14