Source-linked AI summary
Zephyr: Direct Distillation of LM Alignment
Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Clémentine Fourrier, Nathan Habib, Nathan Sarrazin, Omar Sanseviero, Alexander M. Rush, Thomas Wolf
TL;DR
Open distilled language models can gain task accuracy yet remain poorly aligned with natural user intent, motivating alignment without costly human annotation. The paper combines distilled supervised fine-tuning with AI-feedback preferences and dDPO, yielding Zephyr-7B, which reaches leading 7B chat performance and outperforms Llama2-Chat-70B on MT-Bench. The evaluation and method remain limited by GPT-4-based benchmark assessment and the study’s focus on helpfulness rather than safety.
Problem
Distilled models improve accuracy but can remain intent-un-aligned, while measuring and collecting feedback for alignment is difficult and costly.
Method
The method combines dSFT on teacher-generated data with AI-generated preference data and distilled direct preference optimization, avoiding human annotation and sampling-based training.
Results
Zephyr-7B sets a new state-of-the-art among 7B chat models, achieves competitive performance with Llama2-Chat-70B, and scores better on MT-Bench.
Takeaways & Limitations
The results demonstrate that a small open model can be aligned to user intent through distillation and preference optimization, motivating further exploration of smaller open models.
Takeaways & Limitations
The study does not address model safety, and its AlpacaEval and MT-Bench results rely on GPT-4 evaluation that may favor distilled or verbose responses.
Abstract
from arXiv · showhide
We aim to produce a smaller language model that is aligned to user intent. Previous research has shown that applying distilled supervised fine-tuning (dSFT) on larger models significantly improves task accuracy; however, these models are unaligned, i.e. they do not respond well to natural prompts. To distill this property, we experiment with the use of preference data from AI Feedback (AIF). Starting from a dataset of outputs ranked by a teacher model, we apply distilled direct preference optimization (dDPO) to learn a chat model with significantly improved intent alignment. The approach requires only a few hours of training without any additional sampling during fine-tuning. The final result, Zephyr-7B, sets the state-of-the-art on chat benchmarks for 7B parameter models, and requires no human annotation. In particular, results on MT-Bench show that Zephyr-7B surpasses Llama2-Chat-70B, the best open-access RLHF-based model. Code, models, data, and tutorials for the system are available at https://github.com/huggingface/alignment-handbook.
1 INTRODUCTION
The paper addresses the gap between improving open models’ task accuracy through distillation and aligning their behavior with user intent. It proposes dDPO using AI-generated preferences, producing Zephyr-7B without human annotation while focusing on helpfulness rather than safety.
- Distilled models improve task performance but often remain intent-un-aligned and produce incorrect responses to user queries.
- Intent-alignment benchmarks such as MT-Bench and AlpacaEval correlate closely with human ratings and show proprietary models outperform open distilled models.
- dDPO uses AI Feedback from teacher models as preference data to align a small open LLM without human annotation or additional sampling during fine-tuning.
- Zephyr-7B combines UltraChat dSFT, UltraFeedback preferences, and dDPO, achieving performance comparable to 70B-parameter human-feedback chat models.
- The study evaluates helpfulness alignment but does not address safety issues such as harmful outputs or illegal advice.
2 RELATED WORK
Related work established open models, distillation methods, and increasingly capable LLM-based evaluation tools. Zephyr extends distillation beyond supervised fine-tuning to preference optimization and uses Mistral 7B as its starting model.
- Open LLMs have become research artifacts for efficient fine-tuning, retrieval, quantization, and chatbot development.
- Figure 2 organizes Zephyr’s method into dataset construction and dSFT, AI-feedback preference collection, and dDPO.
- Zephyr uses Mistral 7B as its starting point because of its strong performance.
- Earlier distilled models primarily focused on distilling supervised fine-tuning, whereas Zephyr targets both supervised fine-tuning and preference optimization.
- LLM evaluators and benchmarks compare responses through scoring, pairwise rankings, crowdsourced battles, or Elo ratings.
3 METHOD
The method aligns a smaller pretrained model through three stages: distilled supervised fine-tuning, AI-generated preference collection, and direct preference optimization. dDPO trains from static preference data rather than sampling-based reinforcement learning.
- The goal is to align a student model πθ to user intent using access to a larger teacher model πT.
- Distilled Supervised Fine-Tuning (dSFT): dSFT trains a raw LLM on teacher-generated instructions and responses so it can respond to user prompts.
- Distilled Supervised Fine-Tuning (dSFT): The dSFT dataset is constructed through iterative teacher prompting that generates responses, refines instructions, and produces instruction-response pairs.
- AI Feedback through Preferences (AIF): AI Feedback replaces human preferences with teacher judgments of responses generated by an ensemble of models.
- AI Feedback through Preferences (AIF): UltraFeedback collects multiple model responses for each prompt and has a teacher such as GPT-4 score them before forming preferences.
- Distilled Direct Preference Optimization (dDPO): dDPO refines the dSFT model by maximizing the likelihood that preferred responses yw rank above rejected responses yl.
- Distilled Direct Preference Optimization (dDPO): DPO directly optimizes the preference model from static data using a reward expressed through the optimal and reference language-model policies.
- Distilled Direct Preference Optimization (dDPO): Training iterates over each AI-feedback triple, computes response probabilities under the dSFT and dDPO models, and backpropagates the objective.
4 EXPERIMENTAL DETAILS
The experiments use Mistral 7B and distilled dialogue datasets, evaluating instruction-following and academic capabilities with chat and leaderboard benchmarks. SFT and DPO are trained for controlled epoch counts and batch configurations.
- Models and training: Mistral 7B is used as the base language model for all fine-tuning experiments.It is described as a state-of-the-art base model at the 7B parameter scale.
- Datasets: UltraChat contains 1.47M GPT-3.5-TURBO-generated multi-turn dialogues, filtered to approximately 200k helpful examples for dSFT.Truecasing heuristics corrected about 5% of the corpus before filtering.
- Datasets: UltraFeedback provides 64k prompts with four responses rated by GPT-4, from which chosen and rejected responses form binary preferences.The highest mean score is chosen, while a remaining response is randomly selected as rejected.
- Evaluation: MT-Bench evaluates 160 questions across eight knowledge areas using GPT-4 ratings averaged over two conversational turns.Each response receives a score from 1–10.
- Evaluation: AlpacaEval measures GPT-4-rated pairwise win-rate against text-davinci-003 on 805 single-turn questions.The benchmark focuses mostly on helpfulness.
- Evaluation: The Open LLM Leaderboard measures ARC, HellaSwag, MMLU, and TruthfulQA to assess reasoning and truthfulness regressions after fine-tuning.These are multiclass classification tasks rather than conversational evaluations.
- Training configuration: SFT runs for one to three epochs with a 2e-5 peak learning rate, while DPO runs for one to three epochs with a 5e-7 peak learning rate and β = 0.1.The final model uses one SFT epoch followed by three DPO epochs.
5 RESULTS AND ABLATIONS
The results show that dDPO substantially improves chat and academic performance when preceded by dSFT, while training duration and model scale affect the final outcome. Zephyr-7B performs strongly among 7B models but remains weaker than some larger models and in math and coding.
- Chat results: Zephyr-7B achieves state-of-the-art chat performance among open 7B models on MT-Bench and AlpacaEval.It performs significantly better than dSFT models across both benchmarks.
- Chat results: Zephyr-7B scores better than Llama2-Chat 70B on MT-Bench and falls within two standard deviations of it on AlpacaEval.It performs worse than WizardLM-70B and XWIN-LM-70B.
- Chat results: Zephyr-7B performs competitively with proprietary models on several MT-Bench categories but is much worse in math and coding.AlpacaEval comparisons with GPT-3.5-TURBO and Claude 2 require caution because its prompts may not represent real usage or advanced applications.
- Academic results: Zephyr-7B performs best among 7B models on academic tasks and reaches the performance of 40B-scale models.Larger models perform better on some knowledge-intensive tasks.
- Ablations: Skipping initial dSFT causes models to learn poorly from feedback, while the full dDPO-plus-dSFT procedure produces large gains on both chat benchmarks.Applying dSFT directly to preferred UltraFeedback outputs does not improve performance.
- Training dynamics: Perfect training accuracy after one DPO epoch indicates overfitting, yet downstream MT-Bench and AlpacaEval performance remains strongest after one SFT epoch and three DPO epochs.When SFT exceeds one epoch, longer DPO training induces a performance regression.
6 CONCLUSIONS AND LIMITATIONS
The paper presents alignment distillation with dDPO as a way to align smaller pretrained models using AI-feedback preferences, while avoiding sampling-based approaches. Zephyr-7B achieves strong chat performance, but the study identifies evaluator bias, scaling uncertainty, and safety as important boundaries.
- 6 CONCLUSIONS AND LIMITATIONS: Zephyr-7B uses direct preference optimization on AI-feedback data to distill conversational capabilities into a smaller pretrained model.The method avoids rejection sampling and PPO and is based on Mistral-7B.
- 6 CONCLUSIONS AND LIMITATIONS: Zephyr-7B sets a new state-of-the-art for 7B-parameter chat models and outperforms Llama2-Chat-70B on MT-Bench.
- 6 CONCLUSIONS AND LIMITATIONS: The study’s evaluation relies on GPT-4 for AlpacaEval and MT-Bench, whose ratings may favor distilled or verbose responses.
- 6 CONCLUSIONS AND LIMITATIONS: The method’s scalability to much larger models such as Llama2-70B remains unexamined.
A.1 QUALITATIVE EXAMPLES
The qualitative examples examine how dSFT and dDPO respond to mathematical instructions, false premises, and harmful prompts. They illustrate both improved instruction-following behavior and remaining safety weaknesses.
- A.1 QUALITATIVE EXAMPLES: The examples use selected MT-Bench and adversarial prompts to compare instruction following across ordinary and harmful-intent settings.Adversarial completions use nucleus sampling with top-p = 0.95 and T = 0.7.
- A.1 QUALITATIVE EXAMPLES: $16,000 is the dDPO completion for the startup’s two-year software-development investment problem.
- A.1 QUALITATIVE EXAMPLES: Figure 4 uses a cherry-picked MT-Bench prompt to demonstrate dDPO’s ability to follow mathematical instructions.
- A.1 QUALITATIVE EXAMPLES: Figure 5 contrasts dSFT and dDPO responses to instructions containing false premises, including a reference to an “adult-sized helicopter”.
- A.1 QUALITATIVE EXAMPLES: For harmful-intent prompts, dDPO is sometimes more polite than dSFT but sometimes complies directly with the request.The authors suggest that adding red-teaming examples could improve safety capabilities.
A.2 SFT IS A REQUIRED STEP BEFORE DPO
The ablation shows that supervised fine-tuning is necessary before applying DPO in this pipeline. Skipping SFT reduces benchmark performance, while pure DPO also struggles with the chat template.
- A.2 SFT IS A REQUIRED STEP BEFORE DPO: Skipping SFT before DPO significantly reduces both MT-Bench and AlpacaEval performance.
- A.2 SFT IS A REQUIRED STEP BEFORE DPO: Pure DPO struggles to learn the chat template in qualitative MT-Bench generations.
- A.2 SFT IS A REQUIRED STEP BEFORE DPO: After overtaking the second person, the model should place itself second and the overtaken person third.