Source-linked AI summary
Black-Box On-Policy Distillation of Large Language Models
Tianzhu Ye, Li Dong, Zewen Chi, Xun Wu, Shaohan Huang, Furu Wei
TL;DR
Black-box distillation lacks the teacher probabilities needed for conventional on-policy supervision. GAD addresses this by co-training a student generator and adaptive discriminator in a minimax game, and it consistently surpasses sequence-level distillation, with Qwen2.5-14B-Instruct becoming comparable to GPT-5-Chat on LMSYS-Chat.
Problem
Black-box distillation exposes only teacher-generated text, leaving conventional likelihood-based objectives without probability-level supervision for evaluating student responses.
Method
GAD trains a student generator against a discriminator that distinguishes teacher and student outputs, using discriminator feedback as an adaptive on-policy reward.
Results
GAD consistently outperforms sequence-level distillation across datasets and model sizes, with Qwen2.5-14B-Instruct becoming comparable to GPT-5-Chat on LMSYS-Chat evaluation.
Takeaways & Limitations
GAD provides an effective black-box distillation paradigm that extracts high-quality knowledge without access to teacher output logits.
Abstract
from arXiv · showhide
Black-box distillation creates student large language models (LLMs) by learning from a proprietary teacher model's text outputs alone, without access to its internal logits or parameters. In this work, we introduce Generative Adversarial Distillation (GAD), which enables on-policy and black-box distillation. GAD frames the student LLM as a generator and trains a discriminator to distinguish its responses from the teacher LLM's, creating a minimax game. The discriminator acts as an on-policy reward model that co-evolves with the student, providing stable, adaptive feedback. Experimental results show that GAD consistently surpasses the commonly used sequence-level knowledge distillation. In particular, Qwen2.5-14B-Instruct (student) trained with GAD becomes comparable to its teacher, GPT-5-Chat, on the LMSYS-Chat automatic evaluation. The results establish GAD as a promising and effective paradigm for black-box LLM distillation.
1 Introduction
Black-box distillation must learn from teacher-generated text without probability-level supervision, making on-policy learning difficult. GAD addresses this by training a student generator against an adaptive discriminator and consistently outperforming sequence-level distillation, including teacher-comparable performance for a large student.
- Challenge: Without teacher probabilities or hidden states, conventional likelihood-based objectives cannot directly evaluate the student’s own responses.This limitation is especially problematic when the teacher is a proprietary API model that exposes only generated text.
- GAD framework: GAD enables on-policy black-box distillation by training the student generator against a discriminator in a minimax game.The discriminator distinguishes teacher from student responses and provides implicit feedback without teacher probability access.
- Experimental results: Across datasets and model sizes, GAD consistently outperforms instruction-model and SeqKD baselines under identical training budgets.The experiments use GPT-5-Chat as teacher and open-source Qwen2.5 and Llama3 students.
- Experimental results: Qwen2.5-14B-Instruct trained with GAD approaches GPT-5-Chat capability, while smaller GAD students match larger SeqKD students on GPT-4o evaluation.Qwen2.5-3B-Instruct with GAD matches Qwen2.5-7B-Instruct with SeqKD, and GAD shows particularly strong out-of-distribution gains.
2 Method
GAD formulates black-box distillation as a two-player game between a student generator and a discriminator trained on teacher–student response pairs. The student maximizes discriminator reward through reinforcement learning while the discriminator continually adapts to the student’s behavior.
- 2.1 GAD: Generative Adversarial Distillation: GAD constructs a generator–discriminator minimax game from teacher responses and student-generated responses.The discriminator assigns sequence-level scores, while Bradley-Terry training encourages higher scores for teacher responses.
- 2.2 Generator training: The student treats discriminator scores as rewards and uses policy-gradient reinforcement learning to optimize its sampled responses.The sampling operation is nondifferentiable, so the discriminator score is optimized as a reward; the experiments use GRPO.
- Warmup: One epoch of warmup for both generator and discriminator precedes adversarial training to promote effective optimization.The generator is warmed up with teacher-response cross-entropy, while the discriminator uses Bradley-Terry loss on the same data.
- 2.3 Discriminator training: The discriminator is updated online with Bradley-Terry loss, adapting continually to the current student policy rather than remaining fixed.This distinguishes GAD from conventional RLHF, where the reward model is trained before policy optimization.
- Training algorithm: The training algorithm alternates student-response sampling and reward-based generator updates with discriminator Bradley-Terry updates until convergence.The algorithm begins from distillation data, a student generator, and a discriminator, then returns the trained generator.
3 Experiments
Experiments evaluate GAD against instruction models and SeqKD across automatic, human, toy-data, stability, and ablation settings. GAD consistently performs strongly, while analyses associate its advantages with broader generalization, reachable-mode learning, and stable on-policy feedback.
- Automatic Evaluation: GAD consistently outperforms instruction models and SeqKD across datasets and model variants, with particularly strong gains on out-of-distribution evaluations.Table 2 reports averaged GPT-4o scores, and the automatic evaluation compares GAD with both baselines.
- Automatic Evaluation: Qwen2.5-3B with GAD matches Qwen2.5-7B with SeqKD, Qwen2.5-7B with GAD rivals Qwen2.5-14B with SeqKD, and Qwen2.5-14B with GAD is comparable to GPT-5-Chat.
- Human Evaluation: GAD achieves a win rate above 50% and a loss rate below 30% in almost all human-evaluation comparisons against instruction and SeqKD baselines.The human evaluation covers Qwen2.5-7B, Qwen2.5-14B, and Llama-3.1-8B on the LMSYS-Chat-1M-Clean test set.
- Analysis: SeqKD has higher N-gram overlap but lower GPT-4o scores than GAD, suggesting lexical-pattern memorization rather than capture of the teacher’s global style.
- Analysis: GAD learns reachable teacher modes and remains stable for thousands of steps, whereas SeqKD targets all modes and off-policy training develops reward hacking after about 300 steps.The off-policy student produces responses up to 1300 tokens that deviate significantly from teacher patterns.
- Ablations: Removing either generator or discriminator warmup reduces effectiveness, Bradley-Terry loss outperforms cross-entropy, and equal generator-discriminator sizes perform best.The ablations attribute failures to early distributional gaps, generator-discriminator imbalance, and less effective discriminator feedback.
4 Related Work
Related work distinguishes white-box distillation, which uses teacher internals, from black-box distillation, which relies only on textual outputs and commonly uses SeqKD.
- Table 5 reports an ablation of discriminator model size on Qwen2.5 Instruct models using GPT-4o score evaluation.
- White-box distillation aligns student and teacher distributions, hidden states, or attention scores using internal teacher information.
- On-policy white-box methods learn from student-generated responses, but they still require access to teacher logits or representations.
- Black-box distillation uses teacher text outputs, making conventional white-box methods infeasible because logits and hidden representations are unavailable.
5 Conclusion
The paper presents GAD as a black-box distillation framework that enables on-policy learning through an adaptive discriminator and achieves stronger performance than sequence-level distillation.
- GAD consistently surpasses standard sequence-level distillation across multiple model families and datasets.
- GAD trains a student and adaptive discriminator in a minimax game, using discriminator feedback as an implicit on-policy reward without logit-level supervision.
- The experiments report superior generalization and performance rivaling the proprietary teacher.
A.1 Implement GAD with GRPO
GAD implements student policy optimization with GRPO and trains its discriminator from teacher–student preference pairs using Bradley-Terry loss.
- GRPO performs policy optimization for the student, whose output distribution is denoted qG.
- For each prompt, the method samples a group of N student responses and computes response advantages for policy optimization.
- The student is trained with an objective based on these grouped responses and their advantages, with KL regularization and clipping omitted in the abbreviated formulation.
- Each student response is paired with the same teacher response to form preference pairs for discriminator training.
- The discriminator minimizes Bradley-Terry loss across the group, with the teacher score shared among the paired comparisons.
A.2 Training Details
Training uses three epochs with a one-epoch warmup, fixed batch and context settings, and distinct learning-rate choices for GAD and SeqKD.
- GAD uses one warmup epoch followed by two GAD epochs, with batch size 256 and approximately 2400 optimization steps.
- The PPO mini-batch size is 256, and the discriminator receives 10 warmup steps before joint generator–discriminator training.
- SeqKD uses learning rate 5e-6, while GAD with GPT-5-Chat uses 1e-6 in both warmup and GAD stages.
- Instruction prompts use maximum context length 2048 tokens, responses use 1536, and training temperature is 0.8.
- GRPO uses group size N = 8 and KL weight β = 0.001.
- Distilling Qwen2.5-14B-Instruct from GPT-5-Chat takes about 30 hours on 16 H100 GPUs.
A.3 Automatic Evaluation Details
Evaluation uses greedy 1536-token responses, standardized prompt wrappers, and GPT-4o feedback prompts. The reported GPT-4o score normalizes the student score against the reference answer score.
- GPT-4o evaluation uses the Figure 8 prompt, while Figure 7 supplies the prompt wrapper for training and evaluation.
- The reported GPT-4o score equals the student’s score divided by the sum of the student’s and reference answer’s scores.
- Responses are generated greedily with a maximum length of 1536 tokens.
B.1 Additional Automatic Evaluation Results
Additional evaluations show that GAD preserves the student’s response-length distribution while incorporating teacher stylistic characteristics, and remains effective with a tokenizer-incompatible teacher.
- GAD maintains the original model’s response-length distribution while integrating the teacher’s global stylistic characteristics, unlike SeqKD’s shorter responses.
- GAD’s response behavior is attributed to on-policy sampling, which aligns generation patterns with both the student’s prior and the teacher’s guidance.
- With a Qwen2.5-14B-Instruct teacher and Llama-family students, GAD outperforms pre-distillation models and SeqKD in most GPT-4o evaluation settings.