Black-Box On-Policy Distillation of Large Language Models

Challenge

• Without teacher probabilities or hidden states, conventional likelihood-based objectives cannot directly evaluate the student’s own responses.This limitation is especially problematic when the teacher is a proprietary API model that exposes only generated text.Source: §1 · p. 2

GAD framework

• GAD enables on-policy black-box distillation by training the student generator against a discriminator in a minimax game.The discriminator distinguishes teacher from student responses and provides implicit feedback without teacher probability access.Source: §1 · p. 2, Figure 2 · p. 3

Experimental results

• Across datasets and model sizes, GAD consistently outperforms instruction-model and SeqKD baselines under identical training budgets.The experiments use GPT-5-Chat as teacher and open-source Qwen2.5 and Llama3 students.Source: §1 · p. 2
• Qwen2.5-14B-Instruct trained with GAD approaches GPT-5-Chat capability, while smaller GAD students match larger SeqKD students on GPT-4o evaluation.Qwen2.5-3B-Instruct with GAD matches Qwen2.5-7B-Instruct with SeqKD, and GAD shows particularly strong out-of-distribution gains.Source: §1 · p. 2

2.1 GAD: Generative Adversarial Distillation

• GAD constructs a generator–discriminator minimax game from teacher responses and student-generated responses.The discriminator assigns sequence-level scores, while Bradley-Terry training encourages higher scores for teacher responses.Source: §2.1 · p. 3, §2.2 · p. 4

2.2 Generator training

• The student treats discriminator scores as rewards and uses policy-gradient reinforcement learning to optimize its sampled responses.The sampling operation is nondifferentiable, so the discriminator score is optimized as a reward; the experiments use GRPO.Source: §2.2 · p. 4, §2.3 · p. 4

Warmup

• One epoch of warmup for both generator and discriminator precedes adversarial training to promote effective optimization.The generator is warmed up with teacher-response cross-entropy, while the discriminator uses Bradley-Terry loss on the same data.Source: §2.2 · p. 4

2.3 Discriminator training

• The discriminator is updated online with Bradley-Terry loss, adapting continually to the current student policy rather than remaining fixed.This distinguishes GAD from conventional RLHF, where the reward model is trained before policy optimization.Source: §2.3 · p. 4, §2.2 · p. 4

Training algorithm

• The training algorithm alternates student-response sampling and reward-based generator updates with discriminator Bradley-Terry updates until convergence.The algorithm begins from distillation data, a student generator, and a discriminator, then returns the trained generator.Source: §2.3 · p. 5

Automatic Evaluation

• GAD consistently outperforms instruction models and SeqKD across datasets and model variants, with particularly strong gains on out-of-distribution evaluations.Table 2 reports averaged GPT-4o scores, and the automatic evaluation compares GAD with both baselines.Source: §3.2 · p. 5, Table 2 · p. 6
• Qwen2.5-3B with GAD matches Qwen2.5-7B with SeqKD, Qwen2.5-7B with GAD rivals Qwen2.5-14B with SeqKD, and Qwen2.5-14B with GAD is comparable to GPT-5-Chat.Source: §3.2 · p. 6

Human Evaluation

• GAD achieves a win rate above 50% and a loss rate below 30% in almost all human-evaluation comparisons against instruction and SeqKD baselines.The human evaluation covers Qwen2.5-7B, Qwen2.5-14B, and Llama-3.1-8B on the LMSYS-Chat-1M-Clean test set.Source: §3.2 · p. 6, Figure 3 · p. 6

Analysis

• SeqKD has higher N-gram overlap but lower GPT-4o scores than GAD, suggesting lexical-pattern memorization rather than capture of the teacher’s global style.Source: §3.3 · p. 7, Figure 4 · p. 7
• GAD learns reachable teacher modes and remains stable for thousands of steps, whereas SeqKD targets all modes and off-policy training develops reward hacking after about 300 steps.The off-policy student produces responses up to 1300 tokens that deviate significantly from teacher patterns.Source: §3.3 · p. 7, Figure 6 · p. 7

Ablations

• Removing either generator or discriminator warmup reduces effectiveness, Bradley-Terry loss outperforms cross-entropy, and equal generator-discriminator sizes perform best.The ablations attribute failures to early distributional gaps, generator-discriminator imbalance, and less effective discriminator feedback.Source: §3.4 · p. 8, Table 4 · p. 8

• Table 5 reports an ablation of discriminator model size on Qwen2.5 Instruct models using GPT-4o score evaluation.Source: Table 5 · p. 8
• White-box distillation aligns student and teacher distributions, hidden states, or attention scores using internal teacher information.Source: §4 · pp. 8–9
• On-policy white-box methods learn from student-generated responses, but they still require access to teacher logits or representations.Source: §4 · p. 9
• Black-box distillation uses teacher text outputs, making conventional white-box methods infeasible because logits and hidden representations are unavailable.Source: §4 · p. 9

• GAD consistently surpasses standard sequence-level distillation across multiple model families and datasets.Source: §5 · p. 9
• GAD trains a student and adaptive discriminator in a minimax game, using discriminator feedback as an implicit on-policy reward without logit-level supervision.Source: §5 · p. 9
• The experiments report superior generalization and performance rivaling the proprietary teacher.Source: §5 · p. 9

• GRPO performs policy optimization for the student, whose output distribution is denoted qG.Source: §A.1 · p. 13
• For each prompt, the method samples a group of N student responses and computes response advantages for policy optimization.Source: §A.1 · p. 13
• The student is trained with an objective based on these grouped responses and their advantages, with KL regularization and clipping omitted in the abbreviated formulation.Source: §A.1 · p. 13
• Each student response is paired with the same teacher response to form preference pairs for discriminator training.Source: §A.1 · p. 13
• The discriminator minimizes Bradley-Terry loss across the group, with the teacher score shared among the paired comparisons.Source: §A.1 · p. 13

• GAD uses one warmup epoch followed by two GAD epochs, with batch size 256 and approximately 2400 optimization steps.Source: §A.2 · p. 13
• The PPO mini-batch size is 256, and the discriminator receives 10 warmup steps before joint generator–discriminator training.Source: §A.2 · p. 13
• SeqKD uses learning rate 5e-6, while GAD with GPT-5-Chat uses 1e-6 in both warmup and GAD stages.Source: §A.2 · p. 13
• Instruction prompts use maximum context length 2048 tokens, responses use 1536, and training temperature is 0.8.Source: §A.2 · p. 13
• GRPO uses group size N = 8 and KL weight β = 0.001.Source: §A.2 · p. 13
• Distilling Qwen2.5-14B-Instruct from GPT-5-Chat takes about 30 hours on 16 H100 GPUs.Source: §A.2 · p. 13

• GPT-4o evaluation uses the Figure 8 prompt, while Figure 7 supplies the prompt wrapper for training and evaluation.Source: §A.3 · p. 14, Figure 7 · p. 14, Figure 8 · p. 14
• The reported GPT-4o score equals the student’s score divided by the sum of the student’s and reference answer’s scores.Source: §A.3 · p. 14
• Responses are generated greedily with a maximum length of 1536 tokens.Source: §A.3 · p. 14

• GAD maintains the original model’s response-length distribution while integrating the teacher’s global stylistic characteristics, unlike SeqKD’s shorter responses.Source: §B.1 · p. 14, Table 6 · p. 15
• GAD’s response behavior is attributed to on-policy sampling, which aligns generation patterns with both the student’s prior and the teacher’s guidance.Source: §B.1 · p. 14
• With a Qwen2.5-14B-Instruct teacher and Llama-family students, GAD outperforms pre-distillation models and SeqKD in most GPT-4o evaluation settings.Source: §B.1 · p. 14, Table 7 · p. 15