f-Divergence Minimization for Sequence-Level Knowledge Distillation

Motivation

• Knowledge distillation transfers knowledge from a large teacher to a smaller student, supporting compact models because superlarge language models are inaccessible to many users and researchers.The paper focuses on distribution matching, in which the student learns the teacher’s predictive distribution, rather than representation matching.Source: §1 · p. 1

f-DISTILL framework

• The framework formulates sequence-level distillation with four f-divergence variants—KL, reverse KL, JS, and TVD—covering asymmetric and symmetric teacher–student objectives.Existing SeqKD and ENGINE are approximations of the KL and reverse-KL variants, respectively.Source: §1 · pp. 1–2

Distillation behavior

• Symmetric JS and TVD losses outperform asymmetric losses by avoiding the extreme mode averaging and mode collapsing associated with KL and reverse KL distillation.KL can produce overly smooth student distributions, whereas reverse KL can concentrate probability on only a few teacher modes.Source: §1 · pp. 1–2

Efficient computation

• Step-wise decomposition and offline teacher sampling make sequence-level f-divergences tractable while reducing the additional training cost of symmetric objectives.The decomposition converts sequence-level divergence into word-level losses, and offline sampling uses teacher-generated data without affecting deployment-time inference.Source: §1 · p. 2

Experiments

• f-DISTILL consistently outperforms existing distribution-matching KD methods across DART, XSum, WMT16 EN-RO, and Commonsense Dialogue.It also provides an add-on improvement when combined with representation-matching KD methods.Source: §1 · p. 2

2.1 Classic KD and Its Drawbacks

• KL distillation can produce mode averaging, while reverse KL and related ENGINE objectives can produce mode collapsing.KL spreads probability mass broadly because the student distribution appears in the denominator, whereas reverse KL concentrates on selected teacher modes.Source: §2.1 · p. 2, §2.2 · p. 3

2.2 Our Proposed f-DISTILL Framework

• The f-DISTILL framework generalizes sequence-level distillation through f-divergence functions, covering four variants based on different divergence choices.The framework is designed to move beyond standard KL-based distribution matching.Source: §2.2 · pp. 3–4

2.1 Classic KD and Its Drawbacks

• Unlike SeqKD’s decoded hard sequence, KL distillation retains the teacher’s full conditional distribution as soft labels.The paper argues that these soft labels provide more information for sequence-level distillation.Source: §2.1 · p. 2, §2.2 · p. 3

2.2 Our Proposed f-DISTILL Framework

• The proposed JS and TVD variants use symmetric divergences to balance the mode-averaging and mode-collapsing extremes.JS compares teacher and student distributions with their average, while TVD measures their ℓ1 distance without a logarithm.Source: §2.2 · pp. 3–4
• Sequence-level KL, reverse KL, and JS divergences decompose exactly into step-wise terms, while TVD is upper bounded by step-wise terms.These decompositions replace intractable sentence-space objectives with word-level training losses.Source: §2.2 · p. 4
• Symmetric JS and TVD losses require teacher and student sampling during training, so the method uses fixed offline teacher samples to improve efficiency.The teacher remains unchanged, allowing its samples to be generated beforehand while the student continues to be sampled online.Source: §2.3 · p. 4

Baselines and complementarity

• Distillation substantially improves over direct ground-truth MLE training, while pre-distillation alone remains weaker than other distillation methods and f-DISTILL provides an add-on improvement.All compared distillation methods use the same pre-distilled student initialization for fairness.Source: §3.2 · p. 5

Comparison with existing KD

• Soft-label f-DISTILL methods consistently outperform hard-sequence SeqKD and ENGINE, indicating that teacher probabilities provide more informative sequence-level supervision.SeqKD and ENGINE are hard approximations of KL and reverse-KL distillation, respectively.Source: §3.2 · pp. 5–6

Main results

• f-DISTILL variants consistently outperform existing distribution-matching KD methods across the four evaluated datasets, with symmetric losses generally strongest except for WMT16 EN-RO TER.JS and TVD rank first or second on most metrics, while KL slightly surpasses them on WMT16 EN-RO TER.Source: §3.2 · p. 6

Likelihood and coverage

• Teacher diversity affects distillation behavior: Commonsense Dialogue is most multi-modal, whereas machine translation is least multi-modal according to distinct bi-gram percentage.The analysis measures teacher diversity by sampling five outputs per test input and averaging distinct bi-gram percentages.Source: §3.2 · p. 7
• Symmetric JS and TVD losses balance likelihood and coverage risks between KL’s mode averaging and reverse KL’s mode collapse, supporting high performance across tasks.KL achieves lower coverage risk, whereas reverse KL can achieve better likelihood on the highly multi-modal Commonsense Dialogue task.Source: §3.2 · p. 7

Efficiency and human evaluation

• Offline teacher sampling more than doubles training speed while retaining comparable performance, and human evaluation finds better missing-information and hallucination scores without a significant fluency difference versus SeqKD.The SeqKD–TVD p-values are 32.6% for fluency, 1.28% for missing information, and 0.669% for hallucination.Source: §3.2 · pp. 7–8

• Distribution-matching KD trains students to imitate teacher output probabilities, while representation matching targets intermediate-layer representations.Source: §4 · p. 8
• Sequence-level KD methods use sampled hard sequences or teacher-defined energy functions, approximating KL and reverse-KL distillation, respectively.Source: §4 · p. 8
• f-DISTILL requires meaningful student sampling and provides add-on improvements alongside existing word-level and intermediate-layer KD techniques.Source: §4 · p. 8
• Unlike pruning and sparse modeling, this work studies knowledge distillation as the compression setting.Source: §4 · p. 8
• The paper claims to be the first to systematically formulate sequence-level knowledge distillation as f-divergence minimization.Source: §4 · p. 9

• f-DISTILL extends KL distillation with four variants: KL, RKL, JS, and TVD, while relating SeqKD and ENGINE to KL and RKL.Source: §5 · p. 9
• Across four text-generation tasks, f-DISTILL consistently outperforms existing KD methods.Source: §5 · p. 9
• Symmetric JS and TVD losses outperform asymmetric losses by avoiding extreme mode averaging and mode collapsing.Source: §5 · p. 9

• f-DISTILL is less efficient to train than SeqKD and ENGINE because it requires teacher soft probabilities rather than hard sampled sequences.Source: §6 · p. 9
• The study does not report multi-run statistics, citing approximately 2000 GPU hours and prioritizing broad metrics and framework analysis instead.Source: §6 · p. 9
• The published ACL’23 version could not anticipate discussions that were added in the subsequent arXiv manuscript.Source: §6 · p. 9

• JS step-wise decomposition requires the conditional distribution of the middle distribution, and the corrected computation is less efficient because it stores per-step probabilities.Source: §7 · p. 9
• The authors implemented an incorrect JS formula but report that it worked empirically as an approximation.Source: §7 · p. 9
• RKL, JS, and TVD variants sample from the student, but direct backpropagation cannot provide sampling-parameter gradients without reinforcement learning.Source: §7 · p. 9
• Greedy step-wise optimization still pushes the student toward the teacher and reaches zero loss when the two distributions match.Source: §7 · p. 10

• Theorem 1 exactly decomposes sequence-level KL, RKL, and JS divergences into step-wise terms, while upper-bounding TVD with step-wise terms.Source: A Proof of Theorem 1 · p. 14
• The proof defines the teacher and student sequence distributions, introduces their average for JS, and recursively separates the final timestep.Source: A Proof of Theorem 1 · p. 14
• Step-wise word enumeration replaces intractable sequence summation, allowing gradients to reach every word at every timestep.Source: A Proof of Theorem 1 · p. 14
• Reusing partially sampled teacher and student prefixes makes the computation linear rather than quadratic in sequence length.Source: A Proof of Theorem 1 · p. 14
• The TVD proof applies recursive rewriting, the triangle inequality, and two complementary upper bounds before combining them into a more robust bound.Source: A Proof of Theorem 1 · pp. 15–16

• The benchmark spans diverse natural-language-generation tasks and data sizes, using 200M–400M-parameter teachers and 50M–150M-parameter students.Source: B Experimental Details · p. 16
• Training uses Adam for DART, XSum, and Commonsense Dialogue, while WMT16 EN-RO follows the T5 setup with AdaFactor.Source: B Experimental Details · p. 16
• Students receive 28 pre-distillation epochs followed by 12 epochs for each distillation method, using batch size eight to fit teacher and student models.Source: B Experimental Details · p. 16
• The experiments use AMD Milan CPUs and NVIDIA A100 GPUs, totaling an estimated 2000 GPU hours across datasets and model variants.Source: B Experimental Details · p. 17

• On DART, KL and RKL fail to produce coherent responses, whereas JS and TVD correctly identify the train name, origin, and destination.Source: C Case Study · p. 17
• Commonsense Dialogue presents a severe multimodal setting that requires balancing mode averaging and mode collapsing.Source: C Case Study · p. 17
• The KL-distilled student produces a short, generic response associated with mode averaging, while the RKL-distilled student produces a detailed response.Source: C Case Study · p. 17