PaperlayerView exact revision on arXiv

Source-linked AI summary

Understanding Knowledge Distillation in Non-autoregressive Machine Translation

Chunting Zhou, Graham Neubig, Jiatao Gu

arXiv:1911.02727v3cs.CL

TL;DR

NAT benefits from parallel decoding but depends heavily on knowledge distillation, whose contribution to training was not clearly understood. The paper uses empirical analyses and complexity measures to explain this effect, relate distilled-data complexity to NAT capacity, and adjust the data accordingly. It reports state-of-the-art NAT performance and largely matches the autoregressive model.

  • Problem

    The paper investigates why knowledge distillation improves NAT training and how teacher and student capacity determine suitable distilled data.

  • Method

    The authors conduct empirical analyses across autoregressive teachers and NAT students, using complexity and faithfulness measures for parallel data.

  • Results

    The experiments find that higher-capacity NAT models require more complex distilled data for better performance.

  • Takeaways & Limitations

    Adjusting distilled-data complexity to match NAT capacity improves performance and yields state-of-the-art NAT results that largely match the autoregressive model.

  • Takeaways & Limitations

    The quantitative measures rely on assumptions that target tokens are independent given the source and that token distributions follow an alignment model.

Abstract

from arXiv · show

Non-autoregressive machine translation (NAT) systems predict a sequence of output tokens in parallel, achieving substantial improvements in generation speed compared to autoregressive models. Existing NAT models usually rely on the technique of knowledge distillation, which creates the training data from a pretrained autoregressive model for better performance. Knowledge distillation is empirically useful, leading to large gains in accuracy for NAT models, but the reason for this success has, as of yet, been unclear. In this paper, we first design systematic experiments to investigate why knowledge distillation is crucial to NAT training. We find that knowledge distillation can reduce the complexity of data sets and help NAT to model the variations in the output data. Furthermore, a strong correlation is observed between the capacity of an NAT model and the optimal complexity of the distilled data for the best translation quality. Based on these findings, we further propose several approaches that can alter the complexity of data sets to improve the performance of NAT models. We achieve the state-of-the-art performance for the NAT-based models, and close the gap with the autoregressive baseline on WMT14 En-De benchmark.

1 INTRODUCTION

NAT improves decoding efficiency by predicting tokens in parallel, but its weaker modeling of output dependencies makes knowledge distillation central to training. This paper investigates how distillation changes data complexity, how that interacts with NAT capacity, and how to narrow the performance gap with autoregressive models.

  • NAT predicts whole sequences or multi-token chunks simultaneously, trading model capacity for decoding efficiency.Autoregressive systems instead generate tokens step by step, limiting inference parallelism.
  • Sequence-level knowledge distillation replaces training targets with outputs from a pretrained autoregressive teacher model.The teacher is trained on the same corpus with roughly equal parameter count.
  • The paper examines how distillation reduces output modes, how teacher and student capacity interact, and how to further close the gap with autoregressive models.These questions motivate the paper’s systematic empirical analysis.
  • The authors use synthetic visualizations and conditional entropy and KL-divergence metrics to study dataset complexity and faithfulness.The metrics are based on word translation and external alignment information.
  • Systematic experiments cover four autoregressive teachers and six NAT students with varied architectures on WMT14 English-German.The study links NAT capacity with the dataset complexity that yields the best translation quality.
  • The paper proposes adjusting distilled-data complexity to match NAT capacity and reports state-of-the-art NAT performance that largely matches the autoregressive model.

2 BACKGROUND

Autoregressive NMT generates tokens sequentially, whereas NAT predicts outputs in parallel under stronger independence assumptions. Sequence-level knowledge distillation trains sequence models on hard targets selected from a pretrained autoregressive teacher.

  • Autoregressive NMT factorizes output probability by conditioning each target token on previously generated tokens.Generation proceeds left to right, using greedy decoding or beam search.
  • NAT imposes a zeroth-order Markov restriction and can independently factor the conditional distribution across output positions.Standard NAT architectures predict the entire sequence in one decoder forward pass.
  • Because multiple translations can correspond to one input, vanilla NAT may fail to capture output-token dependencies and repeat tokens.
  • Knowledge distillation trains a weaker student using outputs or probabilities produced by a stronger teacher model.The original formulation uses teacher label probabilities as soft targets, with temperature controlling the distribution.
  • Sequence-level distillation approximates the teacher’s sequence distribution with its most likely hard target, using beam search in practice.NAT studies commonly use a pretrained autoregressive Transformer teacher with roughly equal trainable parameters.

3 HOW DOES DISTILLATION IMPROVE NAT?

The paper uses synthetic multilingual data and quantitative corpus measures to show how distillation reduces multi-modality, making output distributions easier for NAT models to represent. It also examines how complexity and faithfulness characterize useful training data.

  • 3.1 SYNTHETIC EXPERIMENT FOR MULTI-MODALITY: Synthetic experiments compare autoregressive and non-autoregressive outputs on a corpus where each English input has German, French, and Spanish targets.The setup explicitly creates three output-language modes for every source sentence.
  • 3.1 SYNTHETIC EXPERIMENT FOR MULTI-MODALITY: AT outputs cluster near simplex vertices, indicating that decoding selects one language mode for each translated sequence.The language posterior is estimated from token posteriors and represented as a point in the three-language simplex.
  • 3.1 SYNTHETIC EXPERIMENT FOR MULTI-MODALITY: NAT outputs scatter across the simplex because the model mixes languages instead of consistently selecting a single mode.This illustrates NAT’s difficulty capturing dependencies when multiple target modes exist.
  • 3.1 SYNTHETIC EXPERIMENT FOR MULTI-MODALITY: Training NAT on randomly selected targets or distilled AT outputs makes its translations more likely to follow a single language mode.Both datasets contain fewer modes than the original multilingual corpus, while distillation may select modes systematically by sentence type.
  • 3.2 QUANTITATIVE MEASURES FOR PARALLEL DATA: The paper measures corpus complexity with conditional entropy and faithfulness with KL-divergence based on an external alignment tool.The complexity calculation uses conditional-independence and alignment-model assumptions to make entropy tractable.
  • 3.2 QUANTITATIVE MEASURES FOR PARALLEL DATA: Distilled data has much lower conditional entropy than concatenated or randomly selected data, while En-De is more complex than En-Es and En-Fr.The reported complexity pattern helps explain why NAT more often predicts Spanish or French modes than German in the synthetic experiment.

4 EMPIRICAL STUDY

The empirical study evaluates how distilled data from autoregressive teachers of varying capacity affects diverse NAT models on WMT14 English-German. It finds that teacher capacity changes data complexity, faithfulness, and monotonicity, while NAT students perform best with distilled data whose complexity matches their capacity.

  • 4.2 ANALYSIS OF THE DISTILLED DATA: Higher-capacity AT teachers produce distilled data with higher complexity and greater faithfulness to the original word-level translation distribution.Complexity C(d) increases while faithfulness F(d) decreases as teacher capacity grows.
  • 4.2 ANALYSIS OF THE DISTILLED DATA: Higher-capacity teachers generate distilled data with higher BLEU relative to the real data set.The result agrees with the observed KL-divergence pattern.
  • 4.2 ANALYSIS OF THE DISTILLED DATA: Distilled data has significantly less reordering than real parallel data, and weaker teachers produce more monotonic alignments.A larger fuzzy reordering score indicates more monotonic alignment.
  • 4.3 DECODING METHODS: Beam search and greedy decoding reduce data complexity most while maintaining high faithfulness, whereas sampling reduces output modes less aggressively.The comparison uses four decoding methods with the same base Transformer teacher and vanilla NAT student.
  • 4.4 DISTILLED DATA V.S. NAT MODELS: Weaker NAT students prefer lower-complexity distilled data, while stronger students perform best with higher-complexity data.Vanilla NAT performs best with data from a small Transformer, whereas LevT performs best with data from a big Transformer.
  • 4.4 DISTILLED DATA V.S. NAT MODELS: Changing the distilled data improves particular NAT models, and distilling from a big AT model enables LevT to close the gap with a similarly sized Transformer base model.FlowSeq increased to 22 BLEU after switching from Transformer(base) to Transformer(small) distilled data.

5 IMPROVEMENTS TO KNOWLEDGE DISTILLATION

The paper proposes techniques for adjusting distilled-data complexity to match NAT capacity. These methods improve lower- or higher-capacity students through additional self-distillation, mixture-of-experts translation, or sequence-level interpolation.

  • 5 IMPROVEMENTS TO KNOWLEDGE DISTILLATION: The optimal dataset complexity correlates with NAT capacity, motivating simplification for weaker students and increased faithfulness for stronger students.The proposed techniques are BANs, MoE, and sequence-level interpolation.
  • Born-Again Networks: Born-Again Networks repeatedly train AT models on decoded outputs from the previous iteration to create progressively simplified datasets.The process starts from real data and yields one distilled dataset per iteration.
  • Born-Again Networks: 2 BLEU improvement is obtained for vanilla NAT using distilled data from reborn iteration 6.The base AT model’s performance remains almost unchanged while distilled-data complexity decreases and eventually becomes constant.
  • Mixture-of-Experts: Using distilled data from a 3-expert MoE model improves NAT performance by 1.21 BLEU over distilled data from a plain base AT model.The best expert’s performance tends to decrease as the number of experts increases, while complexity and faithfulness vary substantially.
  • Sequence-Level Interpolation: Sequence-level interpolation selects the highest-BLEU hypothesis from K-best beam outputs to create data closer to the real distribution.It is intended for stronger NAT models such as MaskT and LevT.
  • Sequence-Level Interpolation: Approximately 0.4 BLEU improvement is obtained for LevT with base-model interpolation, without much increase in dataset complexity.The comparison is against standard distillation from the base AT model.

6 CONCLUSION

The conclusion presents a systematic empirical account of knowledge distillation for NAT and links student capacity to the complexity of data that yields better performance. It also proposes methods to adjust dataset complexity accordingly.

  • 6 CONCLUSION: The study systematically examines why knowledge distillation improves NAT performance across AT teachers of different capacities and a wide range of NAT models.It also defines metrics for quantitatively measuring parallel-data complexity.
  • 6 CONCLUSION: Higher-capacity NAT models require more complex distilled data for better performance, motivating techniques that match data complexity to student capacity.The conclusion frames complexity adjustment as a route to improving NAT models.

A.1 AT MODELS

The experiments use Transformer-based autoregressive teachers and several non-autoregressive student architectures under standardized training and decoding settings.

  • A.1 AT MODELS: All autoregressive teachers are implemented with the Transformer model using fairseq.The listed teacher architecture hyperparameters are provided in Table 5.
  • A.1 AT MODELS: AT models train for 300,000 updates on eight GPUs with an effective batch size of 32,000 tokens, while NAT training generally uses larger batches.NAT training uses eight GPUs and 64,000 effective tokens for most models, with FlowSeq trained on 32 GPUs.
  • A.1 AT MODELS: Teacher-generated distilled data uses beam search with beam size 5, length normalization, and no length penalty.
  • A.1 AT MODELS: The study evaluates multiple NAT architectures, including vanilla NAT, FlowSeq, iNAT, InsT, LevT, and MaskT.Vanilla NAT copies encoder embeddings monotonically, while most models additionally predict output length or placeholder counts.
  • A.1 AT MODELS: NAT models use greedy decoding; iterative models decode for at most 10 steps, whereas vanilla NAT and FlowSeq select token arg maxes directly.
  • A.1 AT MODELS: Final results for all models are computed using tokenized BLEU.

B REAL DATA STATISTICS

On WMT14 En-De, sentence-level conditional-entropy distributions differ across real and distilled datasets, with distilled data showing progressively lower complexity as teacher size decreases.

  • B REAL DATA STATISTICS: The mode of sentence-level conditional entropy is highest for real data and then decreases from big-AT through base-AT, small-AT, and tiny-AT distilled data.This ordering agrees with the proposed dataset complexity measure C(d).
  • B REAL DATA STATISTICS: Figure 7 compares histograms of H(y|x) for real parallel data and datasets distilled from big, base, small, and tiny AT teachers.

C ADDITIONAL METRICS

Additional evaluation metrics reinforce the BLEU-based findings: METEOR, RIBES, ChrF, TER, and BEER follow similar trends across the tested systems.

  • C ADDITIONAL METRICS: METEOR, RIBES, ChrF, TER, and BEER are evaluated alongside BLEU because BLEU may not fully capture system changes.The metrics use upward arrows except TER, which uses a downward arrow.
  • C ADDITIONAL METRICS: All additional metrics correlate well with BLEU and show a similar trend in the experiments.

D SYNTHETIC DATA WITH ACCESS TO THE TRUE DISTRIBUTION

The Bayesian decision-theory background formalizes structured prediction through conditional and overall risk, then distinguishes sequence-level from token-level losses and their corresponding Bayes decisions.

  • D SYNTHETIC DATA WITH ACCESS TO THE TRUE DISTRIBUTION: For structured prediction, conditional risk is the expected loss of predicting an output sequence given the posterior probabilities.The overall risk is minimized by selecting a hypothesis that minimizes conditional risk for each input.
  • D SYNTHETIC DATA WITH ACCESS TO THE TRUE DISTRIBUTION: Under sequence-level loss Lseq(y, y′) = 1 − I(y = y′), the Bayes classifier chooses the most probable output sequence.
  • D SYNTHETIC DATA WITH ACCESS TO THE TRUE DISTRIBUTION: Token-level loss sums zero-one losses at individual time steps rather than evaluating only whole-sequence equality.

D.2 EXPERIMENTAL SETUPS AND ANALYSIS

The experiments compare real, sequence-level distilled, and token-level distilled datasets for a weaker Bi-LSTM classifier across synthetic HMM tasks. Results show that simpler deterministic datasets are easier to model, with the best dataset depending on whether evaluation is token- or sequence-level.

  • Experimental setup: 50 HMM trials compare Bi-LSTM classifiers trained on Dreal, Dtok, and Dseq using token- and sequence-level accuracy.The synthetic data generator varies across 50 random seeds, and both tacc and sacc are evaluated on test data.
  • Analysis: Dseq and Dtok define deterministic conditional distributions that are simpler for the weaker Bi-LSTM to model than the true HMM distribution.The experiment attributes the performance advantage to reduced distributional complexity.
  • Analysis: Token-level and sequence-level evaluation favor different distilled datasets because each dataset matches the corresponding prediction target.The token-level result is associated with Dtok, while the sequence-level result is associated with Dseq.
  • Connection to machine translation: The synthetic findings support the observation that NAT struggles to model the true conditional distribution of sentence pairs.Beam-search translations from a pretrained autoregressive model instead perform better on test BLEU.
Loading 1911.02727v3…