Source-linked AI summary
Non-Autoregressive Neural Machine Translation
Jiatao Gu, James Bradbury, Caiming Xiong, Victor O. K. Li, Richard Socher
TL;DR
Neural MT typically decodes autoregressively, which limits inference parallelism and increases latency. This paper introduces a non-autoregressive Transformer using fertility-based latent planning, distillation, and policy-gradient fine-tuning. Across three datasets, it narrows the teacher gap to 2–5 BLEU points, reaches within 0.2 BLEU of prior state of the art on WMT16 English–Romanian, and achieves roughly tenfold lower latency.
Problem
Autoregressive neural MT generates tokens sequentially, preventing full inference-time parallelism and making decoding particularly slow.
Method
The paper introduces a non-autoregressive Transformer that uses fertility-based latent planning, knowledge distillation, and policy-gradient fine-tuning for parallel decoding.
Results
Across three datasets, NAT performs 2–5 BLEU points below its autoregressive teacher, while WMT16 English–Romanian reaches within 0.2 BLEU points of prior overall state of the art.
Takeaways & Limitations
The model measures translation latencies at one-tenth those of an equal-sized autoregressive model while maintaining competitive BLEU scores.
Takeaways & Limitations
Noisy parallel decoding increases computational resources linearly with sample size, although independently parallel samples can limit latency growth when sufficient parallelism is available.
Abstract
from arXiv · showhide
Existing approaches to neural machine translation condition each output word on previously generated outputs. We introduce a model that avoids this autoregressive property and produces its outputs in parallel, allowing an order of magnitude lower latency during inference. Through knowledge distillation, the use of input token fertilities as a latent variable, and policy gradient fine-tuning, we achieve this at a cost of as little as 2.0 BLEU points relative to the autoregressive Transformer network used as a teacher. We demonstrate substantial cumulative improvements associated with each of the three aspects of our training strategy, and validate our approach on IWSLT 2016 English-German and two WMT language pairs. By sampling fertilities in parallel at inference time, our non-autoregressive model achieves near-state-of-the-art performance of 29.8 BLEU on WMT 2016 English-Romanian.
1 INTRODUCTION
Neural MT inference remains slow because autoregressive decoders generate tokens sequentially, preventing full inference-time parallelism. The paper introduces a Transformer-based non-autoregressive model that predicts fertilities to provide a globally consistent plan for parallel decoding.
- Autoregressive decoders generate each token conditioned on previously generated tokens, making inference sequential and particularly slow for neural MT.Neural networks must be run for each generated token.
- Removing recurrence or using more-parallelizable convolutions and self-attention does not eliminate sequential inference when decoding remains autoregressive.
- The proposed Transformer-based non-autoregressive model predicts input-token fertilities that give the decoder a globally consistent plan for simultaneously computed outputs.Fertilities are supervised during training and used at inference time.
2 BACKGROUND
Autoregressive translation supports correlated target distributions and effective training, but its sequential decoding limits inference parallelism. Non-autoregressive decoding computes outputs in parallel, while complete conditional independence creates a multimodality problem that the paper addresses with a modified model and training techniques.
- Autoregressive decoding: Autoregressive NMT factors output probabilities left to right, enabling direct cross-entropy supervision at each decoding step.The target sentence is known during training, allowing conditional probabilities to be computed in parallel even though inference is sequential.
- Autoregressive decoding: Transformer self-attention reduces sequential computation during training, but autoregressive decoding still prevents full inference-time parallelism.
- Autoregressive decoding: Autoregressive models capture real-translation distributions effectively and achieve state-of-the-art performance, while beam search provides an effective local search method.
- Non-autoregressive decoding: A naïve non-autoregressive decoder computes output distributions in parallel but assumes each token depends only on the source sentence.
- The multimodality problem: Complete conditional independence cannot represent correlated alternatives such as “Danke schön.” and “Vielen Dank.” without also licensing mismatched combinations.
- The multimodality problem: The conditional-independence assumption prevents proper modeling of multimodal target distributions, motivating the NAT model and new training techniques.
3 THE NON-AUTOREGRESSIVE TRANSFORMER (NAT)
The Non-Autoregressive Transformer produces translations in parallel by replacing autoregressive decoder inputs with fertility-conditioned source representations. Fertilities act as a latent sentence-level plan that reduces multimodality while enabling tractable approximate decoding.
- Architecture: NAT combines an encoder, decoder, fertility predictor, and translation predictor to generate an entire translation in parallel.Its feed-forward and attention-based architecture has no inherent sequential execution requirement.
- Decoder stack: The decoder uses copied source inputs, non-causal self-attention, and positional attention to support parallel generation and local reordering.Positional attention uses positional encodings as queries and keys and decoder states as values.
- Modeling multimodality: Fertilities provide a latent sentence-level plan that captures cross-output correlations while leaving local translation decisions for the decoder.The desired latent variable should reduce dependence across output positions without making p(y|x, z) trivial to learn.
- Fertility-conditioned decoding: Fertilities specify how many target words align to each source word, determining decoder inputs and the output length through their sum.The model copies each source representation according to its predicted fertility.
- Fertility prediction: Fertility prediction independently models each source position with a softmax classifier, using encoder representations that incorporate sentence context.The fertility values are supervised during training and condition the decoder at inference time.
- Decoding process: Because exact marginalization over fertility sequences is intractable, NAT uses heuristic decoding methods that reduce the search space.NPD samples fertility sequences and scores the resulting translations, but its computational resources grow linearly with sample size.
4 TRAINING
The training strategy combines supervised fertility modeling, sequence-level knowledge distillation, and reverse-KL fine-tuning to improve non-autoregressive translation. It also addresses residual nondeterminism while exposing dependence on approximate fertility inference.
- Fertility modeling: The non-autoregressive model introduces fertilities as a discrete latent variable, with training based on a proposal distribution q over fertility sequences.The proposal may come from an external aligner or a fixed autoregressive teacher's attention weights.
- Fertility modeling: Supervised fertility training decomposes the maximum-likelihood objective into translation and fertility losses, jointly training the translation and fertility models.This decomposition uses inferred fertilities to supervise the fertility network and the translation model.
- Sequence-level knowledge distillation: Sequence-level knowledge distillation trains a teacher autoregressively and uses its greedy outputs as targets for the non-autoregressive student.The distilled targets are more deterministic and less noisy, but lower in quality than the original dataset.
- Fertility modeling: The supervised fertility approach relies heavily on deterministic approximate inference from an external alignment system instead of end-to-end training of the fertility predictor.The authors identify this reliance as a drawback relative to variational training.
- Fine-tuning: After convergence, fine-tuning adds reverse-KL divergence with the teacher output distribution as a word-level knowledge-distillation objective.The loss favors highly peaked student output distributions over standard cross-entropy.
- Fine-tuning: Joint fine-tuning combines the original distillation loss with fertility-based expectation terms, estimating the non-differentiable gradient with REINFORCE.The external fertility-inference term is trained with ordinary backpropagation.
5 EXPERIMENTS
Experiments evaluate NAT across IWSLT16 En–De, WMT14 En–De, and WMT16 En–Ro, combining distillation, fertility-based inputs, and fine-tuning with noisy parallel decoding. NAT is over 10× faster than greedy autoregressive decoding while narrowing the BLEU gap to its teacher and approaching state-of-the-art performance on WMT16 En–Ro.
- Experimental settings: NAT models use autoregressive Transformer teachers for distillation, scoring, and noisy parallel decoding, with matched student–teacher sizes and hyperparameters.Teacher parameters are frozen after training, and the full training set is decoded once to create each student’s distillation corpus.
- Results: Across three datasets, NAT is 2–5 BLEU points below its autoregressive teacher, while noisy parallel decoding addresses part or all of the gap.On WMT16 English–Romanian, NPD brings NAT within 0.2 BLEU points of the previous overall state of the art.
- Results: More than 10× speedup over greedy autoregressive decoding and 15× over beam search is measured on the development model.With multi-GPU parallelization, NPD latency could be reduced to about 80ms because samples are generated and scored independently.
- Ablation study: Around 5 BLEU points come from training on distilled rather than ground-truth targets, while fertility-based copying adds four points with ground-truth training or two with distillation.The ablation study also finds that positional embeddings alone fail to train the model.
- Ablation study: Using all three fine-tuning terms together improves performance by around 1.5 BLEU points, whereas reinforcement learning or the LBP term alone does not converge.Beam-search distillation performs similarly to greedy distillation.
- Noisy parallel decoding: Noisy parallel decoding samples fertility sequences to produce diverse candidate translations that the autoregressive teacher scores and selects.In examples, NPD removes repeated-word errors seen in NAT outputs, though its translations are noticeably more literal than autoregressive outputs.
6 CONCLUSION
The paper introduces a latent-variable non-autoregressive translation model that exploits Transformer parallelism during inference. It measures one-tenth the latency of an equal-sized autoregressive model while maintaining competitive BLEU scores.
- Conclusion: The latent-variable NAT model enables a Transformer decoder to exploit internal parallelism during inference.The model uses a latent representation to avoid autoregressive decoding while preserving competitive translation quality.
- Conclusion: One-tenth the latency of an equal-sized autoregressive model is measured while maintaining competitive BLEU scores.
A SCHEMATIC AND ANALYSIS
The figures analyze how non-autoregressive translation changes inference latency and how its BLEU score evolves during training and fine-tuning. NAT latency is nearly constant for typical sentence lengths with moderate fertility sampling, while larger sampling can restore linear latency.
- Latency analysis: NAT latency is nearly constant across typical sentence lengths, unlike the autoregressive model's linear decoding latency.The comparison uses single-sentence decoding without minibatching on the IWSLT development set.
- Latency analysis: NPD with sample size 10 preserves nearly constant latency, whereas sample size 100 more than saturates the GPU and produces linear latency.
- Training analysis: Learning curves track BLEU on the IWSLT development set during NAT training and fine-tuning.