Source-linked AI summary
Fully Non-autoregressive Neural Machine Translation: Tricks of the Trade
Jiatao Gu, Xiang Kong
TL;DR
Fully NAT reduces translation latency through parallel token prediction but suffers from output-dependency modeling errors that lower quality. This paper combines dependency-reduction techniques across four aspects, achieving state-of-the-art fully NAT results and performance comparable to autoregressive and iterative systems, including 27.49 BLEU at 16.5× speed-up on WMT14 En-De.
Problem
Fully NAT must close its translation-quality gap with autoregressive and iterative NAT models while retaining its single-pass latency advantage.
Method
The paper reduces output-token dependencies by combining techniques across the training corpus, architecture, training objective, and learning strategy.
Results
27.49 BLEU on WMT14 En-De is achieved at 16.5× faster inference, while the system reaches state-of-the-art fully NAT results and comparable Transformer performance.
Takeaways & Limitations
Dependency-reduction techniques are complementary, and their combination enables fully NAT models to approach autoregressive translation quality without iterative decoding.
Takeaways & Limitations
CTC up-sampling constrains NAT batch size, limiting computation to 15K tokens versus 120K for the AT base model.
Abstract
from arXiv · showhide
Fully non-autoregressive neural machine translation (NAT) is proposed to simultaneously predict tokens with single forward of neural networks, which significantly reduces the inference latency at the expense of quality drop compared to the Transformer baseline. In this work, we target on closing the performance gap while maintaining the latency advantage. We first inspect the fundamental issues of fully NAT models, and adopt dependency reduction in the learning space of output tokens as the basic guidance. Then, we revisit methods in four different aspects that have been proven effective for improving NAT models, and carefully combine these techniques with necessary modifications. Our extensive experiments on three translation benchmarks show that the proposed system achieves the new state-of-the-art results for fully NAT models, and obtains comparable performance with the autoregressive and iterative NAT systems. For instance, one of the proposed models achieves 27.49 BLEU points on WMT14 En-De with approximately 16.5X speed up at inference time.
1 Introduction
Fully NAT generates translations in parallel to reduce latency, but output-token independence causes a quality gap. The paper addresses this gap through dependency reduction across training data, architecture, objectives, and learning strategy, achieving competitive accuracy with substantial speed-ups.
- 1 Introduction: Fully NAT improves inference parallelism but struggles because independently predicted target tokens cannot represent real-data dependencies.Iterative refinement improves accuracy by conditioning on partially observed references, but can reduce the speed advantage.
- 1 Introduction: The paper uses dependency reduction as its central principle for training fully NAT models.The approach revisits methods spanning the training corpus, model architecture, training objective, and learning strategy.
- 1 Introduction: Over 16× speed-up with performance comparable to the Transformer baseline is achieved across five translation directions.The proposed system also achieves state-of-the-art results among fully NAT models.
2 Motivation
NAT predicts all target tokens in parallel, but its conditional-independence assumption conflicts with dependencies in translation outputs. A toy example and prior comparisons motivate improving fully NAT without relying on iterative decoding.
- 2 Motivation: The independence assumption causes serious performance degradation because target-token dependencies generally exist in real translation distributions.The paper identifies this as a fundamental NAT modeling issue.
- 2 Motivation: Vanilla NAT assigns probability to impossible outputs such as AA and BB when training examples are only AB and BA.The example illustrates how maximum-likelihood training fails when output-token dependencies matter.
- 2 Motivation: Iterative NAT can reach autoregressive-level quality, but its latency advantage is reduced by multiple refinement steps.Fully NAT therefore remains an open direction for retaining parallel decoding speed.
- 2 Motivation: Fully NAT makes parallel predictions in one neural-network forward, yet still lags in translation quality behind autoregressive models.This preserves the speed advantage while exposing the central quality challenge.
3 Methods
The proposed fully NAT system reduces target-token dependencies across training data, latent-variable modeling, loss computation, and learning strategy, combining complementary techniques to improve performance.
- 3.1 Training Corpus: Knowledge distillation simplifies training data by replacing original targets with autoregressive-model outputs that are less noisy and more deterministically aligned with inputs.The distilled corpus is generated by a pretrained autoregressive teacher.
- 3.2 Model: Latent Variables: Latent variables model output tokens as conditionally independent given z, reducing dependencies with barely additional inference cost.The paper uses continuous spherical-Gaussian latent variables at encoder positions and trains them with a VAE-style ELBO.
- 3.3 Loss Function: Latent Alignments: CTC marginalizes valid latent alignments, allowing predictions to recover the target after collapsing repeated tokens and removing blanks.CTC reduces sensitivity to target offsets but assumes monotonic alignment, so it cannot remove all target dependencies.
- 3.4 Learning: Glancing Targets: Glancing training conditions prediction on randomly sampled reference tokens, while curriculum learning increases masking as model quality improves.The curriculum exposes poorly trained models to more target tokens and gradually trains them to generate complete sentences.
- The methods target dependency reduction through complementary changes to the training corpus, model architecture, training objective, and learning strategy.The paper emphasizes that no single technique perfectly removes target-token dependencies, motivating their combination.
4 Experiments
Experiments across three translation benchmarks evaluate fully NAT models using dependency-reduction techniques, latency measures, and comparisons with autoregressive and iterative NAT systems. The resulting models achieve strong quality–latency trade-offs, including near-AT quality at over 16× speed-up, while revealing CPU and large-batch limitations.
- WMT’14 EN↔DE & WMT’16 EN↔RO: 27.49 BLEU on WMT14 EN-DE nearly matches AT's 27.48 while inference is 16.5× faster.The model uses a single greedy generation iteration and achieves similar quality to AT beam-search results.
- WMT’14 EN↔DE & WMT’16 EN↔RO: Over 16× speed-up is achieved on EN→DE with single-iteration fully NAT, while KD plus CTC already beats single-iteration NAT state of the art across four directions.VAE or GLAT further closes the gap or outperforms AT on both language pairs.
- WMT’20 JA→EN: The JA→EN benchmark is harder: AT (12-1) and fully NAT models underperform the AT teacher and produce shorter translations with BP < 0.9.The benchmark is larger and noisier, with linguistically distinct Japanese and English.
- Beam search & NPD: 21.41 BLEU with a 4-gram LM and beam search beats the teacher model with 11× speed-up, while CPU beam-search latency remains around 100 ms.Beam search is more effective than NPD with re-ranking, and combining both provides another half BLEU point.
- Quality v.s. Latency: CPU and large-batch GPU settings reduce NAT's speed advantage because CTC up-sampling increases computation and memory demands.AT base (12-1) handles 120K tokens per batch versus 15K for NAT with CTC, identified as a limitation for future research.
- Ablation Study: KD and CTC are fundamental for robust NAT, while adding VAE or GLAT achieves performance similar to AT models.CTC outperforms AXE, latent variables improve accuracy by around 1 BLEU, and GLAT is superior to RNG.
- Distillation corpus: NAT capacity improves marginally from a better distillation corpus, but increasing model size can surpass AT (12-1) with little LGPU latency effect.The evidence indicates that simply switching to a stronger distillation corpus does not fully resolve NAT capacity limitations.
5 Discussion and Future work
The discussion identifies knowledge distillation and CTC as important components for fully NAT, while highlighting information-loss, memory, and dependency-modeling limitations.
- Knowledge distillation is described as indispensable for reducing dependencies when training fully NAT models.
- Knowledge distillation may discard useful lexical information and negatively affect low-frequency-word lexical choices.
- CTC boosts fully NAT performance across datasets, but its required up-sampling constrains very long sequences and memory-limited devices.
- VAE and GLAT help but are insufficient when training NAT solely with monotonic-alignment losses such as CTC.
- The authors suggest stronger dependency-modeling methods and dynamic hierarchical up-sampling as future directions.
6 Related Work
Related work improves translation speed through hybrid autoregressive/non-autoregressive designs or by directly optimizing autoregressive models.
- Semi-autoregressive models use local non-autoregressive decoding while retaining autoregressive behavior globally.
- Local autoregressive NAT models provide an alternative hybrid design for improving decoding speed.
- Quantization and pruning are established approaches for directly accelerating autoregressive translation models.
- Teacher-student training can improve a student model's translation accuracy while providing faster decoding.
7 Conclusion
The work targets the performance gap between fully NAT and autoregressive models by combining dependency-reduction methods, achieving state-of-the-art fully NAT results.
- The proposed models achieve state-of-the-art results among fully NAT systems on three translation benchmarks.