Source-linked AI summary
Glancing Transformer for Non-Autoregressive Neural Machine Translation
Lihua Qian, Hao Zhou, Yu Bao, Mingxuan Wang, Lin Qiu, Weinan Zhang, Yong Yu, Lei Li
TL;DR
Non-autoregressive translation seeks parallel decoding without sacrificing Transformer-level quality, but existing approaches either underperform or require multiple passes. The paper introduces GLM and GLAT, which learn target-word interdependency through adaptive glancing during training while decoding in one pass. GLAT improves NAT by about 5 BLEU, approaches autoregressive performance within 0.9 BLEU while retaining 7.9× speed-up, and achieves competitive performance across benchmarks.
Problem
Non-autoregressive translation must learn target-word interdependency while preserving parallel, single-pass decoding comparable to Transformer.
Method
GLM trains GLAT with adaptive glancing, using initial prediction accuracy to select reference fragments and teaching the model to predict the remaining tokens.
Results
GLAT improves vanilla NAT by about 5 BLEU and closes the autoregressive performance gap within 0.9 BLEU while retaining 7.9× speed-up.
Takeaways & Limitations
GLAT demonstrates that single-pass parallel translation can achieve competitive performance with autoregressive models while maintaining non-autoregressive inference efficiency.
Takeaways & Limitations
Prior iterative methods require multiple decoding passes, while the stated target setting requires non-autoregressive single-pass inference.
Abstract
from arXiv · showhide
Recent work on non-autoregressive neural machine translation (NAT) aims at improving the efficiency by parallel decoding without sacrificing the quality. However, existing NAT methods are either inferior to Transformer or require multiple decoding passes, leading to reduced speedup. We propose the Glancing Language Model (GLM), a method to learn word interdependency for single-pass parallel generation models. With GLM, we develop Glancing Transformer (GLAT) for machine translation. With only single-pass parallel decoding, GLAT is able to generate high-quality translation with 8-15 times speedup. Experiments on multiple WMT language directions show that GLAT outperforms all previous single pass non-autoregressive methods, and is nearly comparable to Transformer, reducing the gap to 0.25-0.9 BLEU points.
1 Introduction
Existing non-autoregressive methods trade translation quality or decoding speed against parallelism. GLM and GLAT address this by adaptively using reference fragments during training while retaining single-pass parallel generation.
- Vanilla NAT lags behind Transformer by about 7.0 BLEU because conditional independence limits target-word interdependency modeling.
- Iterative NAT methods restore dependency modeling through multiple decoding passes, reducing their speed advantage and still lagging behind autoregressive Transformer with single-pass generation.
- GLAT generates translations with one parallel decoding pass and outperforms previous NAT methods while achieving performance comparable to Transformer in multiple cases.
- GLM trains GLAT with adaptive glancing: difficult references expose sampled fragments, and training gradually reduces glancing as the model improves.
- About 5 BLEU improvement over vanilla NAT is achieved without losing inference speed-up, while GLAT remains competitive with iterative Mask-Predict.
- Within 0.9 BLEU point of the autoregressive baseline, GLAT retains 7.9× speed-up and exceeds it for references shorter than 20 on WMT14 DE-EN.
2 Probability Models of Machine Translation
Machine-translation probability models differ in how they represent target-token dependence. Vanilla NAT enables parallel decoding through conditional independence, while iterative masked prediction reintroduces dependence through repeated refinement; GLM targets single-pass dependency learning.
- Transformer uses autoregressive left-to-right factorization, learning target-word interdependency unidirectionally and decoding tokens sequentially.
- Vanilla NAT copies encoder representations into a parallel decoder, generating all target tokens simultaneously under conditional independence.
- The conditional-independence approximation does not generally hold, contributing to vanilla NAT’s inferior translation performance.
- Mask-Predict retains conditional independence but randomly masks target words and iteratively refines them with a masked language model.
- GLM instead trains single-pass parallel generation by selecting reference tokens according to initial prediction accuracy and predicting the remaining tokens.
3 Glancing Transformer
GLAT uses the Glancing Language Model to encourage word interdependency during training while retaining single-pass parallel inference. Its adaptive sampling supplies reference tokens according to prediction difficulty, then trains the decoder to predict the remaining tokens.
- GLAT retains the vanilla NAT encoder-decoder architecture but differs by explicitly encouraging word interdependency through GLM training.
- During training, GLAT selects reference tokens and replaces corresponding decoder inputs before predicting the remaining target tokens.The selected target embeddings are inserted into the decoder input, and the loss is computed on the unselected tokens.
- Adaptive glancing sampling selects more reference tokens when the initial prediction is less accurate, guiding learning from fragments toward whole sentences.The sampling count decreases as training progresses and can be based on differences between the prediction and reference.
- The sampling function randomly selects S tokens from the reference, with S determined by a scaling factor and the prediction-reference difference.Hamming distance is used when the predicted and reference sentences have matching lengths; other distances can handle differing lengths.
- GLAT inference is fully parallel and uses only a single decoding pass, with output length predicted from encoder representations.Length prediction may use an added [LENGTH] token, while NPD and CTC provide more complex alternatives.
4 Experiments
Experiments evaluate GLAT on multiple translation benchmarks against Transformer and non-autoregressive baselines, then analyze speed, source length, repetition, and sampling strategies. GLAT improves translation quality while retaining single-pass inference efficiency, with adaptive glancing producing especially strong results.
- Experimental Settings: GLAT is evaluated on WMT14, WMT16, and IWSLT16 benchmarks against Transformer and representative NAT baselines.The experiments include main comparisons, ablations, and further analysis.
- Main Results: GLAT preserves single-pass inference efficiency while outperforming previous NAT methods in the speed-up–BLEU trade-off.Figure 3 places GLAT above and to the right of competing methods, whereas iterative models retain only minor speed advantages over autoregressive Transformer.
- Main Results: About 5 BLEU points improve over vanilla NAT-base on EN-DE/DE-EN, while GLAT nearly matches autoregressive Transformer.GLAT also outperforms other fully non-autoregressive models by almost +2 BLEU on average.
- Source Input Length: GLAT improves performance substantially on long source sentences, where NAT-base performance drops sharply.GLAT also outperforms autoregressive Transformer when source input length is smaller than 20.
- Repetition Analysis: GLAT significantly reduces token repetition, consistent with its explicit modeling of interdependency between target words.The repetition ratio can be reduced further with noisy parallel decoding.
- Ablation Study: Adaptive sampling outperforms fixed baseline schedules by large margins, while λ = 0.2 beats λ = 0.0 by 2.5 BLEU.The results support gradually moving from easier fragment generation to harder whole-sequence generation.
- Ablation Study: Hard-to-predict words are more useful for glancing than high-confidence words, while random selection performs slightly better than confidence-based strategies.GLAT therefore adopts random selection for its simplicity and good performance.
- Ablation Study: Glancing sampling contributes 5∼6 BLEU points over uniform sampling, while encoder representations add 0.2∼0.3 BLEU points.The ablation identifies adaptive glancing as the largest contributor to the final improvement.
5 Related Work
Prior non-autoregressive translation work addresses target-word dependencies through latent variables, knowledge transfer, alternative objectives, regularization, structured decoding, or iterative refinement. These approaches often add inference computation or multiple decoding passes.
- Fully Non-Autoregressive Models: Latent-variable methods reduce the burden of modeling dependencies among output words in fully non-autoregressive models.
- Fully Non-Autoregressive Models: Other approaches transfer autoregressive knowledge, change training objectives, or add regularization terms to improve non-autoregressive models.
- Non-Autoregressive Models with Structured Decoding: Structured-decoding methods model word dependencies with CRF inference or cascaded CRF decoding, adding sequential decoding after non-autoregressive computation.
- Non-Autoregressive Models with Iterative Refinement: Iterative-refinement methods improve outputs through multi-pass decoding, including denoising, insertion and deletion, or masked-language-model procedures.
6 Conclusion
The paper proposes Glancing Transformer with a glancing language model for single-pass parallel generation. It reports improved non-autoregressive translation performance, competitive results against autoregressive models, and a possible extension to other generation tasks.
- Glancing Transformer uses a glancing language model to improve single-pass parallel generation models.
- The model learns sequence fragments first and gradually moves toward generating whole sequences.
- Experimental results show significantly improved non-autoregressive machine translation performance with single-pass parallel generation.
- GLAT achieves competitive performance compared with autoregressive models, motivating its application to other generation tasks.