Source-linked AI summary
Non-Autoregressive Neural Machine Translation with Enhanced Decoder Input
Junliang Guo, Xu Tan, Di He, Tao Qin, Linli Xu, Tie-Yan Liu
TL;DR
NAT models reduce inference latency but lose accuracy because their decoder lacks strong target-side context. This paper supplies enhanced decoder inputs through phrase-table translation or source-to-target embedding mapping, and reports substantial BLEU gains over NAT baselines across translation tasks. The methods outperform the baseline on all tasks, with gains ranging from 3.47 to 5.02 BLEU.
Problem
NAT models accelerate inference by removing previous-target-token dependencies, but weaker decoder inputs cause accuracy degradation compared with autoregressive translation.
Method
The paper enhances NAT decoder inputs with phrase-table-derived target tokens or source embeddings mapped into the target embedding space using sentence-level alignment and word-level adversarial learning.
Results
3.47 to 5.02 BLEU gains over the baseline are reported across all tasks, with state-of-the-art performance among non-autoregressive baselines.
Takeaways & Limitations
Providing decoder inputs closer to target tokens makes the NAT decoder easier to learn and improves translation accuracy across the evaluated tasks.
Takeaways & Limitations
Phrase-table lookup is sensitive to phrase-table quality, performing poorly on large noisy datasets such as WMT14, while embedding mapping is more stable.
Abstract
from arXiv · showhide
Non-autoregressive translation (NAT) models, which remove the dependence on previous target tokens from the inputs of the decoder, achieve significantly inference speedup but at the cost of inferior accuracy compared to autoregressive translation (AT) models. Previous work shows that the quality of the inputs of the decoder is important and largely impacts the model accuracy. In this paper, we propose two methods to enhance the decoder inputs so as to improve NAT models. The first one directly leverages a phrase table generated by conventional SMT approaches to translate source tokens to target tokens, which are then fed into the decoder as inputs. The second one transforms source-side word embeddings to target-side word embeddings through sentence-level alignment and word-level adversary learning, and then feeds the transformed word embeddings into the decoder as inputs. Experimental results show our method largely outperforms the NAT baseline~\citep{gu2017non} by $5.11$ BLEU scores on WMT14 English-German task and $4.72$ BLEU scores on WMT16 English-Romanian task.
1 Introduction
NAT models accelerate inference by generating target tokens in parallel, but weaker decoder inputs reduce accuracy; this paper enhances those inputs with target-side information.
- 1 Introduction: NAT models generate target tokens simultaneously from source-derived inputs, avoiding the sequential dependence that makes AT inference a bottleneck.NART copies source tokens according to predicted fertilities before parallel prediction.
- 1 Introduction: Weaker, source-domain decoder inputs leave NAT models with less target-side context than AT models, causing inferior accuracy, especially for long sentences.The copied source tokens provide indirect target-side information, whereas AT inputs are previously generated target tokens.
- 1 Introduction: The paper proposes two decoder-input enhancements: phrase-table translation of source tokens and learned mapping of source embeddings into the target embedding space.Embedding mapping combines sentence-level L2 alignment with word-level adversarial learning.
- 1 Introduction: The enhanced decoder generates all target tokens in parallel from global inputs closer to target tokens, reducing the decoder’s learning difficulty.The enhanced input is denoted ˆy and supplies coarse target-side information.
- 1 Introduction: 24.28 and 34.51 BLEU outperform the NAT baseline’s 19.17 and 29.79 on WMT14 English-German and WMT16 English-Romanian, respectively.The experiments cover three translation tasks.
2 Background
AT models generate target tokens sequentially with previous-token context, whereas NAT models parallelize decoder prediction but lose those dependencies and accuracy.
- 2.1 Autoregressive Neural Machine Translation: AT decoding is sequential because each target token conditions on previously predicted tokens, creating an inference bottleneck despite parallelizable training in CNN and self-attention models.The encoder-decoder framework includes RNN, CNN, and Transformer architectures.
- 2.2 Non-Autoregressive Neural Machine Translation: NAT removes target-side conditional dependencies and generates all decoder outputs simultaneously from source-derived inputs, breaking the autoregressive inference bottleneck.NART constructs decoder input z by copying source tokens according to predicted fertilities.
- 2.2 Non-Autoregressive Neural Machine Translation: In NART, target length equals the sum of predicted fertilities, and the NAT objective is defined as a negative log-likelihood over parallel target predictions.The decoder input is consistently denoted z, representing the copied source sequence or enhanced input.
- 2.2 Non-Autoregressive Neural Machine Translation: NAT models can achieve 15× speedup but suffer accuracy degradation because the decoder must infer target-side sentence structure from source-side information alone.The study reports repetitive, semantically incoherent outputs with missing words in NART.
3 Methodology
ENAT enhances NAT decoder inputs with either phrase-table translations or learned target-side embeddings, addressing the weak target context available to parallel decoding. The embedding approach combines sentence-level alignment with word-level adversary learning, while phrase-table lookup is simpler but more sensitive to table quality.
- 3 Methodology: ENAT introduces token-level phrase-table lookup and embedding-level mapping to provide NAT decoders with coarse target-side information.Both methods replace source-side decoder inputs with target-oriented signals.
- 3.1 Phrase-Table Lookup: Phrase-table lookup greedily translates source phrases using a Moses-trained table, adding only 0.14ms latency per sentence on newstest2014.The method searches progressively shorter phrases when longer entries are unavailable and skips untranslated phrases during inference.
- 3.2 Embedding Mapping: Embedding mapping linearly projects source embeddings into the target embedding space and trains the projection end-to-end with the NAT model.The mapped inputs are reconstructed to target length and are intended to encode coarse target-side information in source order.
- 3.3 Discussion: Phrase-table lookup is effective when tables are accurate but can hurt training on noisy data, whereas jointly learned embedding mapping produces more stable results.The paper reports good phrase-table behavior on small, clean IWSLT14 data but poor behavior on large, noisy WMT14 data.
- 3.2 Embedding Mapping: Sentence-level alignment supplies bilingual supervision, while word-level adversary learning adds fine-grained token-level guidance by making mapped embeddings resemble target embeddings.The discriminator distinguishes mapped source embeddings from ground-truth target embeddings, and the generator learns to fool it.
4 Experimental Setup
The experiments use three translation benchmarks with shared preprocessing and architectures matched to the NART baseline. Training includes sequence-level knowledge distillation, while inference predicts target length and can use teacher rescoring for multiple candidates.
- 4.1 Datasets: Experiments cover IWSLT14 German-English, WMT14 English-German, and WMT16 English-Romanian, with 153K, 4.5M, and 2.9M training pairs respectively.Validation and test splits follow established task-specific benchmarks.
- 4.2 Model Configurations: The model uses Transformer-style multi-head attention, encoder-decoder attention, feed-forward networks, and multi-head positional attention to preserve parallel decoding.The architecture matches NART for fair comparison, with smaller five-layer components for IWSLT14 and larger six-layer components for WMT tasks.
- 4.3 Training and Inference: Training applies sequence-level knowledge distillation from an autoregressive Transformer teacher using beam size 4 to provide less noisy student targets.The teacher has the same architecture as the non-autoregressive student.
- 4.3 Training and Inference: At inference, the method predicts target length based on source–target length similarity, optionally generates 9 candidates with B = 4, and uses an autoregressive teacher model to rescore and select the final translation.Inference uses α = 1.1 for English-to-Others tasks and α = 0.9 for Others-to-English tasks.
- 4.3 Training and Inference: Decoder-input construction and teacher rescoring add little latency because lookup, dense-matrix multiplication, and rescoring are parallelizable.BLEU is used for evaluation, with tokenized case-sensitive scoring on WMT and tokenized case-insensitive scoring on IWSLT14.
5 Results
Across three evaluation subsections, the proposed decoder-input enhancements improve NAT translation quality, especially where weak inputs cause errors, while analysis links performance to input quality and complementary mapping losses.
- 5.1 Translation Quality and Inference Latency: Our model achieves state-of-the-art performance among non-autoregressive baselines and is nearly five times faster than NART with rescoring 100 candidates.The comparison covers translation quality and inference latency across multiple tasks.
- 5.1 Translation Quality and Inference Latency: BLEU accuracy increasingly favors the proposed method over NART as sentence length grows, consistent with stronger decoder-input information helping long sentences.Figure 2 compares AT, NART, and the proposed method across reference-length buckets on newstest2014.
- 5.2 Case Study: The case studies show NART missing meaningful words and repeating phrases, whereas both proposed methods correctly translate examples where stronger decoder inputs help.One example concerns a long sentence; another repeats “to you” and omits “all of” under NART.
- 5.3 Method Analysis: Phrase-table lookup performs better on IWSLT14 De-En and WMT14 De-En, while embedding mapping performs better on the other two tasks.The comparison is attributed to phrase-table quality: low-quality tables can provide noisy or misleading decoder signals.
- 5.3 Method Analysis: Weaker word-table decoder inputs reduce NAT accuracy relative to phrase-table inputs on WMT14 En-De.The study directly compares lookup quality and resulting NAT outputs using BLEU scores.
- 5.3 Method Analysis: Adding word-level adversary learning to sentence-level alignment raises the BLEU score to 24.13, showing complementary information from both losses.The ablation evaluates the embedding-mapping approach on IWSLT14 De-En.
6 Conclusion
The paper proposes phrase-table and word-embedding methods for enhancing NAT decoder inputs and reports improvements over the baseline across all evaluated tasks. Future work will broaden evaluation, improve phrase-table use, and explore other decoder-input enhancements.
- 6 Conclusion: The two proposed decoder-input methods outperform the NAT baseline on all tasks by 3.47 to 5.02 BLEU.One method uses a phrase table, while the other uses word embeddings.
- 6 Conclusion: Future work will test the methods on more language pairs and larger datasets, improve phrase-table utilization, and explore alternative decoder-input enhancements.The authors specifically mention sampling multiple candidate target tokens from phrase tables.