Source-linked AI summary
Non-Autoregressive Machine Translation with Auxiliary Regularization
Yiren Wang, Fei Tian, Di He, Tao Qin, ChengXiang Zhai, Tie-Yan Liu
TL;DR
NAT improves inference efficiency through parallel decoding, but its missing target-side sequential dependency causes repeated and incomplete translations. This paper introduces similarity and reconstruction regularization for decoder representations, achieving stronger NAT accuracy while retaining efficient inference, including 24.61 BLEU for WMT14 En-De and 28.90 for De-En.
Problem
NAT omits target-side sequential dependency for efficient parallel inference, but consequently suffers from repeated and incomplete translations caused by poor decoder hidden representations.
Method
The paper trains NAT with similarity regularization for neighboring hidden states and reconstruction regularization through a backward translation model recovering the source sentence.
Results
24.61 BLEU in WMT14 En-De and 28.90 in De-En establish new state-of-the-art NAT performance while improving accuracy without sacrificing efficiency.
Takeaways & Limitations
The two training-only regularizers alleviate repeated and incomplete translations while preserving NAT's fast decoding advantage.
Takeaways & Limitations
The reconstruction approach does not use reconstruction-score reranking during inference because that step hurts inference efficiency.
Abstract
from arXiv · showhide
As a new neural machine translation approach, Non-Autoregressive machine Translation (NAT) has attracted attention recently due to its high efficiency in inference. However, the high efficiency has come at the cost of not capturing the sequential dependency on the target side of translation, which causes NAT to suffer from two kinds of translation errors: 1) repeated translations (due to indistinguishable adjacent decoder hidden states), and 2) incomplete translations (due to incomplete transfer of source side information via the decoder hidden states). In this paper, we propose to address these two problems by improving the quality of decoder hidden representations via two auxiliary regularization terms in the training process of an NAT model. First, to make the hidden states more distinguishable, we regularize the similarity between consecutive hidden states based on the corresponding target tokens. Second, to force the hidden states to contain all the information in the source sentence, we leverage the dual nature of translation tasks (e.g., English to German and German to English) and minimize a backward reconstruction error to ensure that the hidden states of the NAT decoder are able to recover the source side sentence. Extensive experiments conducted on several benchmark datasets show that both regularization strategies are effective and can alleviate the issues of repeated translations and incomplete translations in NAT models. The accuracy of NAT models is therefore improved significantly over the state-of-the-art NAT models with even better efficiency for inference.
Introduction
NAT accelerates inference by generating target tokens in parallel, but omitting target-side sequential dependency degrades translation quality. The paper addresses repeated and incomplete translations by regularizing decoder representations during training.
- Autoregressive decoding generates target words sequentially, substantially limiting NMT inference efficiency.
- NAT generates all target tokens in parallel by omitting target-side sequential dependency, boosting inference speed while reducing translation quality.
- Basic NAT frequently produces repeated and incomplete translations because its decoder hidden representations are difficult to learn without access to prior decoded states.
- Similarity regularization aligns neighboring hidden-state similarity with corresponding target-token embedding similarity to address repeated translation.
- Reconstruction regularization uses an autoregressive backward translation model to require NAT decoder hidden representations to recover the source sentence and address incomplete translation.
- 24.61 BLEU in En-De and 28.90 in De-En establish new state-of-the-art NAT performance on WMT14 while improving accuracy without sacrificing efficiency.
Background
NMT commonly uses an encoder-decoder architecture with autoregressive target generation, creating an inference-latency bottleneck. NAT-related work addresses this bottleneck through parallel decoding, architectural changes, distillation, and discrete variables.
- NMT encodes source sentences into representations and generates target tokens autoregressively according to prior target tokens and the source.
- Autoregressive generation prevents parallelism during inference, limiting GPU utilization and contributing to high decoding latency.
- Sequence-level knowledge distillation trains NAT models using translations sampled from an autoregressive teacher as bilingual training data.
- NAT architectural modifications remove the decoder causal mask and use positional attention to enhance target-position information.
- Earlier NAT methods introduced intermediate discrete variables, such as fertility values or autoregressively generated latent variables, to compensate for lost sequential information.
Model
The model uses a basic NAT backbone and improves decoder representations with similarity and reconstruction regularization. These terms target repeated and incomplete translations while preserving efficient NAT inference.
- NAT-BASE uses the standard encoder-decoder architecture with sequence-level distillation, positional attention, and a decoder without a causal mask.
- Inference inputs: Target length is predicted as Ty = Tx + ∆T, using source length and a constant bias derived from training-set length statistics.
- Inference inputs: Decoder inputs are formed by uniformly mapping source-token embeddings to the predicted target length.
- The two regularizers address repeated and incomplete translations, whose observed causes include similar adjacent hidden states and neglected source tokens.
- Repeated translation: Similarity regularization makes adjacent hidden-state similarity follow the semantic similarity of their corresponding target-token embeddings, rather than imposing one fixed constraint.More dissimilar target tokens receive a stronger dissimilarity effect, while semantically related tokens may retain greater hidden-state similarity.
- Incomplete translation: A backward translation model reconstructs the source sentence from NAT decoder hidden states, encouraging those states to retain comprehensive source information.The backward model is used during training and not during inference.
- Design boundary: The reconstruction approach uses a complete backward encoder-decoder model, whereas a related method used a decoder without an encoder; reconstruction-based reranking is omitted because it hurts inference efficiency.
Evaluation
The evaluation compares the proposed approach with state-of-the-art methods on multiple benchmark datasets, measuring both translation accuracy and efficiency.
- The study evaluates the proposed regularization approach on multiple benchmark datasets against baselines representing state-of-the-art translation accuracy and efficiency.
Experiment Design
Experiments cover IWSLT14, IWSLT16, and WMT14 translation tasks, using NAT baselines, Transformer teachers, and tuned regularization weights.
- Datasets: The benchmark suite includes IWSLT14 De-En, IWSLT16 En-De, WMT14 En-De, and WMT14 De-En.
- Baselines: The baselines are NAT-FT, NAT-IR, and LT, with reported or reproduced performance used across four tasks.
- Teacher models: Transformer models serve as teacher models for sequence-level knowledge distillation and inference rescoring, with matched model sizes and hyperparameters for each NAT model and teacher.
- Model settings: NAT models use Transformer-based settings, including base configurations for WMT and small configurations for the IWSLT tasks.
- Model settings: The regularization weights are selected on the IWSLT14 De-En development set and reused across datasets, with α = 2 and β = 0.5.
Training and Inference
Training follows Transformer optimization practices, while inference generates parallel candidates at several target lengths and evaluates results with BLEU and latency.
- Training: All models are trained with Adam using Transformer learning-rate settings, on 8 GPUs for WMT and 1 GPU for IWSLT; distillation and inference use 1 GPU.
- Inference: Inference generates 2B + 1 candidates by decoding in parallel at target lengths around a source-length-based prediction.
- Evaluation: Table 1 reports test-set BLEU for AT and NAT models, distinguishing NAT-REG, its Transformer teacher, weak-teacher settings, and reproduced baselines.
- Evaluation: Latency is measured as average per-sentence decoding time without minibatching on one NVIDIA Tesla P100, using IWSLT14 DE-EN.
Results
NAT-REG substantially improves translation quality across benchmark datasets and achieves strong WMT14 performance while retaining fast inference.
- NAT-REG achieves the best translation quality on all benchmark datasets except IWSLT16 En-De.
- 24.61 BLEU on WMT14 En-De and 28.90 BLEU on WMT14 De-En establish new state-of-the-art NAT performance.
- 27.12 BLEU with a weak teacher on WMT14 De-En exceeds NAT-IR at 25.48 and NAT-FT at 22.41.
- 15.1× speedup over an AT model when rescoring 9 candidates is comparable to NAT-FT's 15.6× speedup without rescoring.
Analysis
Case studies and ablations show that the two regularizers alleviate repeated and incomplete translations, with similarity regularization specifically reducing repetition.
- Case Study: Table 2 compares AT, NAT-BASE, and NAT-REG outputs, marking incomplete translation pieces in italics and repeated tokens in bold.
- Case Study: NAT-BASE exhibits repeated and incomplete translations, whereas NAT-REG largely alleviates both issues.
- Ablation Study: Each regularization term improves basic NAT BLEU by about 1 point on IWSLT14 De-En, while universal similarity regularization lowers BLEU from 28.73 to 28.32.
- Analysis: With fixed target length, removing repeated tokens can leave more positions for valid tokens, potentially reducing incomplete translation.
- Ablation Study: Similarity regularization reduces per-sentence de-duplication operations from 2.3 to 0.9 on the IWSLT14 De-En development set.
Conclusions
The paper concludes that similarity and reconstruction regularization address two major NAT errors while preserving fast decoding. Their applicability beyond NAT remains an open direction, and the authors aim to surpass autoregressive teachers.
- Similarity and reconstruction regularization address repeated and incomplete translation, respectively, producing strong performance with fast decoding.
- The strategies may also apply to other sequence generation models, including image captioning and text summarization.
- The authors plan to exceed autoregressive teacher performance, noting that NAT has no training-inference gap from exposure bias.