Source-linked AI summary
Towards Robust Neural Machine Translation
Yong Cheng, Zhaopeng Tu, Fandong Meng, Junjie Zhai, Yang Liu
TL;DR
Small input perturbations can destabilize NMT representations and degrade translation quality, especially with noisy inputs. The paper addresses this with adversarial stability training for both encoder and decoder behavior, and experiments across three language pairs report improved robustness and translation performance. The authors note that evaluation on natural noise and newer architectures remains necessary.
Problem
Small input changes can severely distort NMT translations, while noisy ASR outputs, misspellings, and typos challenge model stability.
Method
Adversarial stability training makes encoder and decoder behavior similar for original inputs and their perturbed counterparts using noisy-data and adversarial objectives.
Results
Experiments on Chinese-English, English-German, and English-French tasks show improved robustness and translation performance.
Takeaways & Limitations
The approach supports robustness improvements across language pairs and is intended to extend to arbitrary noisy perturbations.
Takeaways & Limitations
The approach still requires validation on natural noise in practical applications and on advanced architectures such as CNN-based NMT and Transformer.
Abstract
from arXiv · showhide
Small perturbations in the input can severely distort intermediate representations and thus impact translation quality of neural machine translation (NMT) models. In this paper, we propose to improve the robustness of NMT models with adversarial stability training. The basic idea is to make both the encoder and decoder in NMT models robust against input perturbations by enabling them to behave similarly for the original input and its perturbed counterpart. Experimental results on Chinese-English, English-German and English-French translation tasks show that our approaches can not only achieve significant improvements over strong NMT systems but also improve the robustness of NMT models.
1 Introduction
NMT models can be highly sensitive to small input changes, which disrupt translations and creates challenges for noisy inputs such as ASR errors, misspellings, and typos. The paper proposes adversarial stability training for encoder and decoder robustness, reporting improved robustness and translation performance across multiple language pairs.
- Motivation: Small input perturbations can severely distort NMT translations, including changes caused by synonym replacement.The paper also connects this brittleness to noisy ASR outputs, misspellings, and typos.
- Approach: Adversarial stability training makes encoder representations consistent across original and perturbed inputs while jointly training the decoder on both.The method introduces noisy-input objectives and adversarial learning for the encoder and decoder.
- Results: The approach improves both NMT robustness and translation performance.The paper reports these as joint benefits of adversarial stability training.
- Approach: The framework is designed to support arbitrary noisy perturbations, including task-specific perturbation methods.The paper presents this flexibility as an advantage of its training framework.
- Approach: The training method is architecture-independent and can be applied to arbitrary NMT systems.The stated design does not depend on a specific NMT architecture.
- Results: Experiments on Chinese-English, English-French, and English-German tasks show significant improvements across language pairs.The system outperforms the RNN-based GNMT system and obtains comparable performance with a CNN-based NMT system.
2 Background
NMT uses an encoder-decoder architecture to model target translations conditioned on source sentences. Its standard training objective optimizes translation probability, but small input perturbations can severely degrade translation quality.
- The model directly optimizes the probability of a target sentence conditioned on its corresponding source sentence.
- NMT encodes a source sentence into hidden representations and uses a decoder with attention to generate the translation.
- The encoder and decoder parameters are trained by minimizing negative log-likelihood over the training corpus.
- Small perturbations can dramatically deteriorate NMT translation results, motivating stability improvements for distributed representations.
- Figure 1 contrasts forward information flow from the original input x with flow from a perturbed input x′.
3 Approach
Adversarial stability training creates semantically similar perturbed inputs and trains NMT encoders and decoders to preserve representations and translations. The framework combines original and noisy translation objectives with an adversarial encoder objective and supports multiple perturbation strategies.
- 3.1 Overview: The approach constructs perturbed sentences x′ with similar semantics to x and seeks both close encoder representations and robust decoder outputs.
- 3.1 Overview: The invariant objective encourages similar intermediate representations for x and x′ through adversarial learning.
- 3.1 Overview: The noisy-input objective trains the decoder to generate y from x′ and can be expressed as a translation loss or a KL divergence.
- 3.1 Overview: The original translation objective is retained to preserve translation performance while adding stability objectives.
- 3.1 Overview: The training objective jointly optimizes translation and stability losses, with α and β balancing the original task against model stability.
- 3.1 Overview: Training on perturbed neighbors augments the data and can improve both output robustness and performance on the original task.
- 3.2 Constructing Perturbed Inputs: Perturbations are constructed at lexical or feature level by replacing words with semantically similar vocabulary items or adding Gaussian noise to embeddings.
- 3.2 Constructing Perturbed Inputs: The perturbation framework allows task-specific strategies, including homonym replacement for simultaneous translation.
4.1 Setup
The experiments evaluate adversarial stability training across multiple translation tasks using a two-layer RNN-based NMT baseline and lexical- or feature-level perturbations.
- Datasets and evaluation: Experiments report 4-gram BLEU across Chinese-English, English-German, and English-French translation tasks.Chinese-English uses NIST data, while English-German uses WMT 14 and English-French uses IWSLT data.
- Datasets and evaluation: English-German and English-French use case-sensitive tokenized BLEU, whereas Chinese-English uses case-insensitive tokenized BLEU.
- Baseline system: The baseline is a two-layer RNN encoder-decoder with residual connections, GRUs, layer normalization, dropout, shared embeddings, and Adam optimization.
- Adversarial stability training: Adversarial stability training is initialized from maximum-likelihood parameters and includes lexical-level ASTlexical and feature-level ASTfeature variants.Training samples one perturbed neighbor for efficiency and uses a CNN discriminator for variable-length encoder representations.
4.2 Translation Results
Adversarial stability training improves translation quality across Chinese-English, English-German, and English-French tasks, outperforming strong RNN baselines and remaining competitive with CNN-based systems.
- Chinese-English: +3.16 BLEU is the largest improvement over Shen et al. (2016), while gains over Wang et al. (2017) and Zhang et al. (2018) reach +3.51 and +2.74 BLEU.
- Chinese-English: ASTlexical improves the Chinese-English baseline by +1.75 BLEU on average, while ASTfeature improves it by +2.59 BLEU on average and up to +3.34 BLEU on NIST08.
- English-German: +0.66 BLEU lets the English-German system outperform GNMT, while its performance is comparable with the 15-layer CNN system of Gehring et al. (2017).
- English-German: ASTlexical yields a +1.11 BLEU improvement on English-German, and ASTfeature performs slightly better.
- English-French: The English-French models consistently improve translation performance over the maximum-likelihood baseline across the evaluated datasets.ASTfeature significantly improves tst2015, while ASTlexical obtains comparable results there.
4.3 Results on Synthetic Perturbed Data
Synthetic swap, replacement, and deletion perturbations test robustness, and both adversarial stability variants outperform maximum-likelihood training as perturbation severity increases.
- Perturbation design: The robustness evaluation applies swap, replacement, and deletion operations to source sentences.Swap exchanges selected words with right neighbors, replacement substitutes sampled words, and deletion removes randomly selected words.
- Overall robustness: ASTlexical and ASTfeature consistently outperform MLE across all tested numbers of perturbation operations.
- Overall robustness: As operations increase, MLE performance drops quickly, whereas both adversarial stability approaches remain above MLE.
- Perturbation severity: For ASTlexical, the advantage over MLE grows from +2.19 at zero operations to +3.20, +9.39, and +3.12 at five operations for swap, replacement, and deletion.
- Perturbation-specific behavior: ASTlexical performs significantly better than MLE and ASTfeature on replacement perturbations after more than one operation, consistent with its replacement-based training data.
- Perturbation-specific behavior: The results support constructing task-specific perturbations, including pronunciation-preserving wrong-word substitutions for speech-translation pipelines.
4.4 Analysis
The analysis examines loss-function contributions and training dynamics for adversarial stability training. Removing Lnoisy causes the largest measured performance drop, while BLEU and loss curves oscillate during training.
- Ablation Study: Removing Lnoisy decreases translation performance by 1.66 BLEU points, whereas removing Ladv decreases it by 0.64 BLEU point.Using only Lnoisy increases performance by 0.88 BLEU point.
- BLEU Scores over Iterations: ASTlexical and ASTfeature show similar BLEU trajectories, with rapid initial drops followed by quick increases.Both methods are initialized from a model trained by MLE.
- BLEU Scores over Iterations: The largest BLEU declines reach about 7.0 points, and the training curves exhibit oscillations.The analysis attributes the instability to random perturbations and adversarial learning.
- Loss Curves: Ltrue, Linv, and Lnoisy decrease nonsteadily, with oscillations that are less sharp than those in the BLEU curves.Linv converges to around 0.68 after about 100K iterations, indicating that the discriminator cannot distinguish positive and negative samples.
5 Related Work
The work builds on adversarial learning and data augmentation while differing from prior NMT augmentation methods. It constructs perturbed data without complicated generation models or extra monolingual or bilingual corpora.
- Research Foundations: The paper is inspired by adversarial learning and data augmentation.These are identified as the two lines of research motivating the work.
- Adversarial Learning: Adversarial examples have been used to make trained networks resist perturbations and improve robustness.The cited prior work spans computer vision and natural language processing.
- Data Augmentation: Prior NMT data augmentation methods use monolingual corpora and complex models to generate translation equivalents and pseudo-corpora.These pseudo-corpora are then added to parallel training data.
- Distinction from Prior Work: Unlike those approaches, this work does not require complicated perturbed-data generation models or extra monolingual or bilingual corpora.The paper characterizes its approach as more convenient.
6 Conclusion
The paper proposes adversarial stability training to make NMT encoders and decoders robust to input perturbations. Experiments across three language pairs improve both robustness and translation performance, while further evaluation on natural noise and advanced architectures remains needed.
- Conclusion: Adversarial stability training makes encoder and decoder behavior similar for original inputs and perturbed counterparts.The paper proposes two approaches for constructing perturbed data to train the encoder and stabilize the decoder.
- Conclusion: Experiments on Chinese-English, English-German, and English-French tasks improve both robustness and translation performance.The conclusion reports this result across the three evaluated language pairs.
- Future Evaluation: Further evaluation is needed on natural application noise and more advanced NMT architectures such as CNN-based NMT and Transformer.The paper specifically mentions homonyms in simultaneous translation as an example of natural noise.