Source-linked AI summary
Neural Machine Translation with Supervised Attention
Lemao Liu, Masao Utiyama, Andrew Finch, Eiichiro Sumita
TL;DR
Attention-based NMT produces weaker alignments than conventional models because it must predict attention without the current target word. The paper analyzes this as a reordering issue, supervises attention with external alignments, and reports better alignments and translation performance on two Chinese-to-English tasks.
Problem
Attention-based NMT has lower alignment accuracy than conventional alignment models because it predicts alignments without the target word, while conventional models can use the full target sentence.
Method
SA-NMT jointly trains translation and attention, using alignments from conventional off-the-shelf aligners as supervision through a regularized objective.
Results
The approach achieves better alignments and significant gains over standard attention-based NMT on two Chinese-to-English translation tasks.
Takeaways & Limitations
Supervising attention with conventional alignment models improves the reported end-to-end translation results of attention-based NMT.
Takeaways & Limitations
SA-NMT still has an alignment gap with GIZA++; reducing AER may require emphasizing alignment in training or adding target information.
Abstract
from arXiv · showhide
The attention mechanisim is appealing for neural machine translation, since it is able to dynam- ically encode a source sentence by generating a alignment between a target word and source words. Unfortunately, it has been proved to be worse than conventional alignment models in aligment accuracy. In this paper, we analyze and explain this issue from the point view of re- ordering, and propose a supervised attention which is learned with guidance from conventional alignment models. Experiments on two Chinese-to-English translation tasks show that the super- vised attention mechanism yields better alignments leading to substantial gains over the standard attention based NMT.
1 Introduction
Attention-based NMT dynamically uses encoded source context during translation, but its alignments lag conventional models because they cannot use the current target word. The paper proposes supervised attention guided by external aligners and reports gains on two Chinese-to-English tasks.
- Motivation: Attention-based NMT dynamically uses encoded source context while generating translations token by token.This allows fewer hidden layers while maintaining high translation performance.
- Problem: 54 versus 30 AER shows the alignment-quality gap between attention-based NMT and conventional alignment models for Chinese-to-English translation.Attention predicts alignments without access to the target word, whereas conventional models can use it.
- Approach: SA-NMT jointly learns translation and attention using bilingual alignments from off-the-shelf conventional aligners as attention supervision.The supervision is implemented as a regularizer in the joint training objective.
- Approach: The proposed approach is simple, generally applicable to attention-based NMT models, and implemented here on top of Bahdanau et al.'s model.The paper frames supervised attention as guidance from statistical alignment models.
- Results: Up to 2.5 BLEU points over the strongest baseline were achieved on the large-scale task, while the low-resource task gained about 5 BLEU points over its attention-based NMT baseline.These results are reported for two Chinese-to-English translation tasks.
2 Revisiting Neural Machine Translation
The paper interprets NMT attention as a reordering model whose alignment prediction is learned without the target word. Conventional alignment models instead use the full target sentence, enabling richer alignment modeling.
- NMT computation: At timestep t, NMT uses hidden states and context vectors within a recurrent Encode-Decode computation graph.The context vector is a dynamic source representation used during decoding.
- Attention: αt,i represents the probability that target word yt aligns to source word xi, with αt produced by a two-layer feedforward network.The network's top layer is a softmax layer.
- Training: Although translation targets are supervised, attention remains unsupervised during standard NMT training because α is hidden.The distinction is between supervision of translation outputs and supervision of the intermediate attention variable.
- Reordering: Because yt is unavailable at timestep t−1 during testing, NMT must predict αt before observing the next target word.This makes attention function as a reordering model that selects source content likely to be translated next.
- Conventional alignment: Conventional alignment models define α over both x and y and can use the entire target sentence, including yt, to model alignments more sufficiently.The paper relates this access to the weaker alignment quality of attention-based NMT.
3 Supervised Attention
The method converts hard alignments into soft supervision and jointly trains translation with attention. A disagreement loss, weighted by λ, treats attention as an observable variable during training while preserving standard decoding at test time.
- Supervised attention: A conventional aligner supplies training-corpus alignments as supervision, while testing uses the same decoding procedure as standard NMT.The method is designed for unseen test sentences, where alignment supervision is unavailable.
- Preprocessing alignment supervision: Hard alignments are converted into soft distributions because target words may align to multiple source words or none.Unaligned words inherit affiliation from the closest aligned word, preferring the right; multiply aligned words receive equal weights.
- Jointly supervising translation and attention: The model jointly supervises translation and attention with a soft constraint that penalizes disagreement between predicted and reference alignments.The objective balances likelihood and alignment disagreement through the hyper-parameter λ.
- Jointly supervising translation and attention: Treating attention as an observable variable distinguishes supervised attention from standard NMT and resembles multi-task learning.The paper also states that λ can alleviate overfitting and that supervision can address vanishing gradients.
- Jointly supervising translation and attention: Three disagreement measures are investigated: mean squared error, a word-alignment agreement objective, and cross entropy.MSE directly encourages agreement, MUL is designed for word alignment, and cross entropy is natural for distributions.
4 Experiments
Experiments evaluate supervised attention on large-scale and low-resource Chinese-to-English translation, including loss and aligner choices, learning behavior, translation quality, and alignment accuracy. SA-NMT improves over standard NMT across settings, while remaining below Moses on the low-resource task.
- Experimental setup: Experiments use two Chinese-to-English tasks, evaluating translation quality with case-insensitive BLEU4.The tasks cover NIST news-domain translation and low-resource travel-domain speech translation.
- Preparation: Cross-entropy supervision gives the best development BLEU among the tested attention losses, reaching 40.0.Multiplication reaches 39.6 and mean squared error reaches 39.4.
- Preparation: GIZA++ performs slightly better than fast align for supervising SA-NMT, so subsequent experiments use GIZA++.The comparison is based on development-set BLEU.
- Large-scale translation: SA-NMT reaches better early BLEU and more stable learning curves than NMT2, though it requires more updates to peak.NMT2 peaks at update 150000 and becomes unstable afterward.
- Large-scale translation: 2.2 BLEU points are the averaged test-set gain of SA-NMT over its direct NMT2 baseline on the large-scale task.SA-NMT also achieves significant and consistent improvements over Moses, NMT1, and NMT2.
- Low-resource translation: SA-NMT improves substantially over NMT2 in the low-resource task but does not surpass Moses, narrowing the gap toward that system.The paper presents this as a strong result despite the low-resource setting.
5 Related Work
Prior work improved attention-based NMT and explored reordering models, while conventional SMT commonly learned reordering from word- or phrase-level alignments.
- Recent attention enhancements address coverage errors, long-term dependencies, and alignment agreement in neural machine translation.
- Conventional statistical machine translation routinely learns reordering models from alignments at both phrase and word levels.
- Phrase-level work includes lexicalized MSD, feature-rich BTG, and neural BTG reordering models.
6 Conclusion
The paper explains attention’s weaker alignment accuracy through reordering and proposes supervised attention guided by conventional alignment models. On two Chinese-to-English tasks, the approach improves alignments and translation performance over standard attention-based NMT.
- The paper explains attention’s inferior alignment accuracy by viewing attention as a reordering mechanism.
- It proposes supervised attention for NMT using guidance from external conventional alignment models.
- Experiments on two Chinese-to-English tasks show better alignments and significant gains over standard attention-based NMT.