Source-linked AI summary
Modeling Coverage for Neural Machine Translation
Zhaopeng Tu, Zhengdong Lu, Yang Liu, Xiaohua Liu, Hang Li
TL;DR
Attention-based NMT does not track past alignment information, contributing to over-translation and under-translation. The paper adds a coverage vector that records attention history and adjusts future attention; experiments report significant improvements in translation and alignment quality over NMT without coverage.
Problem
Attention-based NMT ignores past alignment information, contributing to over-translation and under-translation.
Method
NMT-COVERAGE appends and updates a coverage vector during decoding, feeding attention history into the attention model to adjust future attention.
Results
NMT-COVERAGE significantly improves translation and alignment quality over NMT without coverage.
Takeaways & Limitations
Coverage models encourage attention to shift toward untranslated source words and alleviate over-translation and under-translation.
Takeaways & Limitations
Further increasing the coverage dimension only slightly improved translation performance in a pilot study.
Abstract
from arXiv · showhide
Attention mechanism has enhanced state-of-the-art Neural Machine Translation (NMT) by jointly learning to align and translate. It tends to ignore past alignment information, however, which often leads to over-translation and under-translation. To address this problem, we propose coverage-based NMT in this paper. We maintain a coverage vector to keep track of the attention history. The coverage vector is fed to the attention model to help adjust future attention, which lets NMT system to consider more about untranslated source words. Experiments show that the proposed approach significantly improves both translation quality and alignment quality over standard attention-based NMT.
1 Introduction
NMT lacks explicit coverage tracking, allowing over-translation and under-translation. The paper proposes NMT-COVERAGE, which tracks attention history to adjust future attention and improves translation and alignment quality.
- NMT lacks a coverage vector, so decoding can stop after the end-of-sentence mark without ensuring every source word is translated.Phrase-based SMT uses coverage to indicate whether source words have been translated.
- Ignoring source-word coverage can cause over-translation, where words are translated multiple times, and under-translation, where words remain untranslated.The paper attributes these problems to attention-based NMT ignoring how often source words have been translated.
- NMT-COVERAGE appends and updates a coverage vector during decoding to track attention history and adjust future attention.The mechanism is intended to help the model attend more appropriately to untranslated source words.
- The approach achieves significant improvement even when coverage is computed as the sum of previous alignment probabilities for each source word.This variant incorporates the simple coverage signal directly into the neural model.
- NMT-COVERAGE significantly outperforms conventional attention-based NMT on both translation and alignment tasks.The paper also illustrates reduced over-translation and under-translation in an example translation.
2 Background
Attention-based NMT jointly aligns and generates translations by selecting source information at each decoding step. However, its attention model does not use past alignment information, which contributes to over-translation and under-translation.
- Attention-based NMT generates one target word at each time step from the input sentence and previously generated target words.The model jointly performs dynamic alignment and target generation.
- The decoder forms a source representation as a weighted sum of source annotations, using attention weights for each decoding step.The attention model scores how well target and source representations match.
- Attention lets the decoder select relevant parts of the source sentence instead of representing the entire source sentence with one vector.This produces an expected source annotation over possible alignments.
- The attention model fails to exploit past alignment information, even though previously translated source words should receive lower alignment probabilities.This limitation is linked to over-translation and under-translation problems.
3 Coverage Model for NMT
The coverage model tracks source-side attention history and feeds it back into future attention, helping NMT reduce repeated or missing translations. The paper presents linguistically inspired and neural-network-based coverage variants, with different representations and update mechanisms.
- Motivation: Coverage tracks which source words have been translated, addressing gaps and overlaps that attention-based NMT can otherwise produce.SMT uses hard coverage, whereas NMT requires a softer mechanism because each source word can contribute partially across decoding steps.
- Coverage mechanism: The coverage vector summarizes each source word’s attention history and can discourage further attention to heavily attended words while redirecting attention toward less-attended segments.Attention weights are normalized to one, so reducing attention to attended regions implicitly increases attention elsewhere.
- Coverage mechanism: The general coverage model updates Ci,j from its previous value, the current alignment αi,j, the source annotation hj, and optional auxiliary inputs Ψ.The update function and auxiliary inputs can vary across coverage implementations.
- Linguistic coverage model: The linguistically inspired model uses soft coverage: accumulated alignment probabilities represent the translated ratio of each source word.It uses a scalar coverage value initialized at zero and can normalize alignment accumulation with a word-specific expected fertility weight Φj.
- Linguistic coverage model: Fertility predicts Φj, the number of target words a source word is expected to generate, while the NMT formulation treats fertility as a constant independent of previous fertilities.This accommodates source words that contribute to different numbers of target words.
- Neural-network coverage model: The neural coverage model represents coverage with a vector and uses a neural update function, incorporating past coverage, past translation information, and source-word annotations.The implementation can use nonlinear or gated activation; the paper adopts a GRU for the gating activation.
4 Training
NMT-COVERAGE is trained end to end, jointly learning the original NMT parameters and coverage-model parameters by maximizing reference-sentence likelihood. An optional fertility-based auxiliary objective slightly improves alignment but consistently worsens translation quality, so the main strategy lets translation likelihood drive training.
- End-to-end training: NMT-COVERAGE jointly learns the original NMT parameters and coverage-model parameters through end-to-end training.The parameters include θ for encoding, decoding, and attention, plus η for coverage annotation and attention guidance.
- End-to-end training: The primary training objective maximizes the likelihood of reference sentences, as in most NMT models.
- Auxiliary objective: An auxiliary fertility objective penalizes discrepancies between summed alignment probabilities and expected fertility.
- Auxiliary objective: The combined objective consistently worsens translation quality while slightly improving alignment quality.
- Training rationale: The training strategy places fewer constraints on the dependency between expected fertility and attention than more explicit alternatives.Because attention weights cannot be interpreted as the proportion of a word translated, penalizing their discrepancy with fertility is not well-founded in this setting.
5 Experiments
Experiments show that coverage-based NMT improves translation and alignment quality over attention-based NMT, while reducing translation errors and handling long sentences better.
- Translation Quality: NMT-COVERAGE significantly improves translation quality over GroundHog across the evaluated cases, with BLEU scores used as the metric.GroundHog is an attention-based NMT baseline, while Moses is the phrase-based SMT comparison system.
- Efficiency: The most complex coverage model adds only 0.1M parameters, while training speed decreases from 960 target words per second for GroundHog to 800 for an NN-based variant.The reported slowdown is described as not significant.
- Translation Quality: The simplest linguistic coverage improves translation performance by 1.1 BLEU points, and fertility further boosts performance by estimating covered source-word ratios.Fertility provides a more informative estimate of how much target-language material each source word generates.
- Translation Quality: NN-based coverage variants outperform GroundHog by averaged gains of 0.8 and 1.3 BLEU points, while increasing coverage dimension adds 0.6 BLEU at higher parameter cost.The larger coverage dimension increases parameters from 10K to 100K.
- Translation Quality: Coverage reduces under-translation and over-translation errors by 33.2% and 40.0%, respectively, improving translation adequacy and fluency by around 0.2 points.The gains address serious under-translation in long sentences containing several sub-sentences.
- Alignment Quality: Coverage improves alignment quality by reducing errors for linguistic coverage with fertility, whereas NN-based coverage needs dimension 10 before significantly reducing alignment errors.These results support using attention-history information to improve source-target alignment.
- Effects on Long Sentences: Coverage makes translated source words less likely to receive attention next and alleviates under-translation, especially in long sentences with multiple sub-sentences.On long inputs, coverage models produce longer translations and push attention toward untranslated source regions.
6 Related Work
The work builds on methods that improve attention-based NMT by importing ideas from SMT and modeling alignment regularities. Related fertility-based approaches differ in whether fertility is predicted before decoding or represented through surrounding alignment scores and prior context.
- 6 Related Work: MRT adapts SMT’s metric-oriented training idea to end-to-end NMT by directly optimizing evaluation metrics.Agreement-based learning was also proposed because attention-based NMT captures only partial attentional regularities.
- 6 Related Work: Other concurrent work also applied fertility or previous attentional context to improve attention-based NMT.The cited approaches are described as independent from this work and similar in spirit to its linguistically inspired coverage.
- 6 Related Work: Fertility-based approaches model alignment capacity, but differ in implementation from this work’s coverage construction.Cohn et al. use surrounding source-word alignment scores, whereas this work predicts fertility before decoding to normalize each source word’s coverage ratio.
7 Conclusion
The paper presents coverage-based NMT, which tracks whether source words have been translated and redirects attention toward untranslated words. Its linguistic and neural-network coverage variants significantly improve translation and alignment quality over NMT without coverage.
- 7 Conclusion: Coverage-based NMT maintains and uses a coverage vector to indicate whether each source word is translated.The approach enhances NMT by incorporating translation-history information into attention.
- 7 Conclusion: The coverage mechanism encourages attention to move away from translated words and toward untranslated words.This is intended to alleviate over-translation and under-translation in traditional attention-based NMT.
- 7 Conclusion: The paper proposes linguistic coverage and NN-based coverage as two variants of the approach.Linguistic coverage uses more linguistic information, while NN-based coverage relies on neural-network approximation flexibility.
- 7 Conclusion: Both coverage variants significantly improve translation quality and alignment quality over NMT without coverage.The conclusion reports improvements on both evaluation dimensions relative to the no-coverage baseline.