Source-linked AI summary
Incorporating Discrete Translation Lexicons into Neural Machine Translation
Philip Arthur, Graham Neubig, Satoshi Nakamura
TL;DR
NMT can mistranslate low-frequency content words despite fluent context, motivating discrete lexical information. The paper integrates probabilistic lexicons into NMT using attention-based lexical prediction and reports substantial BLEU and NIST gains, while performance remains below traditional SMT in KFTT NIST.
Problem
NMT can mistranslate low-frequency content words into natural but incorrect words that fail to reflect the source sentence.
Method
The method augments NMT with discrete probabilistic lexicons, using attention vectors to predict the next word and integrating lexicon probabilities by bias or linear interpolation.
Results
2.0-2.3 BLEU and 0.13-0.44 NIST score increases were achieved, with especially large gains on the more difficult KFTT test set.
Takeaways & Limitations
Discrete probabilistic lexicons improve NMT translation of low-frequency content words within the evaluated English-Japanese experiments.
Takeaways & Limitations
The NMT methods still fall behind traditional SMT in NIST score on the KFTT data set despite BLEU gains.
Abstract
from arXiv · showhide
Neural machine translation (NMT) often makes mistakes in translating low-frequency content words that are essential to understanding the meaning of the sentence. We propose a method to alleviate this problem by augmenting NMT systems with discrete translation lexicons that efficiently encode translations of these low-frequency words. We describe a method to calculate the lexicon probability of the next word in the translation candidate by using the attention vector of the NMT model to select which source word lexical probabilities the model should focus on. We test two methods to combine this probability with the standard NMT probability: (1) using it as a bias, and (2) linear interpolation. Experiments on two corpora show an improvement of 2.0-2.3 BLEU and 0.13-0.44 NIST score, and faster convergence time.
1 Introduction
NMT can mistranslate low-frequency content words despite strong end-to-end performance, motivating discrete lexical information as an additional source for translation. The paper proposes attention-guided lexicon integration and evaluates it on two English–Japanese corpora.
- Motivation: NMT’s continuous word representations can produce natural-sounding translations that misrepresent source content, such as translating “Tunisia” as “Norway.”These errors are especially serious because content words strongly influence sentence meaning.
- Motivation: Traditional SMT rarely makes this error because discrete phrase mappings preserve target words observed with source words in training data.Such mappings can be learned efficiently from very little data, sometimes a single aligned instance.
- Approach: The paper proposes adding discrete probabilistic lexicons to NMT, using attention to convert lexical probabilities into next-word predictions.Lexicon probabilities are obtained from word alignments, external parallel resources, handmade dictionaries, or hybrids.
- Approach: Lexicon probabilities are combined with standard NMT probabilities either as a bias or through linear interpolation.Both integration strategies are evaluated experimentally.
- Evaluation: Experiments on two English–Japanese translation corpora evaluate translation accuracy and training time.The evaluation is designed to test both improved accuracy and reduced training time.
2 Neural Machine Translation
The attentional NMT system encodes each source sentence with a bidirectional encoder and generates target words sequentially with a decoder that focuses on source positions. Its next-word distribution is conditioned on the encoded source and preceding target words.
- Prediction: The model calculates a conditional probability for each next target word from the source sentence and preceding target words.Training minimizes the negative log-likelihood of the training data.
- Encoder: A bidirectional encoder maps each source word into continuous forward and backward representations, concatenated into a matrix R.Each column of R corresponds to one source word.
- Decoder: The decoder generates one target word at a time while tracking prior target words with an LSTM hidden state.The selected word is fed back into the decoder to update its state.
- Attention: Attention scores compare the decoder state with source representations and are normalized into weights over source words.These weights determine how much focus each source word receives at the current step.
- Attention: The attention weights combine source representations into a context vector used with the decoder state to form the current output representation.The resulting representation is passed through an affine transform before prediction.
3 Integrating Lexicons into NMT
The proposed method augments attentional NMT with discrete lexical probabilities derived for each source word and focused through the model’s attention. It then integrates the resulting next-word lexicon distribution as either a bias or an interpolation with NMT probabilities.
- Lexicon probabilities: The method assumes a lexicon assigning each target word a probability conditioned on a source word, with nonzero mass concentrated on a small set of translation candidates.This distribution is converted into a form compatible with the NMT next-word probability.
- Lexicon probabilities: Lexical probabilities are arranged in a matrix whose columns represent source words and rows represent target-vocabulary words.The matrix can be precomputed during encoding because it depends only on the source sentence.
- Attention-guided prediction: The model uses attention weights to combine source-word lexical distributions into a predictive probability for the next target word.Different source words influence the lexicon prediction at different decoding steps.
- Integration methods: The bias method adds the adjusted log lexicon probability to the vanilla NMT softmax input.A small constant prevents zero probabilities from producing −∞; the paper uses ǫ = 0.001.
- Integration methods: The linear method interpolates the standard NMT and lexicon probabilities using a coefficient constrained between 0 and 1 by a sigmoid.The interpolation parameter is learnable and starts at x = 0, giving λ = 0.5.
4 Constructing Lexicon Probabilities
This section defines lexical probabilities from automatically learned, handmade, or hybrid lexicons, including fallback handling for uncovered words and limited target vocabularies.
- Lexicon sources: Lexical probabilities can be constructed from automatically learned lexicons, handmade lexicons, or a combination of both.The hybrid approach uses automatically learned probabilities by default and handmade probabilities for uncovered source words.
- Automatically learned lexicons: IBM-model lexicons estimate source-to-target lexical probabilities from parallel data using expectation maximization.The expectation step estimates expected counts, while the maximization step divides expected counts by all possible counts.
- Automatically learned lexicons: Traditional SMT generally translates low-frequency words more accurately than NMT, although IBM models can still suffer from rare-word problems.The passage cites garbage-collecting effects as an example of an IBM-model failure mode.
- Vocabulary handling: When rare target words fall outside the NMT vocabulary, the remaining lexicon probability mass is assigned to ⟨unk⟩.This accommodates vocabulary limits used for training speed or memory constraints.
- Handmade lexicons: For handmade dictionaries, the translation set K_f receives a uniform distribution because such dictionaries generally lack translation probabilities.Each listed translation is treated equally when constructing p_l(e|f).
- Hybrid lexicons: Handmade lexicons provide broad coverage, but their probabilities may be less accurate than learned probabilities, especially when the latter use in-domain data.The hybrid method addresses this by falling back to handmade probabilities only for words uncovered by the automatic lexicon.
5 Experiment & Result
Experiments on English–Japanese corpora show that discrete lexicon integration improves NMT accuracy, rare-word recovery, and convergence speed, while performance depends on lexicon quality and integration method.
- 5.1 Settings: Experiments use the English–Japanese KFTT and BTEC corpora, with KFTT presenting broader domains, more rare words, and longer sentences.KFTT and BTEC differ in domain breadth, vocabulary rarity, and sentence length.
- 5.2 Effect of Integrating Lexicons: Up to 2.3 BLEU, 0.44 NIST, and 30% Recall gains over baseline attn occur on the more difficult KFTT set.The gains are reported for the proposed bias method compared with baseline attn.
- 5.2 Effect of Integrating Lexicons: The proposed method can exceed traditional SMT systems in BLEU on KFTT without ensembling, although NMT still trails SMT in NIST there.The NIST gap indicates a remaining SMT advantage in translating lower-frequency words.
- 5.2 Effect of Integrating Lexicons: Bias-based lexicon integration reaches BLEU scores in the upper 10s after the first iteration, whereas baseline attn scores around 5 and converges more slowly.The authors interpret this as lexical probabilities bootstrapping NMT learning.
- 5.2 Effect of Integrating Lexicons: Each proposed-method iteration takes longer than baseline attn because it calculates and uses a lexical probability matrix for every sentence.Reported iteration times are 167 minutes for attn and 275 minutes for auto-bias; GIZA++ lexicon training adds 297 minutes.
- 5.3 Comparison of Integration Methods: The bias method generally outperforms linear interpolation, while linear interpolation helps manual lexicons but can reduce performance with automatic and hybrid lexicons.The authors suggest fixed interpolation coefficients may explain the lower linear-method performance.
6 Additional Experiments
Follow-up experiments on the larger ASPEC Japanese-English dataset tested whether the proposed method scales beyond smaller corpora. The auto-bias method improved BLEU over the attentional baseline.
- 1.84 BLEU points separated the attn baseline from auto-bias on ASPEC, increasing the score from 20.82 to 22.66.The ASPEC dataset contains 2 million training examples, 63 million tokens, and an 81,000-word vocabulary.
- The ASPEC experiment showed that the proposed method scales to larger datasets.
7 Related Work
Related work addresses low-frequency translation through unknown-word replacement, copying, vocabulary restriction, and character-based models. The paper distinguishes its lexicon approach by using translation probabilities and attention, while identifying context-dependent interpolation as future work.
- Unknown-word replacement methods use lexicons to substitute target-vocabulary words but do not incorporate lexicon information during learning.
- Copying models resemble linear interpolation but apply only when source words can be copied unchanged into the target language.
- Using context-dependent interpolation coefficients is identified as a promising direction beyond the paper’s fixed coefficient.
- The paper’s method improves on alignment-based vocabulary restriction by using actual translation probabilities and attention to combine them.
- Character-based translation models improve low-frequency-word accuracy, while prior work suggests word-level information can provide additional gains.
8 Conclusion & Future Work
The paper introduces discrete probabilistic lexicons for improving NMT translation of low-frequency words and reports gains in BLEU, NIST, and content-word translation. Future work targets larger tasks, subjective evaluation, and context-dependent interpolation.
- 8 Conclusion: BLEU increased by 2.0-2.3 and NIST by 0.13-0.44 after incorporating discrete probabilistic lexicons into NMT.
- 8 Conclusion: The method also produced qualitative improvements in content-word translations.
- Future Work: Future experiments will examine larger-scale translation tasks and include subjective evaluation of translation quality.
- Future Work: Future work will replace the fixed linear-interpolation coefficient λ with a value calculated from context.