Source-linked AI summary
Improving Lexical Choice in Neural Machine Translation
Toan Q. Nguyen, David Chiang
TL;DR
Rare-word mistranslation in NMT reflects an output layer that can disproportionately reward frequent words and produce fluent but inaccurate translations. The paper fixes output and context-vector norms and adds a jointly trained source-connected lexical module. Across eight language pairs, the approaches improve translation by up to +4.3 BLEU and surpass phrase-based translation in nearly all settings.
Problem
NMT can produce fluent but inaccurate translations of rare words because its output layer disproportionately favors frequent words.
Method
The paper fixes the norms of output-word and context vectors and adds a jointly trained lexical module connecting source words directly to target predictions.
Results
+4.3 BLEU is the largest reported improvement across eight language pairs, with the proposed systems surpassing phrase-based translation in nearly all settings.
Takeaways & Limitations
The authors conclude that these output-layer changes make NMT a more viable choice for low-resource translation.
Takeaways & Limitations
The normalization approach does not completely solve mistranslation, and the authors identify extending evaluation to Transformer and other models as future work.
Abstract
from arXiv · showhide
We explore two solutions to the problem of mistranslating rare words in neural machine translation. First, we argue that the standard output layer, which computes the inner product of a vector representing the context with all possible output word embeddings, rewards frequent words disproportionately, and we propose to fix the norms of both vectors to a constant value. Second, we integrate a simple lexical module which is jointly trained with the rest of the model. We evaluate our approaches on eight language pairs with data sizes ranging from 100k to 8M words, and achieve improvements of up to +4.3 BLEU, surpassing phrase-based translation in nearly all settings.
1 Introduction
NMT can produce fluent but inaccurate translations of rare words because its output layer disproportionately favors frequent words. The paper proposes fixed vector norms and a source-connected lexical module, reporting gains of up to +4.3 BLEU across eight language pairs.
- Motivation: Rare-word mistranslation remains an open NMT problem, with fluent outputs that can fail to reflect source content.An example replaces Anthony Fauci with James Chan, despite the correct surname appearing among the model’s top candidates.
- Problem: The output layer’s context–word inner product can favor common words disproportionately during prediction.The model computes a distribution using We · ˜h + be, where We and be depend on the output word and ˜h depends on source context and prior outputs.
- Approach: The paper fixes the norms of the context and output-word vectors and adds a jointly trained lexical module connecting source words directly to target predictions.The lexical connection is intended to help the model memorize translations of rare words.
- Results: +4.3 BLEU is the largest reported improvement across eight language pairs ranging from 100k to 8M training words.The resulting systems surpass phrase-based translation in nearly all settings.
2 Neural Machine Translation
The NMT system encodes source sentences, attends over encoder states, and generates target words sequentially. Its output weights can also serve as target-word embeddings, with tied embeddings performing as well as or better than the baseline in preliminary experiments.
- Objective: The model seeks a target sequence maximizing the conditional log-probabilities of its successive words given prior outputs and the source.The objective is expressed as the sum of log p(et | e<t, f).
- Preliminary experiments: Tying target embeddings with output-layer weights performs as well as or better than the baseline, while normalizing ˜h outperforms leaving it unnormalized in preliminary BLEU tests.The comparison uses development-set BLEU scores against tokenized references.
- Architecture: NMT encodes source words into hidden states, uses attention to form context, and decodes the target sentence word by word.At each step, attention weights produce a weighted average of encoder hidden states for the decoder.
- Output layer: The rows of the output weight matrix can be interpreted as embeddings of the output vocabulary.The paper ties target embeddings to the output weights in all models after observing comparable or better performance with tying.
3 Normalization
The normalization analysis attributes frequent-word bias partly to output-embedding and context-vector norms. Fixing these norms changes the output layer so directional compatibility is less overwhelmed by magnitude, and the Fauci example illustrates the intended effect.
- Output decomposition: The output score decomposes into context magnitude, cosine compatibility, output-embedding magnitude, and bias, each affecting word probabilities differently.Context magnitude sharpens or flattens the distribution, cosine measures contextual fit, and bias controls how much a word is generated.
- Frequency bias: Output-embedding norms generally correlate with word frequency and can disproportionately favor frequent words when multiplied by contextual similarity.The paper hypothesizes that this interaction causes disproportionately large norms for some words.
- Example: In the Fauci example, cosine similarity and bias favor Fauci, whereas the embedding norm favors the more frequent but incorrect Chan.This illustrates how the most frequently mentioned immunologist can outrank the correct surname.
- Normalization method: The paper fixes every target-word embedding norm to a constant r and also compares replacing ˜h with a normalized vector.The development comparison finds that normalizing ˜h performs better than leaving it unnormalized.
4 Lexical Translation
The lexical translation module addresses the risk that attentional context favors fluent target words without matching source words. It adds a jointly trained source-driven FFNN whose output is combined with the decoder prediction.
- Motivation: The attentional hidden state combines source-word and target-context information, which can produce contextually suitable but source-inaccurate translations.This contrasts with count-based models, which do not model these contexts.
- Lexical Translation: A source-driven FFNN is jointly trained with NMT to generate target words directly from source-word information.Attention weights form the source representation used by the lexical module.
- Lexical Translation: The lexical module uses a one-hidden-layer FFNN with skip connections and combines its output with the decoder output to predict target words.Its source representation is a weighted average of source-word embeddings rather than encoder hidden states.
- Lexical Translation: The lexical module’s parameters are normalized for the same reasons as the main model, but its rows are not tied to word embeddings.Preliminary experiments found that tying these rows produced worse results.
5 Experiments
The experiments evaluate normalization and lexical modeling across diverse language pairs, datasets, baselines, and training conditions. The tested systems include untied and tied NMT, fixnorm, and fixnorm+lex, with standardized preprocessing and inference procedures.
- Data and Tasks: The study evaluates normalization and lexical modeling across eight language-pair settings spanning LORELEI, IWSLT, KFTT, and BTEC data.The language directions include five low-resource languages into English, English–Vietnamese, and English–Japanese.
- Systems: The comparison includes untied and tied NMT baselines, Moses phrase-based translation, Arthur et al.’s discrete lexicon system, fixnorm, and fixnorm+lex.Fixnorm applies normalization, while fixnorm+lex adds the lexical translation module.
- Inference: Inference uses beam search with beam size 12 and replaces each target UNK with the source word receiving the highest attention score.The decoding score is modified to compensate for NMT’s tendency toward short translations.
- Evaluation: Evaluation reports case-sensitive BLEU, uses task-specific tokenization conventions, and tests statistical significance with bootstrap resampling.English-target tasks use detokenized references, whereas English–Japanese and English–Vietnamese follow prior tokenized BLEU settings.
6 Results and Analysis
Across language pairs, fixnorm and fixnorm+lex consistently improve over tied NMT, with fixnorm+lex outperforming Moses in all but Urdu-English and Hausa-English. Analyses link these gains to better rare-word translation, corrected alignments, and a sparse lexical distribution, while showing underfitting or overfitting when r is poorly chosen.
- 6.2 Impact on translation: Fixnorm and fixnorm+lex correct examples where baseline systems confuse numbers or proper names, including 34, Kenya, Afghan, Myanmar, and Fauci.Fixnorm alone can replace Entoni Fauchi with UNK UNK, whereas fixnorm+lex gets the example right.
- 6.3 Alignment and unknown words: Fixnorm alone does not fully resolve shifted alignments or unknown-word errors, but fixnorm+lex restores correct alignment and replaces Deutsche Telekom accurately.Tied and fixnorm shift attention one source word left, causing incorrect unknown-word replacement; fixnorm+lex corrects the alignment.
- 6.1 Overall: Fixnorm and fixnorm+lex consistently improve over tied NMT, while surpassing Moses on all tasks except Urdu-English and Hausa-English.The improvements over tied are significant except for English-Japanese (BTEC), and the models outperform Arthur et al. (2016) on all tasks.
- 6.4 Impact of r: The radius r trades off fitting and generalization: values that are too small indicate underfitting, whereas values that are too large indicate overfitting.Small r yields worse training perplexity; large r yields better training perplexity but decreased development BLEU.
- 6.5 Lexicon: The lexical module produces sparse distributions in which a few top translations account for most probability mass and can strongly favor Fauci over competing names.The lexical distribution is extracted as pℓ(y) = softmax(Wℓhℓ+bℓ), and cos θWle,hl significantly favors Fauci.
- 6.1 Overall: With BPE, the proposed methods still significantly improve translation for both high- and low-resource language pairs, trailing Moses only on Urdu-English.
7 Related Work
The paper distinguishes its approach to rare-word translation from prior methods and situates it among related output-layer and lexical-model work.
- 7 Related Work: The fixnorm approach shares the direction–magnitude output-layer formulation of Liu et al. (2016), but emphasizes magnitudes rather than directional separation.Liu et al. modify the loss to learn a margin-like classifier over class directions.
- 7 Related Work: Subword methods address both unknown and rare words, whereas the paper targets the root of the rare-word problem.Methods that enlarge vocabularies or copy source words primarily address unknown words; the authors expect their approach to benefit from these techniques as well.
- 7 Related Work: The authors identify Transformer compatibility as an open question because Liu and Kirchhoff (2018) evaluated BPE with a substantially different baseline architecture.They suggest testing whether their methods also benefit Transformer and other models.
- 7 Related Work: BPE-based experiments significantly improve over the baseline for both high- and low-resource settings.The reported improvement is statistically significant at p < 0.01.
8 Conclusion
The paper presents two output-layer changes that substantially improve NMT quality on low-resource language pairs. In many tested settings, the resulting systems surpass phrase-based translation when trained on the same data.
- 8 Conclusion: The two output-layer changes substantially improve translation quality on low-resource language pairs.The conclusion characterizes both changes as simple yet effective.
- 8 Conclusion: In many tested language pairs, the improved NMT system surpasses phrase-based translation when both systems use the same training data.The baseline NMT system often performs poorly relative to phrase-based translation in these settings.
- 8 Conclusion: The authors conclude that these methods make NMT a more viable choice for low-resource translation.They remain optimistic that NMT’s repertoire will continue to grow.