Source-linked AI summary
Guiding Neural Machine Translation with Retrieved Translation Pieces
Jingyi Zhang, Masao Utiyama, Eiichro Sumita, Graham Neubig, Satoshi Nakamura
TL;DR
NMT has difficulty recalling and translating low-frequency words or phrases. The paper retrieves similar sentence pairs and uses weighted translation pieces to guide an existing NMT model during decoding. It reports gains of up to 6 BLEU points on three narrow-domain tasks, with little added translation time.
Problem
NMT has difficulty recalling and appropriately translating low-frequency words or phrases.
Source: §1 · p. 1Method
The method retrieves similar sentence pairs, weights target n-grams aligned with overlapping source words by sentence similarity, and rewards outputs containing those translation pieces.
Source: §1 · p. 2, §3.1 · p. 3Results
Up to 6 BLEU points of improvement on three narrow-domain translation tasks came with little translation-time increase and accuracy comparable to Gu et al. (2017), while being simpler to implement and faster at test time.
Source: §1 · p. 2, §5 · p. 9Takeaways & Limitations
The method is reported to be useful for narrow domains where similar sentences can be retrieved, with larger BLEU improvements on higher-similarity test data.
Source: §4.2 · p. 6, §5 · p. 9Improvements were not significant on WMT 2017 English-to-German, where lower train-test similarity likely left few useful translation pieces to retrieve.
Source: §4.2 · p. 6
Abstract
from arXiv · showhide
One of the difficulties of neural machine translation (NMT) is the recall and appropriate translation of low-frequency words or phrases. In this paper, we propose a simple, fast, and effective method for recalling previously seen translation examples and incorporating them into the NMT decoding process. Specifically, for an input sentence, we use a search engine to retrieve sentence pairs whose source sides are similar with the input sentence, and then collect $n$-grams that are both in the retrieved target sentences and aligned with words that match in the source sentences, which we call "translation pieces". We compute pseudo-probabilities for each retrieved sentence based on similarities between the input sentence and the retrieved source sentences, and use these to weight the retrieved translation pieces. Finally, an existing NMT model is used to translate the input sentence, with an additional bonus given to outputs that contain the collected translation pieces. We show our method improves NMT translation results up to 6 BLEU points on three narrow domain translation tasks where repetitiveness of the target sentences is particularly salient. It also causes little increase in the translation time, and compares favorably to another alternative retrieval-based method with respect to accuracy, speed, and simplicity of implementation.
1 Introduction
NMT struggles with low-frequency words and domain generalization, while prior retrieval-based approaches can add computational and implementation complexity. This paper guides an existing NMT model with similarity-weighted translation pieces and reports gains of up to 6 BLEU points on three tasks.
1 Introduction
- NMT is described as state of the art, but it has difficulty with low-frequency words or phrases and with generalizing across domains.Prior work addressed these difficulties with symbolic knowledge, model-structure changes, or domain and topic information.Source: §1 · p. 1
- Retrieval-based phrase and hierarchical phrase-based translation methods were aimed at efficiency rather than improved translation quality.The paper distinguishes these methods from retrieval-based approaches that bias NMT.Source: §1 · p. 1
- Earlier retrieval-based NMT methods fine-tuned model parameters or added retrieved sentences as inputs, bringing added training, computational, or memory costs.The paper positions its method as a simpler alternative to these approaches.Source: §1 · pp. 1–2
- The method retrieves similar sentence pairs, collects target n-grams aligned with source-word overlap, weights them by sentence similarity, and rewards NMT outputs containing those pieces.It guides an existing NMT model without changing its structure or updating its parameters.Source: §1 · p. 2
- Our method improved NMT translation results by up to 6 BLEU points on three translation tasks, with little increase in translation time; accuracy was comparable to Gu et al. (2017), despite simpler implementation and faster test-time performance.Source: §1 · p. 2
2 Attentional NMT
The attentional NMT baseline uses a bidirectional recurrent network to encode the source and generates target words left to right. Attention scores relate source positions to target steps, and beam search selects length-normalized hypotheses.
2 Attentional NMT
- The attentional NMT model comprises an encoder, decoder, and attention model; its encoder uses a bidirectional recurrent neural network for source words.Source: §2 · p. 2
- The decoder generates target words left to right, while beam search retains n-best hypotheses at each time step.Source: §2 · p. 2
- At target step t, attention score α_t,i measures how well the input around source position i matches the output, and the model computes a source representation c_t.Source: §2 · p. 2
- The complete-hypothesis translation score is normalized by sentence length to avoid outputs that are too short.Source: §2 · p. 3
3 Guiding NMT with Translation Pieces
The approach has two stages: retrieve target-side n-grams linked to source words overlapping the input, then use those translation pieces to guide an existing NMT model during decoding.
3 Guiding NMT with Translation Pieces
- The method prepares a word-aligned parallel corpus for retrieval and uses Lucene to find source sentences similar to the input.The paper uses the NMT training corpus for retrieval in this work, though it notes that a separate corpus could be used.Source: §3.1 · p. 3, §3 · p. 3
3.1 Retrieving Translation Pieces
- For each retrieved sentence, edit distance identifies unchanged source words, whose aligned target words yield candidate n-grams up to four words long, including phrases aligned to discontiguous source words.Table 1 gives examples of n-grams and whether they are included as translation pieces.Source: §3.1 · p. 3, Table 1 · p. 3
- Each candidate piece receives a score based on the similarity between the input and retrieved source sentence; pieces from less similar sentences are less likely to be correct.Sentence similarity is computed using normalized edit distance.Source: §3.1 · p. 3
3.2 Guiding NMT with Retrieved Translation Pieces
- During decoding, the method rewards outputs matching collected pieces by updating output probabilities and checking only words in the piece list; it adds little computational and memory cost compared with per-input fine-tuning or retrieved-sentence encoding.Figure 2 demonstrates adding rewards for matched pieces in the NMT output layer.Source: §3.2 · p. 4, Figure 2 · p. 4
4 Experiment
Experiments evaluate retrieved-piece guidance across translation settings and retrieval conditions. The method improves baseline BLEU, with results depending on training–test similarity and retrieval size, and generally compares favorably with SGNMT.
4.1 Settings
- Experiments used JRC-Acquis for en-de, en-fr, and en-es, with an attentional NMT baseline and 100 retrieved sentence pairs as the preliminary setting.The translation-piece weights were tuned on the development set for each language pair.Source: §4.1 · pp. 4–5
4.2 Results
- Up to 6 BLEU points: the method outperformed the baseline NMT system in the main translation results.The authors also report that translation length did not change significantly.Source: Table 2 · p. 5, §4.2 · p. 5, Table 5 · p. 5
- Gains were larger for half-H than half-L; WMT 2017 en-de showed no significant improvement, likely reflecting lower training–test similarity.The authors conclude the method can be useful in narrow domains where similar sentences can be found.Source: §4.2 · p. 6
- Similarity-based translation-piece scores mattered: replacing them with uniform 1/0 rewards significantly reduced performance, though it remained above the NMT baseline.The examples show higher-scored pieces coming from retrieved sentence pairs more similar to the input.Source: §4.2 · p. 6
4.3 Infrequent n-grams
- The method helped significantly with training-seen n-grams occurring fewer than 5 times; gains declined as frequency rose, and it helped little with unseen n-grams.The authors relate this pattern to NMT favoring frequent n-grams while their piece scores use sentence similarity.Source: §4.3 · pp. 7–8
4.4 Computational Considerations
- Retrieval time was negligible and decoding-time increases were small, but piece collection took considerable time despite a Python implementation that could be faster.Because collection occurs only when translating new sentences, the method does not increase NMT training time.Source: §4.4 · p. 8
- Reducing retrieval size sharply reduced piece-collection time, lowered BLEU less sharply, and further reduced decoding time; 10 was a good setting.At that setting, the authors report significant BLEU improvements and little increase in total translation time compared with the baseline.Source: Figure 6 · p. 8, Figure 4 · p. 8, Figure 5 · p. 8
4.5 Comparison with SGNMT
- Our method generally outperformed SGNMT on three translation tasks; SGNMT also adds NMT runs for retrieved pairs, increasing training- and test-time costs.The authors describe their approach as simpler to implement and faster at test time.Source: Table 12 · p. 9, §4.5 · p. 9
5 Conclusion
The paper guides an existing NMT model for narrow domains by retrieving and weighting translation pieces from similar sentence pairs. It reports gains of up to 6 BLEU points on three narrow-domain tasks, with little added translation time and favorable comparison to another retrieval-based method.
5 Conclusion
- The method retrieves similar source-side sentence pairs, weights aligned translation pieces, and rewards outputs containing those pieces during NMT decoding.Source: §5 · p. 9
- Up to 6 BLEU points were gained on three narrow-domain translation tasks.Source: §5 · p. 9
- The method caused little increase in translation time and compared favorably with another retrieval-based method on accuracy, speed, and implementation simplicity.Source: §5 · p. 9