Source-linked AI summary
Dictionary-based Phrase-level Prompting of Large Language Models for Machine Translation
Marjan Ghazvininejad, Hila Gonen, Luke Zettlemoyer
TL;DR
Prompted LLMs can struggle with rare words, a challenge common in low-resource and out-of-domain translation. This paper introduces DIPMT, which adds bilingual-dictionary translation options for selected input words to prompts. It reports improvements in both settings, while noting that hints may be ignored or produce stylistic mismatches.
Problem
Prompted LLMs can struggle to translate rare words, which are common in low-resource and out-of-domain scenarios.
Source: §1 · p. 1Method
DIPMT adds possible bilingual-dictionary translations for selected input words to the translation prompt, without model training or hard constraints on using the hints.
Source: §1 · p. 1Results
Takeaways & Limitations
The method uses dictionary knowledge as prompt-level background information without requiring model training or enforcing how the model uses it.
Source: §1 · p. 1Generated translations can have a style mismatch with the reference, and the model sometimes ignores hints even when they match the reference translation.
Source: §5.3 · pp. 7–8
Abstract
from arXiv · showhide
Large language models (LLMs) demonstrate remarkable machine translation (MT) abilities via prompting, even though they were not explicitly trained for this task. However, even given the incredible quantities of data they are trained on, LLMs can struggle to translate inputs with rare words, which are common in low resource or domain transfer scenarios. We show that LLM prompting can provide an effective solution for rare words as well, by using prior knowledge from bilingual dictionaries to provide control hints in the prompts. We propose a novel method, DiPMT, that provides a set of possible translations for a subset of the input words, thereby enabling fine-grained phrase-level prompted control of the LLM. Extensive experiments show that DiPMT outperforms the baseline both in low-resource MT, as well as for out-of-domain MT. We further provide a qualitative analysis of the benefits and limitations of this approach, including the overall level of controllability that is achieved.
1 Introduction
The introduction frames rare-word translation as a challenge for prompted LLMs and proposes adding bilingual-dictionary hints for individual input words. It reports gains in low-resource and out-of-domain translation and describes analyses of the method’s benefits, limitations, and control over outputs.
1 Introduction
- Bilingual-dictionary hints give prompted LLMs possible translations for selected input words, without training the model or enforcing hard constraints.The method extends prompting from specifying the translation task to supplying options for individual word choices.Source: §1 · p. 1
- DIPMT achieves significant gains in low-resource translation from and into English across multiple languages and language models, with up to 13 BLEU points of improvement in out-of-domain translation.The paper also analyzes the method’s behavior, limitations, and how much its hints control the final translation.Source: §1 · pp. 1–2
- The paper proposes DIPMT for incorporating bilingual dictionary information into prompting-based machine translation and evaluates it for low-resource and out-of-domain translation.It also examines benefits and limitations and measures the extent to which hints control the final translation.Source: §1 · pp. 1–2
2 Machine Translation via Prompting with Dictionary
The section motivates dictionary hints as a way to address translation challenges for low-resource languages and out-of-domain text. DIPMT adds possible word translations to a prompted translation task, using demonstrations, while its dictionaries are word-level.
2 Machine Translation via Prompting with Dictionary
- Prompted MT may struggle when low-resource languages lack sufficient pretraining data or when input text is out of domain.Dictionaries are presented as a source of translation knowledge that can be incorporated into prompts.Source: §2 · p. 2
- DIPMT appends dictionary translations for words in the source sentence between the translation instruction and the prompt’s request for the full translation.The prompt has source-sentence, dictionary-hint, and target-translation parts; demonstrations include the completed translation.Source: §2 · p. 2
- The dictionaries used in DIPMT provide word-level translations; extending the method to phrase-level translations is left for future work.Source: §2 · p. 2
- The Figure 1 example shows the model completing the target-language translation, while the baseline prompt omits the dictionary-hint part.The example uses a 1-shot Indonesian-to-English prompt.Source: Figure 1 · p. 2
- DIPMT is described as effective in improving translation quality, with experimental results presented in Section 4.Source: §2 · p. 2
3 Experimental Setup
The experimental setup uses OPT and Bloom, evaluates low-resource translation with selected Flores-101 languages and out-of-domain German-to-English data, and compares prompts with and without dictionary hints. It uses ground-truth dictionaries in-domain and dictionaries extracted from training data out of domain.
3.1 Language Models
- OPT and Bloom are the two language models used to assess DIPMT, with Bloom serving as the multilingual model.OPT is described as predominantly English-trained; Bloom is trained on ROOTS, which includes 46 natural languages and 13 programming languages.Source: §3.1 · pp. 2–3
3.2 Datasets and Evaluation Metrics
- Low-resource evaluation uses selected Flores-101 languages, while out-of-domain evaluation uses German-to-English data from medical, law, IT, and Koran domains.The Flores-101 selection includes languages on which OPT performs moderately well in four-shot translation; out-of-domain evaluation uses sacreBLEU.Source: §3.2 · p. 3
3.3 Dictionaries
- In-domain experiments use ground-truth bilingual dictionaries, while out-of-domain dictionaries are built from training data using the method of Shi et al. (2021).The in-domain dictionaries were designed to handle word polysemy.Source: §3.3 · p. 3
3.4 Prompting Formulation
- Each test prompt uses four demonstrations with source text, word-level dictionary hints, and a target translation; the baseline omits the hint component.Hints are provided for exact dictionary matches, excluding the 500 most frequent source words; when a word has more than three translations, three are chosen randomly.Source: §3.4 · pp. 3–4
4 Improving Translation Performance
DIPMT improves prompting-based translation in low-resource and out-of-domain settings. In out-of-domain tests it substantially beats the baseline and RePP, while kNN-MT leads in most domains; low-resource gains vary by model and direction.
4.1 Low-resource MT
- OPT gains 0.9 BLEU points and BLOOM 1.1 on average across 10 languages and both translation directions.OPT improves more from English, while BLOOM improves more into English.Source: §4.1 · p. 4, Table 3 · p. 5
4.2 Out-of-domain MT
- For out-of-domain MT, the method assumes parallel data and builds a domain-specific dictionary using word alignment and lexicon induction.It retains multiple target words whose estimated probability meets λ = 0.1.Source: §4.2 · p. 4
- DIPMT beats the baseline by 9.4 BLEU points on average and RePP by 3.3, but kNN-MT leads outside Koran.The paper attributes kNN-MT’s advantage partly to corpus retrieval supporting style changes; it is also more expensive to run.Source: §4.2 · p. 5
5 Analysis
The analysis examines how dictionary coverage and hint quality relate to translation outcomes, then illustrates both helpful uses and failure modes of hints.
5.1 Type Coverage Exploration
- BLEU improves consistently as source word type coverage rises, with a threshold above 20% for Ind→Eng and 5–10% for other tested pairs.The experiment progressively removes random dictionary entries; type coverage is more representative than token coverage because a covered type covers all its instances.Source: Figure 2 · p. 6, §5.1 · p. 6
5.2 Gold Hints
- With only reference-matching gold hints, OPT improves BLEU by another 1.1 points on average over using all available dictionary translations.The oracle experiment gives one matching translation per source word when available.Source: §5.2 · pp. 6–7, Table 6 · p. 7
5.3 Output Analysis
- Hints can help produce a correct translation, and the authors describe such examples as common and a source of BLEU gains.One example uses hints for “serangga,” “melipat,” and “capung.”Source: §5.3 · p. 7, Figure 3 · p. 7
- When the desired translation is absent or the model selects another provided hint, the output may remain correct but differ in style from the reference.The two illustrated cases involve a missing desired option and selection of a different available hint.Source: §5.3 · p. 7, Figure 4 · p. 7, Figure 5 · p. 8
- In a relatively rare case, the model ignores hints that match the reference even though they could improve an incorrect baseline translation.The authors report that the model often uses hints, but not in this example.Source: §5.3 · p. 8
6 Controllability
The section measures whether suggested translations appear in outputs and compares ordinary prompting with hinting controls.
6 Controllability
- The evaluation compares hint-use rates against baseline, random single hints, shuffled false hints, and gold-only hints.The baseline comparison matters because a suggested translation may appear even without being supplied.Source: §6 · p. 8
- DIPMT yields a significant increase over baseline in the percentage of suggested hints used; gold-only hints achieve the highest controllability.A hint counts as used if any available translation appears in the output; multiple hits for one source token count once.Source: §6 · p. 8
- False hints produce almost no change relative to baseline, which the authors interpret as resilience to false word-level hints.Source: §6 · p. 8
7 Related Work
Related work covers prompt-based MT, dictionary methods for supervised MT, lexical constraints, and inference-time domain adaptation.
Prompting Language Models for MT
- Prompting studies test MT with simple prompts, vary templates, explore control of formality or dialect, or select in-context examples; this work uses dictionaries for translation.Source: §7 · pp. 8–9
Using Dictionary in MT
- Dictionary-based supervised MT methods use bilingual dictionaries or translation lexicons to improve rare- or low-frequency-word translation, while another approach builds synthetic parallel data.Source: §7 · p. 9
Domain Adaptation for MT
- Lexical constraints are categorized as hard or soft, while domain-adaptation work enhances pretrained NMT with out-of-domain data.The paper's method adapts at inference without additional training; related phrase-level retrieval work uses a bilingual database and retrieved cues.Source: §7 · p. 9
8 Conclusion
DiPMT improves low-resource MT and uses unsupervised dictionaries to improve out-of-domain translation. The paper also analyzes the method’s benefits and limitations and examines its controllability over final translations.
8 Conclusion
- DiPMT incorporates bilingual dictionary information into prompting-based MT by adding possible word-level translations to the prompt.Source: §8 · p. 9
- DiPMT improves translation quality for low-resource languages and out-of-domain translation using dictionaries created unsupervised.The paper also reports analyzing the method’s benefits and limitations.Source: §8 · pp. 9–10
- The paper analyzes DiPMT’s benefits and limitations and examines its controllability over final translations under different conditions.Source: §8 · pp. 9–10