Source-linked AI summary
Adaptive Machine Translation with Large Language Models
Yasmin Moslem, Rejwanul Haque, John D. Kelleher, Andy Way
TL;DR
Real-time adaptation remains challenging where translation must follow domain-specific terminology and style. This paper uses LLM in-context learning with translation pairs and fuzzy matches for adaptive MT tasks. Results are promising, especially for high-resource languages, while language support varies.
Problem
Real-time adaptation remains challenging, although domain-specific translation may require adherence to approved terminology and corrected translations.
Source: §1 · p. 1Method
Results
Results are promising across tasks, with strong outcomes for high-resource languages and weaker support for English-to-Kinyarwanda and English-to-Arabic, while Arabic post-editing surpassed OPUS and Google.
Source: §10 · pp. 8–9Takeaways & Limitations
Production can use different translation pipelines for different language pairs according to LLM support.
Source: §10 · p. 9The comparison with open-source LLMs such as BLOOM and BLOOMZ was brief, and broader experiments remain future work.
Source: §10 · p. 9
Abstract
from arXiv · showhide
Consistency is a key requirement of high-quality translation. It is especially important to adhere to pre-approved terminology and adapt to corrected translations in domain-specific projects. Machine translation (MT) has achieved significant progress in the area of domain adaptation. However, real-time adaptation remains challenging. Large-scale language models (LLMs) have recently shown interesting capabilities of in-context learning, where they learn to replicate certain input-output text generation patterns, without further fine-tuning. By feeding an LLM at inference time with a prompt that consists of a list of translation pairs, it can then simulate the domain and style characteristics. This work aims to investigate how we can utilize in-context learning to improve real-time adaptive MT. Our extensive experiments show promising results at translation time. For example, LLMs can adapt to a set of in-domain sentence pairs and/or terminology while translating a new sentence. We observe that the translation quality with few-shot in-context learning can surpass that of strong encoder-decoder MT systems, especially for high-resource languages. Moreover, we investigate whether we can combine MT from strong encoder-decoder models with fuzzy matches, which can further improve translation quality, especially for less supported languages. We conduct our experiments across five diverse language pairs, namely English-to-Arabic (EN-AR), English-to-Chinese (EN-ZH), English-to-French (EN-FR), English-to-Kinyarwanda (EN-RW), and English-to-Spanish (EN-ES).
1 Introduction
Real-time adaptive MT must incorporate user corrections, terminology, and style guidance, particularly in domain-specific settings. This paper investigates using LLM in-context learning to adapt translations without further training.
1 Introduction
- Adaptive MT uses user feedback to improve translations over time, addressing domain-specific cases where baseline systems may lack relevant data.Feedback can include corrections, terminology and style guides, and quality ratings.Source: §1 · p. 1
- In-context learning lets pretrained LLMs replicate input-output patterns from examples at inference time without fine-tuning.Prior work explored few-shot and zero-shot MT (Vilar et al., 2022; Wang et al., 2021), synthetic domain data (Moslem et al., 2022), and example selection (Agrawal et al., 2022; Zhang et al., 2023).Source: §1 · p. 2
- The paper investigates LLMs for real-time adaptive MT, including matching approved-pair terminology and style, comparing translation quality, fixing MT outputs, and constraining terminology.The listed tasks are intended to be performed without further training.Source: §1 · p. 2
2 Experimental Setup
The experiments use TICO-19 to simulate document-level translation with approved translation memories, across five English-to-target language pairs.
2 Experimental Setup
- All experiments use GPT-3.5 text-davinci-003 via its official API, with top-p 1 and temperature 0.3 for translation tasks and 0 for terminology extraction.Requests are batched in groups of 20 segments.Source: §2 · p. 2
- Maximum token-length multipliers are set to 8 for Arabic, 5 for Chinese and Kinyarwanda, and 4 for French and Spanish.The choices account for target-language token lengths relative to English source words.Source: §2 · p. 2
- The domain-specific TICO-19 context dataset contains 3,070 unique segments and is used to simulate document-level translation with a project or client translation memory.The language pairs are English-to-Arabic, Chinese, French, Kinyarwanda, and Spanish.Source: §2 · p. 2
3 Adaptive MT with Fuzzy Matches
GPT-3.5 few-shot translation benefits from context drawn from similar approved translations, with more fuzzy matches improving quality but yielding smaller incremental gains.
3 Adaptive MT with Fuzzy Matches
- Embedding-based retrieval selects up to ten similar source sentences from the context dataset using Sentence-Transformers paraphrase mining and all-MiniLM-L6-v2.The experiments vary the number of retrieved sentences from one to ten.Source: §3 · pp. 2–3
- Zero-shot prompts provide the source sentence and language names, while few-shot prompts add translation examples to influence output style.The illustrated prompts show examples as source-target translation pairs.Source: §3 · p. 3
- Across five language pairs, GPT-3.5 with fuzzy-match context beats random examples, and adding matches improves translation quality with smaller incremental gains.The comparison covers 2, 5, and 10 matches; Table 1 reports consistent results for EN-AR, EN-ES, EN-FR, EN-RW, and EN-ZH.Source: §3 · p. 3, Table 1 · p. 3
- Table 2 reports English-source similarity counts and percentages for the 2-shot and 5-shot fuzzy-match experiments across five language pairs.Similarity is calculated using the English source.Source: Table 2 · p. 3, §3 · p. 3
4 GPT-3 vs Encoder-Decoder MT Models
GPT-3.5 few-shot translation with fuzzy matches surpasses strong encoder-decoder systems for several high-resource language pairs, though results vary across languages and comparison settings.
4 GPT-3 vs Encoder-Decoder MT Models
- The comparison includes DeepL, Google Cloud Translation, OPUS, and NLLB-200; DeepL does not support Arabic or Kinyarwanda.OPUS and NLLB models were converted to CTranslate2 with int8 quantization for efficiency.Source: §4 · p. 3
- GPT-3.5 with five fuzzy matches outperforms strong encoder-decoder systems for English-to-French and English-to-Spanish, while Chinese requires ten matches.The results for English-to-Arabic and English-to-Kinyarwanda are not on par with the other three pairs.Source: Figure 2 · p. 4, §4 · p. 4
- ModernMT with a translation memory improves over its own no-memory results, but GPT-3.5 with five or ten fuzzy matches outperforms it for Chinese, French, and Spanish.ModernMT zero-shot translation outperforms GPT-3.5 for the four language pairs it supports.Source: §4 · p. 4
5 Incorporating Encoder-Decoder MT
This section tests two ways to combine encoder-decoder MT with GPT-3.5 few-shot translation: adding MT for the new segment, or for both the new segment and fuzzy-match examples. The gains vary with the language pair and the quality of the existing translations.
5.1 Fuzzy matches + new segment MT
- Adding encoder-decoder MT output to fuzzy matches improved performance over the standalone baselines for English-to-Arabic and English-to-Chinese, with similar behavior observed for English-to-Kinyarwanda.The reported examples use OPUS for Arabic and Kinyarwanda, and Google for Chinese.Source: §5.1 · p. 5
- For English-to-French and English-to-Spanish, adding encoder-decoder MT did not enhance GPT-3.5 when its fuzzy-match translation already outperformed the encoder-decoder baseline.The authors note that further gains may not be achieved when fuzzy-match translation is already substantially better.Source: §5.1 · p. 5
5.2 Fuzzy matches + all segments MT
- The second scenario adds MT for every fuzzy-match example as well as for the new source segment, building on the new-segment approach that enhanced GPT-3.5 in-context learning.The authors say it remains unclear whether translating all examples outperforms translating only the new segment for English-to-Kinyarwanda and English-to-Spanish; the result depends on original MT quality and needs further investigation.Source: §5.2 · p. 5
6 Bilingual Terminology Extraction
The section uses GPT-3.5 to extract bilingual terms from sentence pairs, then evaluates extraction quality for a sample of language pairs. Most extracted terms were accurate, though some were invented or partial.
6 Bilingual Terminology Extraction
- GPT-3.5 was prompted to extract five bilingual terms from each sentence pair in the context dataset.The extraction used temperature 0 and top p 1.Source: §6 · p. 5
- More than 95% of GPT-3-extracted terms in the human-evaluated sample were accurate.The evaluation covered English-to-Arabic, English-to-Spanish, and English-to-French.Source: Table 4 · p. 5
- Evaluators assessed 500 randomly sampled sentence pairs for Arabic, French, and Spanish, judging term equivalence and whether terms appeared in the pair on a 0–1 scale.Relevant inflections were accepted.Source: §6 · pp. 5–6
- The original English-to-French TICO-19 data included misaligned pairs, so the evaluation sample was manually refined to retain well-aligned pairs.The authors note that automatic semantic filtering could be applied to large datasets.Source: §6 · p. 5
- Some extracted pairs contained terms made up by the model or partial terms, sometimes because the maximum token length was reached.The authors note that invented terms could nevertheless be correct but absent from the provided pair.Source: §6 · p. 6
7 Terminology-Constrained MT
This section tests adding terms from fuzzy matches or a glossary to few-shot and zero-shot translation. Glossary terms improved quality across the five language pairs in zero-shot tests, while human evaluation found more term transfer but also possible grammar costs.
7 Terminology-Constrained MT
- The tested approaches use terms matching 1–5-gram phrases in the source; glossary construction selects frequent term pairs and prioritizes longer n-grams.The glossary includes terms occurring at least twice, selects the most frequent target when a source has multiple targets, and excludes stop words and empty sides.Source: §7 · p. 6
- In zero-shot translation, adding glossary terms significantly improved translation quality across all five language pairs.The experiments also found that, with two fuzzy matches, using glossary terms outperformed using terms from those matches alone.Source: §7 · p. 6
- For Arabic, French, and Spanish, terminology-constrained MT transferred provided glossary terms more often than translation without terminology incorporation.Forcing glossary terms sometimes improved overall quality but could be detrimental to grammatical accuracy.Source: §7 · p. 6
- Few-shot translation with fuzzy matches used more terms successfully than zero-shot translation, and glossary terms in a zero-shot prompt could yield quality gains comparable to fuzzy-match few-shot translation.The authors connect successful term use in few-shot translation with consistency with approved translations.Source: §7 · p. 7
8 ChatGPT
The section compares ChatGPT models with GPT-3.5 Davinci for zero-shot and fuzzy-match few-shot translation. GPT-4 improves zero-shot quality, while few-shot quality is relatively similar across the models.
8 ChatGPT
- GPT-3.5 Turbo is more efficient than text-davinci-003 while showing comparable quality for zero-shot and fuzzy-match few-shot translation.Source: §8 · p. 7
- Providing both “New York Times” and “New York” can confuse the model when only the former should be transferred without translation.Source: §8 · p. 7
- GPT-4 provides better zero-shot translation quality, while its few-shot quality is relatively similar to the other models.Table 8 compares GPT-3.5 text-davinci-003, gpt-3.5-turbo, and gpt-4 using two fuzzy matches for few-shot translation.Source: §8 · p. 7, Table 8 · p. 7
9 BLOOM and BLOOMZ
The section compares GPT-3.5 with the open-source models BLOOM and BLOOMZ, including translation with fuzzy matches. GPT-3.5 generally outperforms both on most language pairs, with exceptions for English-to-Arabic and comparable English-to-French quality.
9 BLOOM and BLOOMZ
- BLOOM is a general-purpose LLM, whereas BLOOMZ is designed to follow human instructions in a zero-shot manner.Source: §9 · pp. 7–8
- Terminology-constrained MT with GPT-3.5 outperformed zero-shot and two-shot fuzzy-match translation, with larger gains for zero-shot; glossary terms worked better than fuzzy-match-derived terms in the two-shot comparison.Source: Table 6 · p. 8
- For BLOOM, greedy search performed better for translation than the GPT-3.5 sampling settings, and the same settings were used for BLOOMZ comparisons.The experiments used batch size 1 and capped BLOOM output at 250 tokens.Source: §9 · p. 8
- With two fuzzy matches, GPT-3.5 generally outperforms BLOOM and BLOOMZ on most language pairs, except English-to-Arabic.English-to-French quality is comparable between BLOOM and GPT-3.5; Table 9 compares the models for two-match few-shot translation.Source: §9 · p. 8, Table 9 · p. 8
10 Conclusion
The conclusion reports promising GPT-3.5 results across translation tasks, with stronger support for several high-resource language pairs and weaker support for Kinyarwanda and Arabic. It highlights post-editing gains for Arabic and outlines limits and future directions.
10 Conclusion
- GPT-3.5 experiments covered fuzzy-match adaptive MT, MT post-editing, terminology extraction, and terminology-constrained MT, with promising overall results.Source: §10 · p. 8
- English-to-French, English-to-Spanish, and English-to-Chinese show excellent results, while English-to-Kinyarwanda has low-resource constraints and English-to-Arabic faces tokenizer issues.Source: §10 · pp. 8–9
- GPT-3.5 post-editing of OPUS English-to-Arabic translations significantly surpassed both OPUS and Google Translation API quality.Source: §10 · p. 9
- The authors briefly compared GPT-3.5 with open-source LLMs BLOOM and BLOOMZ and propose expanding those experiments to cover more aspects.Source: §10 · p. 9
- Future work could select only high-quality fuzzy matches, glossary terms, or other-system MT outputs, and test longer phrases instead of terms for terminology extraction.For fuzzy-match adaptive MT, the authors propose selecting matches above a similarity threshold rather than a fixed number for every sentence.Source: §10 · p. 9
- The study focuses on out-of-the-box LLM quality across tasks and language pairs; future experiments may fine-tune models first, especially for low-resource languages and rare domains.Source: §10 · p. 9
A Prompts
The appendix illustrates prompts for zero-shot and few-shot translation, post-editing, terminology extraction, and terminology-constrained MT. The examples show how source text, translation examples, MT output, and term pairs are arranged as context for a new translation or extraction request.
A.1 Zero-shot Translation
- The EN-AR zero-shot prompt supplies a source segment and target-language label without translation examples.Source: §A.1 · p. 11
A.2 Adaptive MT with Fuzzy Matches
- The EN-AR two-shot prompt places two source-target fuzzy-match pairs before the new source segment.Source: §A.2 · p. 11
A.3 MT Post-editing
- The EN-ZH post-editing examples provide two fuzzy matches and an MT output for the new segment before prompting for Chinese.The appendix shows variants with the new segment's MT output only and with MT outputs for the fuzzy matches as well.Source: §A.3 · p. 11
A.4 Terminology Extraction
- The terminology-extraction prompt pairs a source sentence with its target sentence, then requests a specified number of equivalent terms in a defined line format.Source: §A.4 · p. 11
A.5 Terminology-constrained MT
- The EN-ES zero-shot glossary prompt supplies term pairs before the source segment and requests a Spanish translation.Source: §A.5 · p. 11
- The EN-ES two-shot fuzzy-terms prompt provides each match with its extracted terms, source, and target, then supplies terms from both matches for the new segment.Source: §A.5 · p. 11
- The EN-ES two-shot glossary variant uses fuzzy-match examples but supplies glossary terms for the new segment instead of terms gathered from the matches.Source: §A.5 · p. 11