Source-linked AI summary
Using natural language prompts for machine translation
Xavier Garcia, Orhan Firat
TL;DR
This paper asks how natural-language prompts can control machine-translation outputs and target-language selection. It adapts multilingual translation models to use prompts and evaluates their effects on formality, dialects, and translation into languages unseen during fine-tuning. The authors report positive transfer to zero-shot language pairs and find that adapting already-trained models works better than training from scratch with prompts.
Problem
The paper asks how natural language can control multilingual translation outputs, including target-language selection, and whether prompts can support translation into languages unseen during fine-tuning.
Source: §1 · p. 1, §4.3 · p. 7Method
Results
Adapting already-trained translation models to use prompts improves zero-shot directions and translation into languages without parallel data, while supervised performance changes little.
Source: §4.3 · p. 8, §5 · p. 8Takeaways & Limitations
English-language names in prompts allow translation into languages seen during pre-training but not fine-tuning, and prompt adaptation can extend trained models to those targets.
Source: §1 · p. 2, §5 · p. 8Some comparisons have mismatched or unclear conditions: FLAN/GPT-3 BLEU preprocessing is not confirmed, and the German-Czech WMT baseline uses no parallel data.
Source: §4.1 · p. 5
Abstract
from arXiv · showhide
We explore the use of natural language prompts for controlling various aspects of the outputs generated by machine translation models. We demonstrate that natural language prompts allow us to influence properties like formality or specific dialect of the output. We show that using language names to control the output language of multilingual translation models enables positive transfer for unseen language pairs. This unlocks the ability to translate into languages not seen during fine-tuning by using their English names. We investigate how scale, number of pre-training steps, number of languages in fine-tuning, and language similarity affect this phenomenon.
1 Introduction
Multilingual translation uses a shared model to draw on multiple language pairs, while language tags specify the target language. This section motivates replacing such artificial tags with natural-language prompts, which may flexibly express tasks and target languages beyond those seen in training.
1 Introduction
- Language tags such as <2es> tell a multilingual model to translate into Spanish and require no architecture changes, with only minor memory costs per language.During training, the model learns the association between the tag and the translation task.Source: §1 · p. 1
- Multilingual MT trains one model on data from available language pairs, and joint training can yield disproportionate gains for low-resource pairs.The authors connect this positive transfer to a broader pattern in which training across more tasks can strengthen transfer, motivating flexible task specification.Source: §1 · p. 1
- Natural-language prompts add task instructions to a unified text-to-text input, potentially allowing models to address tasks unseen during training.The section contrasts this with language-specific input tokens used to select a target language.Source: §1 · pp. 1–2
- The paper adapts multilingual models to prompts such as “Translate to Spanish:” and reports that this enables translation into languages absent from fine-tuning by using their English names.It also reports improved performance on zero-shot language pairs and examines effects of scale, pre-training, language-pair count, and language similarity.Source: §1 · p. 2
2 Related works
Prior prompting work largely studied English-centric models, while alternatives to language tags had also been explored for multilingual translation. This section identifies a gap: natural-language prompts had not been compared with language tags or studied for unseen tasks.
Text generation through prompting
- Earlier prompting studies typically focused on large, English-centric language models, leaving prompting for multilingual models underexplored.The passage notes that Lin et al. (2021) had recently begun studying multilingual prompting.Source: §2 · p. 2
Language bias for multilingual machine translation
- Prior alternatives to language tags included language or subword embeddings, task-specific attention, and decoder-side language tags.These approaches modify or supplement how multilingual models represent or condition on the target language.Source: §2 · p. 2
- Raffel et al. introduced prompts of the form “Translate to {language_name}:” but did not compare them with language tags or study their effects on unseen tasks.The paper frames its work as addressing both gaps in multilingual translation.Source: §2 · p. 2
3 Preliminary exploration
The preliminary experiment tests whether a prompt can shift a black-box translation model toward formal second-person pronouns. Across five target languages, prompted translations receive higher formality scores than unprompted translations.
3 Preliminary exploration
- The experiment translates 100 English sentences containing “you” into five languages, using “Your majesty:” as a prefix before translation.The researchers tracked whether a second-person pronoun appeared and whether it was formal or informal; the prompt was removed from the translated output.Source: §3 · p. 2
- In the Spanish example, prompting changes the output from informal “tu” to the formal pronoun and also yields the subjectively more formal word “obtuvo.”The authors use Google Translate as a black-box model that accepts the text to translate but does not expose model controls.Source: §3 · p. 2, Figure 1 · p. 3
- Across all languages considered, native speakers annotated second-person T–V pronoun use, and summed formality scores were higher for translations generated with prompts than for those generated without prompts.Formal second-person pronouns received 1, informal pronouns −1, and cases with no second-person pronoun received 0.Source: §3 · p. 3
4 Experiments
The experiments examine natural-language prompting for unseen-language translation, dialect control, language-similarity and multitask effects, and adaptation of multilingual translation models. They also assess how prompt-based control affects zero-shot and supervised performance.
4.1 Experiments with mT5
- Adapted mT5-XXL outperformed FLAN, GPT-3 (including few-shot), and XGLM; the 10%-pretrained XXL was comparable to XL on English-French and English-German.For English-Romanian, the paper used Caswell et al. (2019) results because they explicitly used sacreBLEU; Liu et al. (2020) used different preprocessing.Source: Table 2 · p. 4, §4.1 · p. 5
- The fine-tuning scheme used “Translate to {language_name}:” with the target language’s English name at inference, including for languages unseen during fine-tuning.Source: §4.1 · p. 4
- Unseen-language translation degraded over time as models specialized; German-French fell faster than English-centric pairs, and step-for-scale tradeoffs were less efficient for English-Kazakh and non-English-centric pairs.Source: §4.1 · p. 5
4.2 Assessing the value of language similarity and multi-task setups
- Language similarity was critical but one related pair was insufficient; more language pairs mitigated catastrophic forgetting, while low-resource Romanian was more vulnerable than French.Figure 2 compares French and Romanian and notes that the full 51-language-pair setup showed some robustness in Romanian.Source: Figure 2 · p. 6, §4.2 · p. 7
4.1 Experiments with mT5
- Dialect-name prompts matched or exceeded base-language BLEU for every tested dialect, with half significantly higher at 0.01, without dialect-labeled data.Source: Table 4 · p. 6, §4.1 · p. 6
4.3 Adapting multilingual translation models
- Prompt training from scratch yielded little improvement over language tags and caused large regressions on English-German and Czech-German.The adaptation approach instead fine-tuned a language-tags model for 100 steps with prompts, aiming to avoid regression and isolate prompt effects.Source: §4.3 · pp. 7–8, Table 5 · p. 8
- Adapting a language-tag-trained model to prompts brings massive improvements in zero-shot directions and languages without parallel data, with very little change in supervised performance.The authors characterize the gains as coming at almost no cost.Source: §4.3 · p. 8
- Prompts are multi-token language tags that exacerbate issues with traditional language tags and reduce the examples that fit in a batch.Source: §4.3 · p. 8
5 Conclusion
The conclusion reports that natural-language prompts can control formal pronoun use and target languages seen during pre-training but absent from fine-tuning. Training from scratch with this strategy was initially weak, while adapting already-trained models through a short stage produced positive results.
5 Conclusion
- Language-name prompts let users translate into languages seen during pre-training but not fine-tuning.The conclusion also reports positive results from a short prompt-adaptation stage on already-trained translation models.Source: §5 · p. 8
- Prepending certain phrases increased the occurrence of formal pronouns in translations, as measured by the T–V distinction.Source: §5 · p. 8
- Training translation models from scratch with this strategy initially yielded weak results, unlike short-stage adaptation of already-trained models.Source: §5 · p. 8
A Example Appendix
The passage only identifies this material as an appendix.
A Example Appendix
- The passage labels the material as an appendix but gives no further substantive content.Source: A Example Appendix · p. 10