Source-linked AI summary
Prompting Large Language Model for Machine Translation: A Case Study
Biao Zhang, Barry Haddow, Alexandra Birch
TL;DR
Prompting for machine translation remains under-explored, so this paper systematically studies templates, demonstrations, monolingual data, and transfer with GLM-130B. It finds that prompting choices substantially affect translation, pseudo-parallel examples and cross-setting demonstrations can help, but optimal demonstrations do not generalize reliably and important failure modes remain.
Problem
The paper addresses the under-explored problem of how prompting strategies affect machine translation.
Method
It conducts a systematic GLM-130B study of templates, demonstration selection, monolingual-data prompting, and transfer across languages, domains, and translation levels.
Results
Prompt templates and demonstration selection substantially affect translation; pseudo-parallel examples and demonstrations transferred from other settings can improve performance, although demonstration optimality does not generalize reliably.
Takeaways & Limitations
Prompting for MT benefits from retaining source-target mapping signals and can use zero-shot back- or forward-translation to construct effective pseudo-parallel demonstrations.
Takeaways & Limitations
The study relies heavily on INT4-quantized GLM-130B, mainly covers three languages, and leaves generalization to other pretrained LLMs unclear.
Abstract
from arXiv · showhide
Research on prompting has shown excellent performance with little or even no supervised training across many tasks. However, prompting for machine translation is still under-explored in the literature. We fill this gap by offering a systematic study on prompting strategies for translation, examining various factors for prompt template and demonstration example selection. We further explore the use of monolingual data and the feasibility of cross-lingual, cross-domain, and sentence-to-document transfer learning in prompting. Extensive experiments with GLM-130B (Zeng et al., 2022) as the testbed show that 1) the number and the quality of prompt examples matter, where using suboptimal examples degenerates translation; 2) several features of prompt examples, such as semantic similarity, show significant Spearman correlation with their prompting performance; yet, none of the correlations are strong enough; 3) using pseudo parallel prompt examples constructed from monolingual data via zero-shot prompting could improve translation; and 4) improved performance is achievable by transferring knowledge from prompt examples selected in other settings. We finally provide an analysis on the model outputs and discuss several problems that prompting still suffers from.
1 Introduction
This study addresses the limited exploration of prompting for machine translation by systematically examining templates, examples, monolingual data, and transfer. Experiments with GLM-130B show that prompting choices matter, pseudo-parallel examples help, transfer is possible but imperfect, and translation still has recurring failure modes.
- The paper studies machine-translation prompting as a test of LLM cross-lingual generation and its implications for universal LLM design.
- The study systematically evaluates prompting strategies, template choices, prompt-example features, monolingual-data construction, and transfer across settings.Experiments use GLM-130B on FLORES and WMT evaluation sets across English, German, and Chinese.
- An English template in a simple form works best for machine translation, while language-specific templates mainly help when translating into languages used in pretraining.
- Prompt-example features correlate significantly but weakly with performance, so feature-based selection can beat random selection inconsistently.
- Monolingual examples hurt translation, whereas pseudo-parallel examples constructed through back- or forward-translation improve prompting, with back-translation better and more robust.
- Demonstrations from other settings can improve translation over zero-shot prompting, but demonstration superiority rarely generalizes across settings.
- Prompting for machine translation still exhibits copying, entity mistranslation, hallucination, inferior direct non-English translation, and prompt traps.
2 Setup
The setup defines zero-shot and few-shot translation prompting with templates and demonstrations, then evaluates GLM-130B across languages, datasets, metrics, and example-selection conditions. The experiments vary prompt-example counts and use ablation sets to measure performance and variability.
- MT prompting converts each test input into a template-based prompt and generates its translation with a fixed pretrained LLM.The study considers both zero-shot and few-shot prompting.
- Few-shot prompting adds labeled prompt examples, whose source-target pairs are concatenated before the test input.
- Cross-lingual transfer allows prompt examples to use a different language pair from the test data.
- Template language denotes the language in which the template is expressed, such as Chinese or English.
- Experiments use the INT4-quantized GLM-130B, pretrained on Chinese and English monolingual corpora, with English, German, and Chinese translation settings.
- The evaluation covers FLORES Wiki data, WMT21 News data, and Multi-Domain data, using detokenized BLEU and COMET.
- Ablation experiments randomly sample 100 test instances while using the remainder as the default example-selection pool.
3 Prompting Strategy for MT
The study shows that MT prompting is highly sensitive to template and demonstration choices. More examples can improve average translation, but example quality and selection remain unstable, while feature-based selection offers inconsistent gains.
- Prompt templates: The simple English template works best for MT on average, while language-specific templates produce mixed results across languages.German templates can substantially degrade translation, whereas Chinese templates help when translating into Chinese.
- Prompt examples: Using more prompt examples generally improves average translation but increases GPU memory use and inference time.Performance can vary substantially at the same number of examples, and 1-shot prompting often underperforms zero-shot prompting.
- Language effects: Few-shot prompting greatly improves translation into Chinese because examples help avoid zero-shot outputs in traditional Chinese with messy codes.The reference text is always simplified Chinese.
- Example selection: Several demonstration features correlate significantly with prompting performance, but the relationships are generally weak and fragile.LMScore and CaseSemScore-Tgt correlate most strongly on average, followed by TLength and SemScore; most Spearman’s ρ values are below 0.5.
- Example selection: Feature-based example selection can outperform random sampling, with SemScore performing well across settings, but improvements are not consistent.A combined SemScore, LMScore, and TLength strategy is used for low-quality pools and outperforms random selection by varying degrees.
4 Monolingual Data for Prompting
The study tests whether monolingual data can support MT prompting. Directly pairing or using monolingual examples harms translation, whereas zero-shot construction of pseudo-parallel examples improves prompting, especially through back-translation.
- Direct monolingual demonstrations: Directly using monolingual examples for demonstrations almost always hurts translation, with greater degeneration as more examples are added.Randomly paired examples generally perform worst, while source-only examples are slightly better than target-only examples except for translation into Chinese.
- Direct monolingual demonstrations: Demonstrations must retain genuine source-target mappings for MT prompting.Randomly pairing monolingual sources and targets misleads the model, indicating that MT differs from settings where label genuineness may matter less.
- Pseudo-parallel demonstrations: Pseudo-parallel examples constructed by zero-shot forward- or back-translation improve prompting despite their low quality.Back-translation of target monolingual data performs better and more robustly than forward-translation, sometimes approaching real parallel examples.
5 Transfer Learning for Prompting
The paper tests whether prompt demonstrations transfer across language pairs, domains, and translation granularity. Transfer can improve translation, but demonstration quality rankings are not stable across settings.
- Research Questions: The study evaluates whether demonstrations selected in one setting retain their superiority and whether out-of-setting demonstrations beat zero-shot prompting.The transfer settings include cross-lingual, cross-domain, and sentence-to-document translation.
- Cross-Setting Ranking: Demonstration rankings generalize poorly across settings, with weak and often insignificant Spearman correlations, including reversed directions and similar domains.The results indicate that setting-specific demonstrations are needed for optimal translation quality.
- Transfer Results: Out-of-setting demonstrations can improve translation, with relatively easier transfer when the source or target language is shared or directions are reversed.Cross-domain transfer can also succeed when the in-setting example pool is low quality.
- Transfer Results: Sentence-level demonstrations improve both BLEU and document-specific evaluation in transfer to document-level translation.The document-level experiment uses one-shot prompting with sentence-level demonstrations.
- Transfer Limits: Transfer is unstable and can perform worse than zero-shot prompting, while selecting demonstrations for transfer settings remains unresolved.The paper leaves transfer-specific example selection for future work.
6 Discussion
The discussion identifies recurrent prompting failures in translation, including off-target outputs, copying, entity errors, hallucination, prompt traps, and weak direct German–Chinese translation. Pivoting through English improves non-English translation in the reported setting.
- Observed Errors: Prompting may reject translation by producing empty or off-target outputs, especially when translating into Chinese.Chinese outputs may instead be traditional Chinese with messy codes, contributing to unstable performance.
- Observed Errors: Observed errors also include under-translation, source copying, code-switching, entity mistranslation, and hallucination.The paper illustrates these errors through a case study of prompting outputs.
- Prompt Trap: Prompt traps arise when input text contains prompt-template phrases, causing the model to copy those phrases rather than translate them.This makes translating the prompt itself non-trivial and creates a possible input-format attack surface.
- Cross-Lingual Translation: Direct German–Chinese translation quality is very poor, while pivoting through English greatly improves non-English translation.The discussion attributes the observed pattern to GLM-130B’s cross-lingual ability mainly centering around English.
7 Related Work
Related work connects prompting for translation to prompt-template design, example retrieval, multilingual neural MT, and concurrent studies of prompting strategies for machine translation.
- Prompt Design and Retrieval: Prior prompting research develops automatic template generation and retrieves examples close to the test input.These approaches reflect sensitivity to prompt surface representation and example choice.
- Neural MT Connections: Multilingual neural MT uses target-language tags, attribute tags, and retrieved phrases or sentences to control or improve translation.Prompting research has also incorporated such ideas into neural MT systems.
- Concurrent MT Prompting Studies: Concurrent studies disagree on whether randomly chosen high-quality examples or input-specific examples provide the stronger prompting strategy.One study reports random high-quality examples performing comparably or better, while another finds n-gram-overlap retrieval beneficial.
8 Conclusion and Future Work
The conclusion summarizes a systematic study showing that prompt templates and demonstrations substantially affect MT, while monolingual data, pseudo-parallel construction, and transfer offer uneven benefits. The authors highlight unstable transfer and other unresolved prompting problems.
- Main Findings: Prompt templates and demonstration selection substantially affect translation, while feature-based example selection helps only inconsistently because correlations are weak.The study covers prompting strategy, unlabeled monolingual data, and transfer learning.
- Monolingual Data: Prompting demonstrations must retain source-target mapping signals; monolingual examples hurt, whereas zero-shot back- or forward-translation constructs effective pseudo-parallel examples.The conclusion presents pseudo-parallel construction as a simple solution for using monolingual data.
- Transfer Learning: Sentence-level demonstrations can improve translation in other domains, language pairs, and document-level settings, but transfer performance is unstable.The conclusion distinguishes positive transfer results from the failure of demonstration optimality to generalize.
- Future Work: Future work includes addressing off-target generation and prompt traps, testing other LLMs, and improving cross-lingual ability.The authors specifically mention GPT-3, OPT, and PaLM as additional models of interest.
Limitations
The study is constrained by reliance on INT-4 quantized GLM-130B, limited language coverage, and substantial variation across language pairs.
- The experiments rely heavily on INT-4 quantized GLM-130B, whose quantization may weaken capabilities in unknown ways.The paper states that generalization to other pretrained LLMs remains unclear.
- The findings may not generalize to other pretrained LLMs because GLM-130B used different pretraining objectives from GPT and PaLM.
- Resource constraints limited experiments mainly to three languages, while results varied greatly across language pairs.
A Appendix
The appendix documents dataset statistics, prompting templates, example-selection analyses, transfer experiments, and detailed translation results across Wiki and WMT settings.
- Dataset statistics: Table 9 reports statistics for the ablation and full datasets, including instance counts and WikiMatrix.v1 data.
- Prompting templates: Table 10 compares zero-shot prompting templates and template languages, with English Template A achieving the best overall BLEU and COMET performance.
- Prompt-example analysis: Figure 5 shows COMET and BLEU distributions across K = 1, 5, 10, and 20 prompt examples using 100 random samples per setup.The figure also marks the zero-shot baseline and plots means with standard-deviation shading.
- Prompt-example analysis: Figures 6 and 7 plot relationships between LMScore and BLEU or COMET for 1-shot prompting across Wiki language-pair ablations.
- Prompt-example analysis: Table 11 reports Spearman’s ρ between demonstration features and prompting performance for 600 sampled demonstrations per pool.The analysis compares high-quality default-pool examples with low-quality WikiMatrix.v1 examples and marks statistical significance.
- Prompt-example analysis: Table 12 shows that SemScore top-ranked parallel examples can be highly semantically similar yet insufficiently informative, motivating their removal.
- Transfer and full-set results: Figures 8 and 9 and Tables 15–19 cover monolingual-data prompting, cross-domain transfer, pivoting, and multi-domain translation results.Table 17 reports Spearman’s ρ and relative BLEU for cross-domain transfer, while Table 19 averages results over three demonstrations.