PaperlayerView exact revision on arXiv

Source-linked AI summary

Prompting PaLM for Translation: Assessing Strategies and Performance

David Vilar, Markus Freitag, Colin Cherry, Jiaming Luo, Viresh Ratnakar, George Foster

arXiv:2211.09102v3cs.CL

TL;DR

This paper investigates how prompting strategies affect PaLM’s translation ability despite its lack of intentionally included parallel training text. It studies example selection and evaluates optimized prompts with modern automatic and human metrics, finding that example quality matters most while PaLM still trails state-of-the-art systems.

  • Problem

    PaLM’s translation ability requires systematic assessment because it achieves strong MT performance despite training on multilingual data without intentionally included parallel text.

  • Method

    The paper fixes prompt templates, compares random and kNN example selection across candidate pools, and evaluates translations with BLEURT and MQM.

  • Results

    Example quality is more important than pool domain or source proximity, with random selection from small high-quality pools performing well.

  • Takeaways & Limitations

    PaLM remains highly fluent and creative but prone to omissions and accuracy errors, broadly matching NMT fluency while lagging its accuracy.

  • Takeaways & Limitations

    The conclusions are limited by few language pairs, English-centered directions, sentence-level evaluation, and incomplete exploration of prompting methods.

Abstract

from arXiv · show

Large language models (LLMs) that have been trained on multilingual but not parallel text exhibit a remarkable ability to translate between languages. We probe this ability in an in-depth study of the pathways language model (PaLM), which has demonstrated the strongest machine translation (MT) performance among similarly-trained LLMs to date. We investigate various strategies for choosing translation examples for few-shot prompting, concluding that example quality is the most important factor. Using optimized prompts, we revisit previous assessments of PaLM's MT capabilities with more recent test sets, modern MT metrics, and human evaluation, and find that its performance, while impressive, still lags that of state-of-the-art supervised systems. We conclude by providing an analysis of PaLM's MT output which reveals some interesting properties and prospects for future work.

1 Introduction

The paper systematically investigates PaLM’s machine translation prompting and reassesses its performance with modern evaluation practices. It finds that example quality strongly affects results, while PaLM still lags state-of-the-art supervised MT.

  • Motivation: PaLM demonstrates strong translation capabilities despite training on an overwhelmingly English, multilingual corpus without intentionally included parallel text.The study focuses on the 540B-parameter PaLM model, reported as the strongest MT-performing LLM trained on non-parallel multilingual data.
  • Prompting study: The study varies both the candidate pool and selection strategy for few-shot translation examples under fixed prompt templates.It compares random selection with k-nearest-neighbour selection customized to the source input.
  • Prompting study: 516 of 1000 sentences show more than 1 BLEURT point of difference between two random 5-shot PaLM runs, highlighting prompt sensitivity.Each histogram bar represents a 1-BLEURT-point difference range.
  • Evaluation: The evaluation uses recent WMT test sets, BLEURT, and expert MQM human assessment to reduce leakage concerns and characterize translation errors.The paper replaces BLEU with BLEURT for sentence-level quality assessment and adds detailed human error categories.
  • Findings: Example quality matters more than example domain or lexico-semantic proximity, while PaLM’s impressive translation capacity still lags state-of-the-art MT.These are stated as the paper’s principal findings and contributions.

2 Related Work

Related work establishes hard and soft prompting as broad approaches to adapting LLM behavior, with prior MT studies using templates and randomly selected examples. Subsequent work expands prompt selection, data augmentation, and translation control methods.

  • Prompting approaches: Hard prompting uses explicit text, whereas soft prompting learns embeddings, activations, or attention weights to condition a model.Hard prompts are described as easier to interpret and modify, while soft prompts are more expressive and efficient.
  • Prompting approaches: Earlier hard-prompting research developed handcrafted, gradient-guided, exhaustive-search, and example-ordering methods.These methods target prompt construction, labels, templates, or the ordering of demonstrations.
  • Input conditioning: Prior input-conditioned prompting used kNN retrieval and reported gains over random selection as domain-controlled example pools grew.The cited work applied RoBERTa embeddings to select relevant GPT-3 prompts for several tasks.
  • MT prompting: Early MT prompting with GPT-3 and PaLM compared zero-, one-, and n-shot random sentence-pair selection, with performance increasing as shots increased.Reported BLEU results were competitive with previous unsupervised SOTA and, in some settings, supervised SOTA.
  • Recent directions: Other work studies prompt templates, output attributes, autoregressive prompting, prompt quality, source proximity, dictionaries, chain-of-thought-inspired retrieval, and bilingual data.Recent research also explores controlling formality and terminology, sometimes through human interaction.

3 Prompting for Machine Translation

The paper simplifies prompting for machine translation by fixing the template and concentrating on selecting few-shot examples. It compares random and kNN-based selection using lexical or semantic representations.

  • Prompting setup: The prompting pipeline includes template generation, slot filling, answer search, and answer mapping, but this study fixes the template and investigates slot filling.In MT, generated answers can be used directly, simplifying answer search and mapping.
  • Prompting setup: The template inserts language-labeled translation pairs, the current source sentence, and a target-language slot before PaLM generates the translation.Each example fills an (X_i, Y_i) pair, while the final source slot contains the current input.
  • Inference procedure: Translation examples are selected, inserted into the template, and followed by greedy generation that stops when PaLM outputs a newline.The examples may depend on the current source text.
  • Example selection: Random selection provides the baseline, while kNN selection retrieves examples close to the source text from a chosen parallel-text pool.The study tests whether source similarity helps guide similar translations.
  • Inference procedure: The study found that nonzero sampling temperature degraded translation quality, so the reported generation uses greedy decoding.This implementation choice is stated in a footnote.
  • Example selection: Bag-of-words retrieval favors lexical similarity, whereas multilingual RoBERTa embeddings are intended to capture semantic similarity.Cosine distance is used for bag-of-words vectors and Euclidean distance for RoBERTa embeddings.

4 Data

The experiments cover Chinese, French, and German translation into and out of English using recent WMT news data and three prompt pools. The pools vary in size, quality, domain match, and alignment format.

  • Language pairs and test data: The experiments cover translation into and out of English for Chinese, French, and German, with Chinese representing a smaller training-corpus share and a more difficult task.English accounts for 78.0% of PaLM’s training corpus, compared with German at 3.5%, French at 3.3%, and Chinese at 0.4%.
  • Language pairs and test data: The evaluation uses WMT 2021 news data for most language pairs and older WMT14 data for French because French was absent from WMT21.The French test sets are not purely source-original, unlike more recent sets.
  • Prompt pools: Prompt selection uses WMT-full, WMT-dev, and a manually curated high-end pool containing professionally edited bilingual articles.WMT-full is largest but varies in quality; WMT-dev is generally higher quality and closer in domain to the test set.
  • Prompt pools: The high-end pool spans multiple domains and is used in both translation directions despite frequently lacking one-to-one sentence alignment.Its examples are treated as symmetrical paragraph pairs.

5 Experiments

Experiments show that example-pool quality matters more than selection strategy for PaLM prompting, while PaLM remains behind specialized SOTA MT systems on recent evaluations. Its translations are generally fluent but less accurate, especially because of omission errors.

  • Evaluation setup: Five-shot prompting is used for WMT pools after gains diminish beyond five sentence pairs, while the high-end pool uses one paragraph example.Random-selection results are reported using the median BLEURT score across five seeds.
  • Evaluation setup: Sentence-level evaluation uses recent WMT test sets, BLEURT and BLEU metrics, and expert MQM human assessments.The study also compares PaLM with 2021 WMT systems and Google Translate; French lacks SOTA and human evaluation because its test set is older.
  • Selection strategies and pools: The choice of example pool matters more than selection method: WMT-dev outperforms WMT-full, while RoBERTa kNN has no consistent advantage over random selection.The larger full pool can provide closer source matches, but source-only matching increases the risk of poor translations.
  • Results on all language pairs: For German and Chinese, WMT-dev random selection outperforms WMT-full, while high-end examples perform similarly to WMT-dev despite domain and style mismatch.These results support example quality as the main determinant of PaLM’s output quality.
  • Results on all language pairs: French reverses the general pattern: WMT-full random selection performs best, but all methods differ by only approximately 0.5 BLEURT.The authors suggest the age and quality of newstest2014 as one possible explanation.
  • Results on all language pairs: Specialized SOTA systems exceed PaLM by 1–3 BLEURT points, while Google Translate’s gap is generally under 1 point except for Chinese→English at 1.8.PaLM is relatively stronger into English, with smaller drops than in the reverse direction.
  • Results on all language pairs: MQM shows PaLM has fluency scores generally similar to SOTA but lower accuracy, dominated by Major Accuracy/Omission errors.Some best-performing PaLM systems make fewer Style/Awkward errors than SOTA in certain languages.
  • Comparison to previous results: Using the tokenizer from the earlier study makes the English→French score match exactly; the reported discrepancy otherwise is 1.7 BLEU points.The directly comparable evaluation uses WMT14 French test data and WMT-full BLEU scores.

6 Analysis

The analysis examines PaLM’s translation behavior, prompt-selection effects, output characteristics, and test-set overlap. It finds that relevant examples can help but noisy alignments can induce hallucinations, while newer test sets reduce overlap concerns.

  • Evaluation: The study uses BLEURT and human MQM evaluation alongside comparisons with supervised and off-the-shelf systems.The evaluation design combines recent test sets, modern automatic scoring, and expert error analysis.
  • Prompt selection: kNN retrieval can improve translation when it finds relevant examples, but alignment errors in retrieved examples can cause hallucinated output.A football-domain example shows both the benefit of relevant retrieval and the risk of noisy alignments.
  • Output characteristics: PaLM translations are generally less literal than supervised NMT outputs, occasionally missing important source content.Supervised systems are more faithful to the source, although that can sometimes produce less natural target-language translations.
  • Test-set overlap: Newer WMT test sets have much smaller target-side overlap with PaLM’s training data than older French→English and German→English sets.Overlap is measured using 15-gram matches tokenized with mBERT, and clean-set sizes are reported in Table 6.

7 Conclusion

The conclusion identifies example quality as the central prompt-selection factor and reassesses PaLM against current MT systems. PaLM remains impressive but trails stronger systems in accuracy despite comparable fluency.

  • 7 Conclusion: Example quality matters more than domain or translation style when selecting few-shot prompts.Random selection from small, high-quality pools performs well, while kNN’s relevance advantage is offset by susceptibility to corpus noise.
  • 7 Conclusion: PaLM’s sentence-level translation performance significantly lags competition-grade SOTA systems and, to a lesser extent, COTS systems on recent WMT test sets.The comparison uses BLEURT and MQM human evaluations across German, Chinese, and French translations into and out of English.
  • 7 Conclusion: PaLM’s output is creative and fluent but prone to omissions and other accuracy errors across the tested languages.Broadly, it matches conventional NMT in fluency while lagging in accuracy.
  • 7 Conclusion: Future work should examine document-level translation and more expressive prompt tuning to improve accuracy without sacrificing fluency.The paper specifically identifies these directions as ways to exploit long contexts and tighten PaLM’s MT accuracy.

Limitations

The study’s conclusions have limited generality because it evaluates a small set of well-represented languages only into and out of English. Sentence-level evaluation may also underestimate PaLM’s capabilities.

  • Limitations: The conclusions may not generalize beyond well-represented languages or translation directions involving English.The authors evaluated only a small number of language pairs and explicitly qualify the scope of their conclusions.
  • Limitations: Independent sentence-level evaluation may underestimate PaLM because some accuracy problems could be less severe in whole-document translation.The authors note that document context could change the interpretation of observed accuracy errors.

Ethical Considerations

The paper situates few-shot MT prompting within broader ethical concerns about large language models. It specifically considers the implications of relying on trusted sources for prompt examples.

  • Ethical Considerations: Large language-model research raises ethical concerns that also apply when MT is studied as one task among many.The paper points to prior discussions of these concerns while focusing on example selection for few-shot MT prompting.
  • Ethical Considerations: Emphasizing prompt quality could encourage systems to draw examples from small sets of trusted sources.The authors identify this as a possible deployment scenario and describe their high-end pool as one such example.

Appendices

The appendices examine prompt-template choices and example counts for few-shot translation, while documenting the construction of the high-end prompt pool. They report that template choice matters less than the number and quality of examples, motivating five-example prompts.

  • A Prompt Exploration: The paper’s main template prepends each example with the corresponding language name in English.Alternative formats include language names in German, two-letter language codes, an added translation header, and textual instructions.
  • A Prompt Exploration: Table 7 evaluates BLEURT across prompt templates and different numbers of randomly selected examples for English→German translation.The reported values are medians over five runs.
  • A Prompt Exploration: Prompting strategies have a crucial impact with few shots, but their effect decreases as more examples are added.Increasing the number of examples also improves translation quality.
  • A Prompt Exploration: The study uses five examples because additional prompts yield diminishing returns and may exceed the maximum input length.This choice balances quality against practical input-length implications.
  • Prompt Pool: The high-end prompt pool contains manually downloaded articles divided into bilingual paragraphs, with domain distributions reported separately.The pool comprises all paragraphs from the included articles.

C Variability of Random Runs

The appendix examines variability across random prompt selections and reports automatic and human-evaluation comparisons with SOTA systems. Random-run score ranges are generally small, while prompt-selection outcomes and translation behaviors vary by direction and method.

  • C Variability of Random Runs: Less than 0.5 BLEURT points separates all random runs for the German→English directions.Using WMT-dev instead of WMT-full reduces the observed range by at least 0.1 in both directions.
  • C Variability of Random Runs: MQM scores compare PaLM WMT-dev random with SOTA systems across four language pairs, including Accuracy and Fluency components.Detailed MQM error counts are provided for English→German and German→English.
  • C Variability of Random Runs: PERM-BOTH pair-wise significance testing supplies the p-values reported in Table 13.The table highlights results with p<0.05.
  • C Variability of Random Runs: Tables 14a and 14b show examples where kNN or random selection performs better, respectively.Additional tables illustrate German→English and English→Chinese translations.

H Overlap Analysis

The overlap analysis shows that train–test overlap can distort comparisons between PaLM and Google Translate, although the overall score effects are limited. The appendix also reports fixed-prompt comparisons and documents translation-quality trade-offs between PaLM and SOTA systems.

  • H Overlap Analysis: A clean evaluation is needed because train–test overlap can inflate apparent differences between PaLM and Google Translate.The analysis compares original, clean, and fully overlapping test subsets using BLEU and BLEURT.
  • H Overlap Analysis: 2.0 BLEU is the clean de→en difference between PaLM and Google Translate, compared with 1.5 BLEU on the original test.The fully overlapping subset shrinks this difference to 0.4 BLEU, while PaLM’s overlap benefit is 0.5 BLEU in the comparison.
  • H Overlap Analysis: Up to 0.7 BLEU or 0.3 BLEURT of system-delta distortion occurs with 20–30% overlap.The passage emphasizes that overlap effects are small overall and smaller than raw original-versus-clean score differences suggest.
  • H Overlap Analysis: The difficulty difference between clean and fully overlapping subsets remains unexplained for systems without overlap.Average sentence lengths are similar across the compared de→en and fr→en subsets.
  • Fixed Prompt Selection: A fixed maximum-likelihood prompt performs as well as or better than the average of five per-input random prompts for every language pair except Chinese→English.In Chinese→English, the fifth-ranked prompt still beats the random average, suggesting a problem with the held-out set.
  • Translation Output Analysis: PaLM is less literal than SOTA in German→English, sometimes missing source information, while in English→Chinese it has more omissions and mistranslations.The examples include cases where SOTA is overly literal but remains generally more faithful to the source.
Loading 2211.09102v3…