PaperlayerView exact revision on arXiv

Source-linked AI summary

In-context Examples Selection for Machine Translation

Sweta Agrawal, Chunting Zhou, Mike Lewis, Luke Zettlemoyer, Marjan Ghazvininejad

arXiv:2212.02437v1cs.CL

TL;DR

The paper addresses the unclear effects of in-context example selection, ordering, number, and domain on machine translation. It analyzes task-level and example-specific prompts, introducing n-gram-overlap reranking for retrieved examples. The approach improves translation quality and outperforms kNN-MT on two of four out-of-domain datasets.

  • Problem

    The effects of in-context example choice, ordering, number, and domain on MT translation quality remain unclear.

  • Method

    The paper evaluates task-level prompts optimized on development BLEU and reranks BM25-retrieved example-specific prompts using recall-based source n-gram overlap.

  • Results

    The approach outperforms kNN-MT on 2 of 4 out-of-domain datasets while remaining memory and compute efficient.

  • Takeaways & Limitations

    Good, meaning-equivalent examples and source-similar examples improve MT prompting relative to random or poorly selected examples.

Abstract

from arXiv · show

Large-scale generative models show an impressive ability to perform a wide range of Natural Language Processing (NLP) tasks using in-context learning, where a few examples are used to describe a task to the model. For Machine Translation (MT), these examples are typically randomly sampled from the development dataset with a similar distribution as the evaluation set. However, it is unclear how the choice of these in-context examples and their ordering impacts the output translation quality. In this work, we aim to understand the properties of good in-context examples for MT in both in-domain and out-of-domain settings. We show that the translation quality and the domain of the in-context examples matter and that 1-shot noisy unrelated example can have a catastrophic impact on output quality. While concatenating multiple random examples reduces the effect of noise, a single good prompt optimized to maximize translation quality on the development dataset can elicit learned information from the pre-trained language model. Adding similar examples based on an n-gram overlap with the test source significantly and consistently improves the translation quality of the outputs, outperforming a strong kNN-MT baseline in 2 out of 4 out-of-domain datasets.

1 Introduction

The paper studies how in-context example choice, ordering, number, and domain affect machine-translation quality. It proposes retrieval and reranking strategies that improve translation, including against kNN-MT on two out-of-domain datasets.

  • In-context learning enables large language models to perform NLP tasks from a few input-output examples.A 7.5B multilingual model reportedly outperformed a supervised baseline in 45 translation directions using 32 random examples.
  • MT commonly uses randomly sampled development examples, leaving the effects of example choice and domain on translation quality unclear.The paper identifies out-of-domain generalization as an important unresolved challenge.
  • The study systematically analyzes how example choice, number, ordering, and domain affect MT outputs.
  • A single noisy unrelated example can substantially damage translation quality, whereas a well-formed meaning-equivalent example performs better than random examples.
  • The proposed n-gram-coverage reranking strategy improves BM25-retrieved examples on WMT’19 English↔German and English↔Russian.The method is designed for limited context windows and selects examples covering source n-grams.
  • Combining optimized task-level and example-specific prompts outperforms kNN-MT on two out-of-domain datasets while remaining memory and compute efficient.

2 Background: In-context Learning

In-context parallel examples condition decoder-only multilingual language models for translation by specifying the task and supplying information about the unseen source. Prompts concatenate demonstrations with a test source under a source-target template.

  • Decoder-only multilingual models require in-context parallel examples to generate translations.
  • Examples provide task format and knowledge, while also guiding generation with information related to the unseen source sentence.
  • A prompt concatenates k source-target demonstrations with the test source according to a template.
  • The model generates the output translation with parameters θ using greedy decoding.
  • The template represents each demonstration and test input as source text followed by target text.

3 Prompt Selection

The paper selects task-level prompts by development-set translation quality and example-specific prompts by source n-gram overlap. A recall-based reranker addresses BM25’s incomplete coverage under limited context windows.

  • 3 Prompt Selection: The paper asks how prompt choice, quantity, and ordering affect MT quality and what constitutes a good MT example.
  • 3.1 Task-level In-context Examples: A task-level example should elicit pretrained information, so examples are ranked by development-set BLEU after prompting every development source.
  • 3.1 Task-level In-context Examples: The task-level strategy can cover all terms in the example input with two examples.
  • 3.2 Example-specific In-context Examples: Embedding-based retrieval may overlook rare lexicons that matter for translation in unseen medical or IT domains.
  • 3.2 Example-specific In-context Examples: BM25 retrieves informative MT examples through source n-gram overlap, but independently scored candidates may omit terms because BM25 favors rare word matches.
  • 3.2 Example-specific In-context Examples: The reranker computes recall-based n-gram overlap between the test source and retrieved example sources.
  • 3.2 Example-specific In-context Examples: After selecting the highest-scoring example, the method downweights its covered n-grams by λ before selecting subsequent prompts.
  • 3.2 Example-specific In-context Examples: The algorithm iterates over retrieved candidates until a score threshold is reached, producing ordered selected prompts.

15 Return T

The retrieval process repeats until the score threshold is reached. In the illustrated German input, reranking covers all source terms with the top two examples.

  • The reranking process repeats over the retrieved pool until the selected-example score reaches a threshold.
  • For the German example input, BM25’s top candidates repeat information about the product phrase, while lower-ranked examples cover other source terms.
  • The proposed reranking strategy covers all terms in that input with the top two examples.

4 Evaluation Settings

The evaluation uses XGLM7.5B on in-domain WMT’19 and four out-of-domain datasets, comparing random, task-level, retrieved, re-ranked, and kNN-MT approaches with sacreBLEU.

  • Datasets: In-domain evaluation covers WMT19 German↔English and Russian↔English, while out-of-domain evaluation covers Medical, Law, IT, and Koran.
  • Evaluation: Translation quality is evaluated with sacreBLEU after detokenized outputs are length-truncated, with outputs capped at twice the source length.
  • Model: Experiments use XGLM7.5B, a 32-layer decoder-only multilingual causal language model with hidden dimension 4096.
  • Baselines and comparisons: The comparisons include random, development-optimized task-level, BM25-retrieved, and re-ranked R-BM25 in-context examples.
  • Reported results: Table 2 reports WMT’19 results, where p and qmax denote the numbers of task-level and example-specific prompts.
  • Out-of-domain baseline: Out-of-domain comparisons additionally include kNN-MT using λ = 0.1, threshold = 1.0, and similarity-based example ordering.

5 Results

Results show that prompt quality, quantity, ordering, domain, and complementarity all affect MT output quality. Re-ranked example-specific prompts and their combination with task-level prompts provide the strongest reported gains, including advantages over kNN-MT in two domains.

  • In-domain evaluation: A single optimized task-level prompt is competitive with 16 random few-shot examples and improves BLEU in 2 of 4 translation directions.
  • In-domain evaluation: Using up to 16 BM25 prompts improves output quality over task-level prompts by an average of 1.41 BLEU.
  • In-domain evaluation: Re-ranking BM25-retrieved examples consistently improves BLEU over BM25 for qmax = 1 and qmax = 16, showing that example choice and ordering matter.
  • In-domain evaluation: Concatenating task-level and R-BM25 prompts improves output quality over either method independently.
  • Out-of-domain evaluation: Domain-specific task-level examples outperform examples from a distant WMT corpus by +1.4 BLEU in 1-shot and +2.7 BLEU in 16-shot settings.
  • Out-of-domain evaluation: Re-ranked example-specific prompts improve translation quality across out-of-domain settings, with up to 23 BLEU gain in Law using one example over a task-level prompt.
  • Out-of-domain evaluation: Combining task-level and R-BM25 prompts outperforms kNN-MT on 2 of 4 out-of-domain domains without dense token-level retrieval during inference.
  • Out-of-domain evaluation: In IT, combined prompts gain +0.86, while task-level prompts remain useful when retrieved training examples lack lexical overlap with test sources.

6 Analysis

Prompt selection, semantic equivalence, noise, and ordering substantially affect MT quality. Retrieved examples help when they overlap with test sources, while optimized task-level prompts are more stable and complementary to example-specific prompts.

  • Choice of Few-shot Examples: BLEU varies by up to 20 points across 100 random 1-shot prompts, showing that noisy or unrelated examples can sharply reduce translation quality.
  • Impact of Pool Size on Task-level Prompt Selection: Increasing the task-level selection pool from 100 to 1000 examples yields only a 0.5-point gain, while selected prompts reach BLEU 36 with standard deviation 0.18.
  • Properties of good Task-level prompts: Task-level prompts score higher than random 1-shot prompts on semantic-equivalence and translation-quality measures across language pairs.
  • Impact of Noise: A noisy unrelated example produces BLEU 1, and adding it to a good prompt lowers the best combined score to 31.43 from 32.14.
  • Impact of Ordering: Task-level prompts are less order-sensitive and achieve higher BLEU than randomly selected examples across all tested permutations.
  • Informativeness of Example-specific Prompts: Source lexical overlap predicts prompt informativeness in 3 of 4 domains, with Koran as the exception because target translations show lower overlap with references.

7 Output Analysis

The model can mimic stylistic and structural patterns from in-context examples, including contractions and domain-specific templates. Multiple retrieved examples can also compensate for a smaller datastore by improving source-term coverage.

  • Stylistic Outputs: A prompt containing contractions leads the model to generate contractions, which can be penalized by BLEU despite meaning-equivalent output.
  • Template-based MT: Retrieved prompts enable template-based translation synthesis in medical, legal, IT, and e-commerce domains.
  • Datastore Size and Example Count: Multiple in-context examples can achieve similar BLEU to a larger in-domain datastore by covering more source terms.

8 Related Work

Related work has studied prompting, retrieval, and domain adaptation for MT, but the impact of example properties on MT quality remains insufficiently explored. This paper addresses that gap with prompt selection for inference-time adaptation.

  • Prior work uses natural-language prompts for multilingual MT control and BM25-retrieved examples during supervised training.
  • Domain-adaptation methods improve pretrained sequence-to-sequence MT models using out-of-domain data during training or inference.
  • The impact of in-context example choice and properties on MT quality remains unexplored in prior work.

9 Conclusion

The paper studies in-context example selection for MT across in-domain and out-of-domain settings and proposes recall-based reranking of similar training examples. Task-level prompts complement example-specific prompts, while generated outputs can mimic prompt style and support template-based synthesis.

  • The study investigates in-context example selection for MT in both in-domain and out-of-domain settings.
  • The proposed recall-based reranking approach uses similar training examples as prompts and improves performance across multiple datasets and domains.
  • Task-level prompts complement example-specific prompts, outperforming a strong kNN-MT baseline on 2 of 4 out-of-domain datasets while remaining memory- and compute-efficient.
  • Manual analysis indicates that the model can mimic prompt style and synthesize template-based translations, motivating future study of diverse and style-specific MT outputs.

A Statistics of Datasets

The experiments report translation quality with BLEU and Comet across WMT’19 and multi-domain test sets, while varying prompt selection, reranking, and ordering. The reported analyses include dataset statistics, development-set tuning, and task-level prompt examples.

  • Evaluation metrics: Comet is used to evaluate translation quality because the selected model correlates better with human judgments than lexical-overlap metrics.The eamt22-cometinho-da model is also described as computationally efficient.
  • Evaluation results: The re-ranking strategy with qmax = 16 performs best across settings except Koran and beats kNN-MT in 3 out of 4 multi-domain settings.Adding a task-level prompt to 16 R-BM25 prompts further improves quality in 5 out of 8 settings.
  • Prompt ordering: Ordering the most similar retrieved examples from left to right achieves higher BLEU than the reverse ordering on the Medical development set.The reported trend may vary with training-data noise, similarity, and the number of retrieved examples.
  • Reported analyses: The paper reports BLEU for two orderings of the top-16 example-specific BM25 prompts and Comet scores on WMT’19 and multi-domain test sets.Dataset statistics are provided for the training, development, and test sets used in the experiments.
  • Development-set tuning: λ = 0.1 and threshold = 1.0 are selected because they achieve the best BLEU on the Medical development set.The corresponding analysis varies λ, the threshold, and the average number of selected in-context examples with qmax = 16.
Loading 2212.02437v1…