PaperlayerView exact revision on arXiv

Source-linked AI summary

The unreasonable effectiveness of few-shot learning for machine translation

Xavier Garcia, Yamini Bansal, Colin Cherry, George Foster, Maxim Krikun, Fangxiaoyu Feng, Melvin Johnson, Orhan Firat

arXiv:2302.01398v1cs.CL

TL;DR

Mined parallel data is impractical to obtain at scale for most languages and can overrepresent particular registers. This paper builds few-shot translation models conditioned on a few high-quality inference-time examples, reporting competitive benchmark performance and control over output register.

  • Problem

    State-of-the-art translation often depends on vast mined parallel corpora, which are impractical to obtain for most languages and can carry data-quality and register biases.

    Source: §1 · p. 1

  • Method

    They build compact few-shot translation models that use a few carefully selected, high-quality translation pairs as inference-time demonstrations, without back-translation or large-scale parallel-text mining.

    Source: §1 · pp. 1–2

  • Results

    Few-shot models outperform PaLM on the evaluated task except for German–English, and the trilingual model beats WMT’21’s best English–Chinese system while coming within 0.1 BLEURT of its best English–German system.

    Source: §2.3 · p. 4

  • Takeaways & Limitations

    Choosing demonstrations in a desired register can steer output toward that register and yield gains on register-aware translation benchmarks.

    Source: §1 · p. 2

    The models remain 8 billion parameters because preliminary 1-billion-parameter experiments were unsatisfactory, and their size may inhibit adoption.

    Source: §4 · p. 8

Abstract

from arXiv · show

We demonstrate the potential of few-shot translation systems, trained with unpaired language data, for both high and low-resource language pairs. We show that with only 5 examples of high-quality translation data shown at inference, a transformer decoder-only model trained solely with self-supervised learning, is able to match specialized supervised state-of-the-art models as well as more general commercial translation systems. In particular, we outperform the best performing system on the WMT'21 English - Chinese news translation task by only using five examples of English - Chinese parallel data at inference. Moreover, our approach in building these models does not necessitate joint multilingual training or back-translation, is conceptually simple and shows the potential to extend to the multilingual setting. Furthermore, the resulting models are two orders of magnitude smaller than state-of-the-art language models. We then analyze the factors which impact the performance of few-shot translation systems, and highlight that the quality of the few-shot demonstrations heavily determines the quality of the translations generated by our models. Finally, we show that the few-shot paradigm also provides a way to control certain attributes of the translation -- we show that we are able to control for regional varieties and formality using only a five examples at inference, paving the way towards controllable machine translation systems.

1. Introduction

Large mined parallel corpora support state-of-the-art machine translation but are impractical for most languages and can reflect data-quality and register biases. This work explores few-shot translation as a middle ground, reporting competitive results with compact models and showing that demonstrations can steer translation style.

1. Introduction

  • State-of-the-art machine translation typically relies on vast mined parallel corpora, an impractical data requirement for most language pairs.The passage also notes potential downsides of mined data, including poisoning attacks, memorization of low-quality examples, and register biases.Source: §1 · p. 1
  • Unsupervised translation has matched strong academic baselines, but commonly uses multilinguality, back-translation, or very large models and is rarely compared with state-of-the-art supervised systems.Monolingual-data quality and over-represented registers can also affect these systems.Source: §1 · p. 1
  • Few-shot learning offers a middle ground: a small set of carefully selected, high-quality translation pairs in the desired register can serve as inference-time demonstrations.This dependence on demonstrations is presented as a way to address issues associated with mined parallel data.Source: §1 · p. 1
  • Few-shot models outperform commercial baselines, are competitive with WMT’21 submissions, and show low-resource and register-control potential using 8B-parameter models without back-translation or large-scale parallel mining.The study includes WMT’21 English–German, English–Chinese, and English–Icelandic, as well as register-aware benchmarks.Source: §1 · p. 2

2. Experiments on high-resource languages & results

This section evaluates few-shot translation models on WMT’21 high-resource language pairs and extends the study to Icelandic. It describes the data, model and training choices, then compares performance with WMT and general-purpose systems.

2.1. Datasets

  • Evaluation uses recent WMT’21 news test sets to address older-set quality and possible overlap concerns; target-side overlap is checked with 15-gram matching.Source: §2.1 · p. 2

2.2. Architecture and Training Procedure

  • Models use decoder-only Transformers with a modified UL2 objective; each test source gets five randomly sampled development pairs, with BLEURT-based MBR over 64 samples.Source: §2.2 · pp. 2–4
  • Trilingual models add a third language while retaining the bilingual models’ per-language data amounts, at the cost of more training compute.Source: §2.2 · p. 3

2.3. Main results and discussion

  • Both few-shot LMs beat PaLM except German→English; the trilingual model beats top English→Chinese WMT, while English→XX excels over commercial systems and XX→English trails supervised baselines.Source: Table 4 · p. 4, §2.3 · p. 4

2.4. Performance on a low-resource language

  • Icelandic has two orders of magnitude less data than German or Chinese; both directions remain competitive with WMT, with English→Icelandic beating commercial translation.Source: Table 6 · p. 5, §2.4 · p. 5
  • After 2 epochs, Icelandic→English gains little further, while English→Icelandic improves through 6; a single pass is suboptimal.Source: §2.4 · p. 5, Figure 1 · p. 6

3. Influence of few-shot demonstrations

The section examines how exemplar quality and style influence few-shot translation, including tests of controllability for Mandarin regional varieties and German formality.

3.1. Quality of the demonstrations strongly influences quality of generated translations

  • In general, translation quality degrades as demonstrations come from buckets with increasing CDS scores.Examples are divided into three normalized CDS-score buckets, and the paper uses CDS as a proxy for example quality; Figure 2 reports BLEURT on WMT’21 English−German.Source: §3.1 · p. 6, Figure 2 · p. 6

3.2. Controllability of output language variety through few-shot demonstrations

  • Mismatched models beat PaLM-62B on FRMT score but show very low lexical accuracy; matched demonstrations sharply improve both metrics, establishing a new state of the art.The matched-setting result is reported despite models being two orders of magnitude smaller than the previous state of the art.Source: §3.2 · p. 7, Table 7 · p. 7

3.3. Controllability of formality through few-shot demonstrations

  • Few-shot models outperform all available models in translation quality and exceed UMD’s formality accuracy on the informal test set, despite five inference examples; matched exemplars also raise BLEURT over mismatched ones.UMD’s submission has higher formality accuracy than the commercial baseline, suggesting a quality-control trade-off.Source: §3.3 · p. 8, Table 8 · p. 8

4. Limitations and Future Work

The approach has limitations in model size, multilingual scaling, use of additional parallel data, and decoding cost. Future work should address these constraints while preserving the controllability of few-shot translation.

Model size

  • The resulting 8-billion-parameter models remain an order of magnitude larger than traditional machine translation systems, which usually have hundreds of millions of parameters; preliminary 1-billion-parameter experiments yielded unsatisfactory results.Source: §4 · p. 8

Multilinguality

  • Initial trilingual experiments benefited from multilinguality, but prior work reports that adding languages eventually decreases per-language performance.The authors leave open how many languages these systems can support before performance degrades.Source: §4 · p. 8

Incorporating larger amounts of high-quality parallel data

  • The approach uses five inference-time examples despite the possibility of obtaining hundreds of high-quality examples, and fine-tuning on more data might improve translation while reducing controllability.Future work should seek ways to use larger high-quality datasets while retaining few-shot flexibility.Source: §4 · p. 8

Decoding algorithm

  • The best results relied on MBR decoding rather than beam search, but this decoding approach costs more to serve and may inhibit adoption.The authors also expect PaLM's translation quality to improve with MBR, since Vilar et al. (2022) did not use it.Source: §4 · p. 8

5. Conclusion

The paper evaluates few-shot translation against strong supervised baselines on high-resource language pairs. It finds competitive performance under a high-quality-demonstration assumption, while demonstration quality affects translations and the paradigm can control style.

5. Conclusion

  • Few-shot translation systems can be competitive with supervised models when supplied with a small set of high-quality demonstrations.The evaluation examines traditionally high-resource language pairs against the strongest supervised baselines the authors could find.Source: §5 · p. 8
  • The quality of few-shot demonstrations heavily influences the quality of generated translations.Source: §5 · p. 8
  • The few-shot paradigm provides a way to control the style of generated translations.Source: §5 · p. 8

A. BLEU scores for the WMT’21 Language Pairs

The reported BLEU scores show a large gap between the WMT models and the authors' models. The authors observe the same pattern with beam search and suggest few-shot models produce qualitatively different translations.

A. BLEU scores for the WMT’21 Language Pairs

B. Model architecture

The described decoder uses 32 Transformer layers, 16 heads, a 4096-dimensional hidden state, and multi-query attention. The passage also gives feed-forward and attention-head sizes and a SacreBLEU signature.

B. Model architecture

  • The model uses 32 Transformer layers, 16 heads, and a hidden dimension of 4096.Source: B. Model architecture · p. 12
  • The architecture uses multi-query attention, a feed-forward size of 16384, and an attention head size of 256.Source: B. Model architecture · p. 12
  • The SacreBLEU signature specifies one reference, mixed case, no effective stemming, exponential smoothing, version 2.1.0, and tokenization as 13a or zh.Source: B. Model architecture · p. 12
Loading 2302.01398v1…