PaperlayerView exact revision on arXiv

Source-linked AI summary

Multilingual Machine Translation with Large Language Models: Empirical Results and Analysis

Wenhao Zhu, Hongyi Liu, Qingxiu Dong, Jingjing Xu, Shujian Huang, Lingpeng Kong, Jiajun Chen, Lei Li

arXiv:2304.04675v4cs.CL

TL;DR

The paper addresses limited evidence about LLMs’ multilingual translation performance and the factors shaping their translation ability. It evaluates eight popular LLMs across 102 languages and 606 directions, compares them with supervised and commercial systems, and analyzes in-context learning patterns. GPT-4 surpasses NLLB on 40.91% of English-centric directions but still trails Google Translate, while resource-efficient acquisition and cross-lingual exemplars emerge as notable patterns.

  • Problem

    LLMs’ multilingual translation ability and the factors affecting how they acquire translation ability remain under-explored.

  • Method

    The paper evaluates eight popular LLMs across 102 languages and 606 translation directions, compares them with supervised and commercial baselines, and analyzes in-context learning.

  • Results

    GPT-4 achieves higher performance than NLLB on 40.91% of English-centric translation directions but remains behind Google Translate, particularly for low-resource languages.

  • Takeaways & Limitations

    LLMs can acquire translation ability resource-efficiently, tolerate unreasonable instructions when exemplars are provided, and benefit from cross-lingual exemplars for low-resource translation.

  • Takeaways & Limitations

    The evaluation mainly covers English-centric, French-centric, and Chinese-centric translation, leaving directions such as Russian-centric and Arabic-centric translation for future work.

Abstract

from arXiv · show

Large language models (LLMs) have demonstrated remarkable potential in handling multilingual machine translation (MMT). In this paper, we systematically investigate the advantages and challenges of LLMs for MMT by answering two questions: 1) How well do LLMs perform in translating massive languages? 2) Which factors affect LLMs' performance in translation? We thoroughly evaluate eight popular LLMs, including ChatGPT and GPT-4. Our empirical results show that translation capabilities of LLMs are continually involving. GPT-4 has beat the strong supervised baseline NLLB in 40.91% of translation directions but still faces a large gap towards the commercial translation system like Google Translate, especially on low-resource languages. Through further analysis, we discover that LLMs exhibit new working patterns when used for MMT. First, LLM can acquire translation ability in a resource-efficient way and generate moderate translation even on zero-resource languages. Second, instruction semantics can surprisingly be ignored when given in-context exemplars. Third, cross-lingual exemplars can provide better task guidance for low-resource translation than exemplars in the same language pairs. Code will be released at: https://github.com/NJUNLP/MMT-LLM.

1 Introduction

This paper investigates how well large language models perform multilingual machine translation across many languages and which factors affect their performance. It benchmarks LLMs against supervised systems and analyzes new in-context learning patterns for translation.

  • Main findings: GPT-4 exceeds NLLB on 40.91% of English-centric translation directions but remains behind Google Translate, especially for low-resource languages.French-centric and Chinese-centric translation are more challenging for GPT-4 than English-centric translation, indicating unbalanced capability across languages.
  • In-context learning patterns: LLMs can acquire translation ability resource-efficiently and produce moderate translations even for zero-resource languages.This is identified as a new working pattern of LLMs in multilingual machine translation.
  • In-context learning patterns: In-context exemplars can override unreasonable instruction semantics, while mismatched translation pairs cause LLMs to fail to translate.These findings emphasize the importance of exemplars in in-context learning for machine translation.
  • In-context learning patterns: Cross-lingual translation pairs can guide low-resource translation better than exemplars from the same language.The paper presents this as a surprising advantage of cross-lingual exemplars for multilingual translation.
  • Evaluation scope: The study benchmarks popular LLMs on multilingual translation across 102 languages and 606 translation directions.The directions cover English-centric, French-centric, and Chinese-centric translation.
  • Evaluation scope: It systematically compares LLMs with M2M-100, NLLB, and Google Translate to characterize differences between translation paradigms.The comparison examines both supervised baselines and a commercial translation system.

2 Background

The paper frames LLMs as universal multitask learners whose in-context learning enables task adaptation without parameter updates. It defines prompts as templates wrapped around input-output exemplars, with sequence-generation outputs produced through decoding strategies.

  • LLMs can acquire multiple tasks from unsupervised training, with scaling associated with stronger performance and emergent abilities such as in-context learning.
  • In-context learning adapts an LLM to a target task from a prompt without updating model parameters.
  • A prompt combines an in-context template with k input-output exemplars, where each target output corresponds to its input.
  • During inference, the prompt guides generation of the output for a test input.
  • For machine translation, sequence outputs can be generated using strategies such as greedy search or beam search.

3 Experiment Setup

The experiment benchmarks eight LLMs for multilingual translation on FLORES-101, using in-context examples and comparing them with supervised and commercial systems. Performance is measured primarily with SentencePiece BLEU, alongside COMET and SEScore.

  • The benchmark uses FLORES-101 to assess translation quality across a wide range of languages.
  • Eight LLMs are evaluated, including XGLM, OPT, BLOOMZ, Falcon, LLaMA2 variants, ChatGPT, and GPT-4.
  • The benchmark evaluates the first 100 sentences of each direction’s test set because evaluating massive languages through APIs is costly.
  • Each model receives eight randomly selected development-set translation pairs as in-context exemplars with an “<X>=<Y>” template.
  • The comparisons include M2M-100-12B, NLLB-1.3B, and Google Translate as supervised and commercial baselines.
  • SentencePiece BLEU is the primary metric, supplemented by COMET and SEScore, which are reported to correlate with human judgments.

4 Benchmarking LLMs for Massively Multilingual Machine Translation

LLM translation quality improves across model generations, with GPT-4 strongest among evaluated LLMs but still weaker than established systems on long-tail languages. Performance is uneven across language families and can be distorted by public-dataset leakage.

  • GPT-4 achieves the highest average BLEU and COMET scores on most directions and surpasses ChatGPT and earlier open-source LLMs.
  • LLMs generally translate into English better than into non-English languages, with stronger results for English-similar than English-dissimilar languages.
  • GPT-4 faces greater difficulty on French-centric and Chinese-centric translation than on English-centric translation.
  • 40.91% of translation directions show GPT-4 achieving higher BLEU scores than NLLB, while it still trails NLLB and Google Translate on long-tail low-resource languages.
  • BLOOMZ is excluded from FLORES-101 results because instruction tuning on XP3 may have exposed it to FLORES test cases.
  • On the no-leakage NEWS2023 set, BLOOMZ deteriorates significantly while other models maintain consistent performance across datasets.

5 Analyzing Factors That Influence LLM’s Translation Performance

The analysis examines how pre-training resources, prompt templates, exemplar content, selection, and prompt position influence LLM translation. It finds resource-efficient transfer, strong template sensitivity, and distinct effects from exemplar language, consistency, quantity, and placement.

  • 5.1 Findings on Pre-training Corpus Size: XGLM generates moderate translation for low-resource languages with less than 1% of English resources and translates unseen languages through in-context learning.The analysis uses Catalan, Swahili, Occitan, and Asturian as examples.
  • 5.2 Findings on In-context Template: A 16-BLEU gap separates the best and worst average template performance, with “<X>=<Y>” achieving the highest average BLEU.The best template differs by translation direction, while “[TGT]” formatting achieves the lowest score despite use in other prompting settings.
  • 5.2 Findings on In-context Template: Task-unrelated templates can still produce translation when exemplars are present, showing that instruction semantics may be ignored during in-context learning.For example, “<X> can be summarized as <Y>” can elicit translation rather than summarization.
  • 5.3 Findings on In-context Exemplar: Cross-lingual exemplars hurt German-English translation but usually improve low-resource Chinese-English translation, even when both exemplar languages change.The authors identify translation direction as a distinctive factor in exemplar usefulness.
  • 5.3 Findings on In-context Exemplar: BLEU rises rapidly from 1 to 8 exemplars, plateaus afterward, and usually declines with 32 exemplars regardless of selection strategy.Random, BM25, TopK, and Oracle are the compared selection approaches; semantically related or oracle-selected exemplars do not consistently outperform random selection.
  • 5.3 Findings on In-context Exemplar: Mismatched, word-level, document-level, or duplicated exemplars degrade translation, indicating that semantic consistency, sentence-level granularity, and diversity matter.Mismatched translation pairs cause complete failure, while duplicated exemplars also reduce performance.
  • 5.3 Findings on In-context Exemplar: Reversing exemplars in the prompt tail consistently performs worse than reversing exemplars in the head, suggesting greater influence from tail exemplars.Reversing translation direction causes failure, motivating the prompt-position experiment.

6 Related Work

Related work establishes LLM evaluation for multilingual translation, exemplar-based in-context learning, and the broader challenge of building efficient multilingual systems. This paper extends that literature with broader evaluation and analysis of exemplar effects.

  • In-context learning for machine translation: Prior studies evaluated GPT-3, XGLM, BLOOM, and ChatGPT on narrower multilingual translation settings, whereas this paper evaluates popular LLMs across 102 languages and 606 directions.The cited prior evaluations covered 6 to 182 translation directions.
  • In-context learning for machine translation: Exemplar-selection research compares strategies such as random, BM25, TopK, and Oracle selection to improve machine translation in-context learning.The paper reports that random selection is simple and effective, while oracle selection is not consistently better.
  • Multilingual machine translation: Multilingual machine translation addresses the impracticality of building a separate bilingual system for every direction, but efficient high-quality MMT remains challenging.The paper focuses on LLMs to examine their potential for MMT.

7 Conclusion

The paper evaluates LLM multilingual translation across 102 languages and 606 directions, then analyzes factors shaping performance. GPT-4 reaches new performance levels but remains challenged on low-resource languages, while LLMs show resource-efficient learning and unusual exemplar effects.

  • 7 Conclusion: GPT-4 reaches new performance height but still faces challenges on low-resource languages.The conclusion also highlights instruction-semantic insensitivity and the usefulness of cross-lingual exemplars for low-resource translation.
  • 7 Conclusion: LLMs can acquire translation ability in a resource-efficient way, indicating potential for multilingual machine translation.This conclusion is presented alongside the observed in-context learning patterns.

Limitations

The study mainly evaluates English-centric, French-centric, and Chinese-centric translation, leaving other translation-center configurations for future investigation.

  • Limitations: The evaluation is limited primarily to English-centric, French-centric, and Chinese-centric translation directions.The authors propose examining Russian-centric and Arabic-centric translation in future work.

A Detailed Results on Each Language

The paper reports detailed evaluation results across 102 languages using BLEU, COMET, SEScore, and Figure 8. BLEU covers all translation directions, while COMET and SEScore cover only subsets.

  • Detailed results are provided in Tables 7–9 and Figure 8 for BLEU, COMET, SEScore, and language-family comparisons.
  • BLEU results cover all translation directions, whereas COMET and SEScore support only subsets.
  • 102 languages are evaluated, with names, ISO codes, and language families listed in Table 10.

C Cross-lingual Exemplars

The paper illustrates cross-lingual in-context exemplars by using Russian-English examples to guide Chinese-English translation.

  • Cross-lingual exemplars use Russian-English examples for Chinese-English translation.

D Used Scientific Artifacts

The work uses scientific artifacts for in-context learning, pretrained multimodal models, and multilingual evaluation materials. It also documents example translation pairs and detailed results across languages and metrics.

  • OpenICL provides an interface for in-context learning, while Transformers provides pretrained models for text, vision, and audio tasks.
  • The artifacts include Russian-English and Chinese-English example translations used in the work.
  • Tables 7–9 and Figure 8 report detailed BLEU, COMET, and SEScore results, while Table 10 lists language names, ISO codes, and families.
Loading 2304.04675v4…