PaperlayerView exact revision on arXiv

Source-linked AI summary

BigTranslate: Augmenting Large Language Models with Multilingual Translation Capability over 100 Languages

Wen Yang, Chong Li, Jiajun Zhang, Chengqing Zong

arXiv:2305.18098v3cs.CL

TL;DR

Many LLMs support only dozens of languages, limiting the reach of their translation capabilities. BigTranslate adapts LLaMA through Chinese and multilingual parallel-data training, followed by translation instruction tuning. It performs comparably with Google Translate and ChatGPT in many languages and surpasses ChatGPT in eight languages in GPT-4 evaluation.

  • Problem

    Many existing LLMs are English-dominant and support only dozens of languages, while more than 7,000 natural languages exist.

    Source: §1 · p. 1

  • Method

    BigTranslate adapts LLaMA-13B with Chinese monolingual training, a 102-language parallel corpus, and multilingual translation instruction tuning.

    Source: §3.1 · p. 2, §5 · p. 9

  • Results

    BigTranslate performs comparably with Google Translate and ChatGPT in many languages and surpasses ChatGPT in 8 languages under GPT-4 evaluation.

    Source: §5 · p. 9

  • Takeaways & Limitations

    BigTranslate extends an LLM’s multilingual translation capability to over 100 natural languages while retaining its basis as an LLM.

    Source: §5 · p. 9

    Unbalanced data leaves BigTranslate weak in dozens of languages.

    Source: §5 · p. 9

Abstract

from arXiv · show

Large language models (LLMs) demonstrate promising translation performance among various natural languages. However, many LLMs especially the open-sourced ones, such as BLOOM and LLaMA, are English-dominant and support only dozens of natural languages, making the potential of LLMs on language translation less explored. In this work, we present BigTranslate which adapts LLaMA that covers only 20 languages and enhances it with multilingual translation capability on more than 100 languages. BigTranslate is built upon LLaMA-13B and it is optimized in three steps. First, we continue training LLaMA with massive Chinese monolingual data. Second, we continue training the model with a large-scale parallel dataset that covers 102 natural languages. Third, we instruct-tune the foundation model with multilingual translation instructions, leading to our BigTranslate model. The preliminary experiments on multilingual translation show that BigTranslate performs comparably with ChatGPT and Google Translate in many languages and even outperforms ChatGPT in 8 language pairs. We release the BigTranslate model and hope it can advance the research progress.

1 Introduction

BigTranslate adapts LLaMA to address the limited language coverage of many existing LLMs, using Chinese and multilingual training followed by translation instruction tuning. Preliminary experiments report performance comparable with Google Translate and ChatGPT in many languages, with wins over ChatGPT in eight language pairs.

1 Introduction

  • Many existing LLMs are English-dominant and cover only dozens of languages; LLaMA supports 20, while more than 7,000 natural languages exist.The passage notes that this leaves a large population unable to benefit from LLM multilingual capability.Source: §1 · p. 1
  • The three-step construction continues LLaMA-13B training on Chinese texts, then a 102-language parallel dataset, and finally multilingual translation instructions.The parallel-data stage uses curriculum learning.Source: §1 · p. 1
  • BigTranslate performs comparably with Google Translate and ChatGPT in many languages and outperforms ChatGPT in 8 language pairs.The authors describe these multilingual translation experiments as preliminary.Source: §1 · p. 2

2 Related Work

Related work describes strong translation abilities in LLMs alongside limited language coverage, and distinguishes LLM adaptation from multilingual translation systems built solely for translation. BigTranslate is presented as an LLM-based approach supporting more than 100 languages.

2.1 Large Language Models

  • LLMs such as ChatGPT and GPT-4 perform well in several translation settings, but most existing LLMs cover only dozens of languages.Prior work reports strong results particularly for high-resource and spoken-language translation, document translation, and some multilingual translation settings.Source: §2.1 · p. 2
  • BigTranslate is presented as an LLM approach that supports more than 100 languages.The related-work discussion contrasts this language coverage with that of most existing LLMs.Source: §2.1 · p. 2

2.2 Multilingual Neural Machine Translation

  • Unlike multilingual translation systems that only translate, BigTranslate adapts an LLM while seeking to retain its generic ability.The passage notes NLLB covers more than 200 languages but can only perform translation.Source: §2.2 · p. 2

3 BigTranslate Construction

BigTranslate starts from LLaMA-13B and expands its Chinese and multilingual capabilities through continued training on Chinese text and a 102-language parallel corpus. Its construction also addresses uneven data resources through dataset balancing and incremental training, followed by multilingual instruction tuning.

3.1 LLaMA as Foundation Model

  • LLaMA-13B is the foundation model, selected for its performance on English benchmarks and continued training on Chinese and parallel corpora.The stated motivation for the foundation-model choice includes reducing computational cost.Source: §3.1 · p. 2

3.2 Augmenting Foundation Model with Chinese

  • Chinese vocabulary expansion and large-scale Chinese pre-training produce Chinese LLaMA, addressing its limited Chinese understanding and generation.The model’s vocabulary is extended with 6,223 Chinese tokens, and training uses CLUE, Chinese News, and Chinese question-answering data.Source: §3.2 · pp. 2–3

3.3 Augmenting Foundation Model with 102 languages

  • The 102-language parallel-training stage uses incremental sampling to address uneven learning between high- and low-resource languages.The authors describe this strategy as training on a harmonized multilingual parallel corpus at each stage.Source: §3.3 · p. 3

3.3.1 Large-scale Parallel Dataset Construction

  • The parallel corpus draws on public datasets including IWSLT, WMT, CCMT, and OPUS-100, and augmentation expands 142 translation directions to 242.For pairs with fewer than 1 million examples, the corpus is flipped in full; for larger corpora, half is flipped to add the reverse direction.Source: §3.3.1 · p. 3

3.3.2 Tokenizer

  • The multilingual tokenizer extends LLaMA’s 32,000-token vocabulary to 53,613 tokens, with added tokens trained on text spanning 102 languages.The vocabulary training samples up to 1,000,000 instances per language to address vocabulary skew.Source: §3.3.2 · pp. 3–4, Figure 1 · p. 4

3.3.3 Incremental Multilingual Pre-training

  • Incremental pre-training sorts language-pair intervals by sample mean and progressively adds lower-resource intervals as the untrained data in a current interval falls to the next interval’s mean.The approach begins with high-resource pairs and progressively introduces low-resource pairs; the stated interval sizes are 10,000 above 10,000 instances and 5,000 below that threshold.Source: §3.3.3 · pp. 4–6

3.4 Multilingual Translation Instruction Tuning

The instruction-tuning stage constructs multilingual translation instructions for 102 languages and fine-tunes BigTranslate to activate its multilingual machine-translation ability. It uses parallel sentence pairs and a set of translation prompts.

3.4 Multilingual Translation Instruction Tuning

  • The instruction-tuning dataset covers 102 languages and 242 language pairs, and is used to fine-tune BigTranslate for multilingual machine translation.It contains 241,128 parallel sentence pairs, selecting up to 1,000 from each pair’s training set.Source: §3.4 · p. 6
  • Each parallel sentence is paired with a randomly selected prompt from a set of 28 multilingual translation prompts.The resulting instruction-tuning data is scrambled for randomness and diversity.Source: §3.4 · p. 6

3.5 Training Details

BigTranslate is trained through Chinese and multilingual pre-training, then fine-tuned on translation instructions; inference uses beam search.

3.5 Training Details

  • The model is pre-trained on 42.8B Chinese tokens and 47.0B multilingual tokens, then fine-tuned on 240k multilingual translation instructions for 3 epochs.Pre-training uses a 5e-5 learning rate and batch size 65,536; instruction fine-tuning uses batch size 32 and learning rate 2e-5.Source: §3.5 · p. 6
  • Inference uses beam search with a beam size of 5.Source: §3.5 · p. 6
  • Training runs on 16 A100 GPUs with 80GB of RAM, using DeepSpeed, FlashAttention, and Gradient Checkpointing.The authors report these tools for data parallelization and memory savings.Source: §3.5 · p. 6

4 Experiments

Experiments compare BigTranslate with ChatGPT and Google Translate on translations into English or Chinese. GPT-4 evaluation finds it comparable to ChatGPT in many languages and ahead in eight; BLEU comparisons show a different pattern against Google Translate.

4 Experiments

  • The experiments compare BigTranslate, ChatGPT, and Google Translate on translations from 102 languages into English or Chinese.Source: §4 · pp. 6, 8

4.2 Main Results

  • By BLEU, BigTranslate performs on par with ChatGPT in many languages and exceeds it in 9 language pairs, but lags Google Translate in most pairs.The authors caution that BLEU has low correlation with human preference, particularly above a score threshold such as 20.0.Source: Figure 3 · p. 7, §4.2 · p. 8
  • Under GPT-4, BigTranslate scores above 3.5 in 28 of 70 language pairs, matches ChatGPT within 0.3 points in 27, and outperforms it in 8.Source: Figure 4 · p. 7, §4.2 · p. 9

4.1 Datasets and Evaluation Metrics

  • Evaluation uses 50 FLORES-200 devtest sentences per direction and evaluates only translations into English or Chinese.The authors limit directions because they cannot well evaluate languages other than English and Chinese.Source: §4.2 · p. 8, §4.1 · p. 8

4.3 Discussion

  • BigTranslate supports translation across more than 100 languages, but the authors say it still needs improvement for extremely low-resource languages.They also describe transferring English and Chinese NLP abilities to languages including Tibetan and Mongolian.Source: §4.3 · p. 9

5 Conclusion and Future Work

The conclusion presents BigTranslate as a multilingual translator built by extending LLaMA through Chinese and multilingual continued training followed by instruction tuning. It reports comparable performance to Google Translate and ChatGPT in many languages, while identifying data imbalance as a remaining weakness.

5 Conclusion and Future Work

  • The three-stage process extends LLaMA with Chinese and 102-language parallel-data training, followed by multilingual translation instruction tuning.Source: §5 · p. 9
  • With GPT-4 evaluation, BigTranslate performs comparably to Google Translate and ChatGPT in many languages and surpasses ChatGPT in 8 languages.Source: §5 · p. 9
  • Data imbalance leaves BigTranslate weak in dozens of languages; future work aims to improve low-resource language ability and performance on other NLP tasks.Source: §5 · p. 9

A GPT-4 prompt for Evaluating Machine Translation

The GPT-4 prompt rates translation quality by comparing translated and reference sentences, emphasizing semantic closeness and consistency of style.

A GPT-4 prompt for Evaluating Machine Translation

B Detailed Results on 102 Languages with BLEU

Table 1 presents detailed BLEU results for the evaluated models across 102 languages.

B Detailed Results on 102 Languages with BLEU

C Detailed Results on 70 Languages with GPT-4 Evaluation

Table 2 presents detailed GPT-4 evaluation scores for the evaluated models across 70 languages.

C Detailed Results on 70 Languages with GPT-4 Evaluation

D The corresponding table of the language code and its name

Table 3 lists the language codes used in the paper alongside their corresponding language names.

D The corresponding table of the language code and its name

Related papers

Loading 2305.18098v3…