PaperlayerView exact revision on arXiv

Source-linked AI summary

LLaMA Beyond English: An Empirical Study on Language Capability Transfer

Jun Zhao, Zhihao Zhang, Luhui Gao, Qi Zhang, Tao Gui, Xuanjing Huang

arXiv:2401.01055v2cs.CLcs.AI

TL;DR

English-dominant pretraining and unequal language resources leave many non-English languages underrepresented in LLMs. This study empirically examines transferring LLaMA’s generation and instruction-following capabilities, reporting comparable performance to Open Chinese LLaMA with less than 1% of the training data and similar trends across 13 low-resource languages.

Abstract

from arXiv · show

In recent times, substantial advancements have been witnessed in large language models (LLMs), exemplified by ChatGPT, showcasing remarkable proficiency across a range of complex tasks. However, many mainstream LLMs (e.g. LLaMA) are pretrained on English-dominant corpus, which limits their performance in other non-English languages. In this paper, we focus on how to effectively transfer the capabilities of language generation and following instructions to a non-English language. To answer this question, we conduct an extensive empirical investigation based on LLaMA, accumulating over 1440 GPU hours. We analyze the impact of key factors such as vocabulary extension, further pretraining, and instruction tuning on transfer. To accurately assess the model's level of knowledge, we employ four widely used standardized testing benchmarks: C-Eval, MMLU, AGI-Eval, and GAOKAO-Bench. Furthermore, a comprehensive evaluation of the model's response quality is conducted, considering aspects such as accuracy, fluency, informativeness, logical coherence, and harmlessness, based on LLM-Eval, a benchmarks consisting instruction tasks from 17 diverse categories. Our evaluation results demonstrate that comparable performance to state-of-the-art transfer models can be achieved with less than 1% of the pretraining data, both in terms of knowledge alignment and response quality. Furthermore, the experimental outcomes across the thirteen low-resource languages also exhibit similar trends. We anticipate that the conclusions revealed by the experiments will aid the community in developing non-English LLMs.

Introduction

The paper investigates how to transfer LLaMA’s language-generation and instruction-following capabilities to non-English languages without costly large-scale retraining. It reports comparable knowledge and response quality to Open Chinese LLaMA with less than 1% of the training data, with similar trends across 13 low-resource languages.

Introduction

  • Language resources are highly imbalanced: BLOOM’s 46 languages represent 0.66% of roughly 7,000 languages, and English text in its corpus is 2.8 million times more abundant than Chitumbuka.The introduction situates this imbalance against the emergence of LLM capabilities across complex tasks.Source: Introduction · p. 1
  • English-dominant LLaMA has inferior performance on non-English languages without sufficient training, while collecting large-scale language-specific data incurs high computational and collection costs.The paper identifies a need for fine-grained, systematic investigation of transfer.Source: Introduction · p. 2
  • The study examines vocabulary extension, further pretraining, and instruction tuning as factors in transferring LLaMA’s capabilities to non-English languages.The authors also report that Chinese-only transfer compromises English proficiency, while multilingual joint training alleviates that concern.Source: Introduction · p. 2
  • Comparable knowledge level and response quality to Open Chinese LLaMA were achieved with less than 1% of the training data, with similar trends across 13 low-resource languages.Evaluation used four standardized testing benchmarks and LLM-Eval.Source: Introduction · p. 2

Background and Overview

This section outlines how pretraining and instruction tuning contribute to LLM capabilities, then frames non-English transfer as a three-step process. The paper empirically studies vocabulary extension and training scale.

Background and Overview

  • The overview presents pretraining and instruction tuning as the essential steps for developing an instruction-following LLM.It then reviews common approaches for transferring that model to a non-English language.Source: Background and Overview · p. 2

Step 1: Pretraining to acquire language capability and knowledge

  • Pretraining predicts the next token from preceding tokens and is described as a source of language-generation capability and world knowledge.The section says massive corpora help models capture language structures, semantics, context, concepts, facts, and their connections.Source: Step 1: Pretraining to acquire… · p. 2

Step 2: Instruction tuning for aligning with human intent

  • Instruction tuning uses instruction-response pairs to teach models to answer instructions rather than merely continue preceding text.Tuning on diverse instruction tasks is described as improving instruction following and generalization to unseen instructions.Source: Step 2: Instruction tuning for… · p. 2

Extrapolating LLMs to non-English languages

  • Non-English extrapolation commonly extends the vocabulary, further-pretrains on target-language data, and applies target-language supervised fine-tuning.The paper compares models before and after vocabulary extension and across pretraining and SFT scales.Source: Extrapolating LLMs to non-English… · pp. 2–3

Experimental Setup

The experiments use Chinese as the main empirical setting and validate observed phenomena across more than ten low-resource languages. Models with varying Chinese-pretraining scales, instruction datasets, and standardized benchmarks support comparisons of response quality and knowledge.

Experimental Setup

  • Chinese is the starting point for the empirical study, with observed phenomena further validated across over ten low-resource languages.The setup introduces the datasets, models, and evaluation methodology.Source: Experimental Setup · p. 3

Models

  • The model comparisons include LLaMA and LLaMA2 checkpoints without explicit Chinese pretraining, Chinese LLaMA checkpoints trained on 30 billion Chinese tokens, and Open Chinese LLaMA trained on 100 billion tokens.The study uses open-source models at varying Chinese-corpus scales to avoid unnecessary large-scale repetitive pretraining.Source: Models · p. 3

Datasets

  • BELLE supplies Chinese instruction data and Bactrian-X supplies multilingual instructions; evaluation covers LLM-Eval and four standardized testing benchmarks.LLM-Eval has 453 instruction tasks from 17 categories, and Table 1 describes response-quality scores across further-pretraining and SFT scales.Source: Datasets · pp. 3–4, Table 1 · p. 4

Evaluation Protocol

  • LLM-Eval scores accuracy, fluency, informativeness, logicality, and harmlessness from 0 to 3 using GPT-4, while benchmark accuracy uses zero-shot or 5-shot evaluation.AGI-Eval and GAOKAO-Bench use zero-shot settings; C-Eval and MMLU use 5-shot settings.Source: Evaluation Protocol · p. 4

Main Results

The results show that vocabulary extension is not favorable at the tested training scales, while further pretraining and instruction-tuning scale affect response quality differently. Chinese-only transfer training improves Chinese perplexity but raises English perplexity; multilingual joint training alleviates that trade-off.

The Impact of Vocabulary Extension on Transfer

  • 0.5 billion tokens on the original vocabulary significantly outperformed vocabulary-extended Chinese LLaMA across 1K, 5K, and 950K instruction-tuning settings, including a 950K comparison using the same pretraining data.The authors conclude vocabulary extension is not favorable at scales of tens of billions of tokens, while leaving open its efficacy at larger scales such as trillions of tokens.Source: The Impact of Vocabulary Extension on… · p. 4

Training Scales Required for Effective Transfer

  • With 1K and 5K instruction tuning, response quality improved as further pretraining increased; at 950K instruction tuning, models showed no significant response-quality differences.The authors hypothesize that more further pretraining may accelerate instruction alignment, while tens of billions of tokens are insufficient to add more world knowledge; Chinese-LLaMA is an exception because of vocabulary extension.Source: Training Scales Required for Effective… · p. 5
  • LLaMA 7B, Chinese LLaMA 7B, and Open Chinese LLaMA 7B performed comparably on C-Eval, GAOKAO-Bench, and AGI-Eval; LLaMA2-7B and LLaMA-13B outperformed Open Chinese LLaMA on C-Eval, MMLU, and AGI-Eval.The authors suggest trillion-level pretraining and larger model sizes may enhance knowledge levels.Source: Training Scales Required for Effective… · p. 5, Figure 2 · p. 5

How about the Original English Capabilities

  • As further pretraining scale increased, Chinese perplexity decreased steadily while English perplexity notably increased.Table 2 reports model perplexity by further-pretraining scale.Source: How about the Original English… · p. 5, Table 2 · p. 5
  • Multilingual training kept both Chinese and English perplexities low, and the authors report that it alleviated the English-proficiency compromise associated with relying exclusively on Chinese corpora.Open Chinese LLaMA incorporates both Chinese and English training data.Source: How about the Original English… · p. 5

Extending the Analysis to Multiple Languages

Across 13 low-resource languages, response quality improves as SFT data increases, while code-switching appears during transfer. The examples and rates suggest that cross-language alignment may persist in LLaMA.

Extending the Analysis to Multiple Languages

  • Response quality significantly improves with more SFT data across all 13 languages, with Arabic, Indonesian, and Vietnamese performing best.The authors relate the stronger results for these three languages to LLaMA encountering them more often.Source: Extending the Analysis to Multiple… · p. 6
  • Code-switching occurs in about 2%–5% of samples across the target languages, and the mixed-language responses are described as semantically coherent.The authors suggest this may indicate cross-lingual concept alignment learned during pretraining.Source: Extending the Analysis to Multiple… · p. 6, Figure 4 · p. 7
  • A Chinese healthcare prompt elicits a response that mixes Chinese and English.Figure 3 marks Chinese target-language text in red and code-switching output in cyan.Source: Extending the Analysis to Multiple… · p. 6, Figure 3 · p. 6

Related Work

Related work frames non-English LLM performance around unequal language resources, cross-lingual transfer, and code-switching. Prior work links transfer to universal semantic abstractions, parameter sharing, and language distance.

Resource Gap in LLMs

Cross-Lingual Transfer

  • Multilingual models transfer supervised language capability across languages, with prior work associating performance with universal semantic abstractions, parameter sharing, and language distance [Wu and Dredze 2019; Pires, Schlinger, and Garrette 2019; Winata et al. 2021b; Artetxe, Ruder, and Yogatama 2020; Chi, Hewitt, and Manning 2020; Conneau et al. 2020b; Dufter and Schütze 2020; Wu, Papadimitriou, and Tamkin 2022; Eronen, Ptaszynski, and Masui 2023].The paper investigates transferability through new LLaMA-based experiments.Source: Cross-Lingual Transfer · p. 7

Code-Switching

Conclusions

The conclusion reports that vocabulary extension is unnecessary for the studied transfer setup and that comparable performance is attainable with less than 1% of the further-pretraining data. Similar trends appear in 13 low-resource languages.

Conclusions

  • Comparable transfer performance to state-of-the-art models is achieved with less than 1% of the further-pretraining data, with similar trends across 13 low-resource languages.Source: Conclusions · p. 7
  • The study finds vocabulary extension unnecessary and observes code-switching during transfer training, suggesting cross-lingual alignment may be internalized in the model.Source: Conclusions · p. 7
  • The authors present their findings as guidance for developing non-English LLMs.Source: Conclusions · p. 7

Appendix

The appendix gives LLM-Eval’s scoring criteria for judging model responses to subject-specific questions.

Appendix

  • LLM-Eval scores accuracy and fluency on a 0/1/2/3 scale, assessing correctness and clear, convention-following language.Source: Appendix · p. 9
  • Informativeness and logicality assess relevant practical content and logically sound presentation of viewpoints and arguments.Source: Appendix · p. 10
  • Harmlessness assesses avoidance of unethical or harmful content; unanswered responses receive zero on other criteria but not harmlessness.Source: Appendix · p. 10

Related papers

Loading 2401.01055v2…