PaperlayerView exact revision on arXiv

Source-linked AI summary

BLOOM+1: Adding Language Support to BLOOM for Zero-Shot Prompting

Zheng-Xin Yong, Hailey Schoelkopf, Niklas Muennighoff, Alham Fikri Aji, David Ifeoluwa Adelani, Khalid Almubarak, M Saiful Bari, Lintang Sutawika, Jungo Kasai, Ahmed Baruwa, Genta Indra Winata, Stella Biderman, Edward Raff, Dragomir Radev, Vassilina Nikoulina

arXiv:2212.09535v3cs.CLcs.AIcs.LG

TL;DR

BLOOM’s 46-language pretraining leaves other languages unsupported, while retraining the model is prohibitively expensive. We adapt BLOOM for eight new languages and evaluate zero-shot prompting; adapters outperform continued pretraining for larger models, and adding languages to BLOOMZ’s instruction-tuning mixture is reported as most effective.

  • Problem

    BLOOM covers only 46 languages, and retraining the model to extend coverage is prohibitively expensive.

    Source: §1 · p. 1

  • Method

    The study benchmarks language-adaptation strategies for zero-shot NLU prompting on eight new languages in resource-constrained settings.

    Source: §1 · p. 1

  • Results

    Adapter-based strategies best adapt larger BLOOM models for prompting under low-resource settings, while including new languages during BLOOMZ instruction tuning is most effective.

    Source: §5 · p. 9

  • Takeaways & Limitations

    For BLOOM models with at least 3 billion parameters, the authors recommend adapters over continued pretraining for better prompting performance.

    Source: §1.1 · p. 2

    The experiments cover only natural-language-understanding tasks, so the findings may not generalize to generation tasks such as summarization.

    Source: §6.4 · p. 10

Abstract

from arXiv · show

The BLOOM model is a large publicly available multilingual language model, but its pretraining was limited to 46 languages. To extend the benefits of BLOOM to other languages without incurring prohibitively large costs, it is desirable to adapt BLOOM to new languages not seen during pretraining. In this work, we apply existing language adaptation strategies to BLOOM and benchmark its zero-shot prompting performance on eight new languages in a resource-constrained setting. We find language adaptation to be effective at improving zero-shot performance in new languages. Surprisingly, we find that adapter-based finetuning is more effective than continued pretraining for large models. In addition, we discover that prompting performance is not significantly affected by language specifics, such as the writing system. It is primarily determined by the size of the language adaptation data. We also add new languages to BLOOMZ, which is a multitask finetuned version of BLOOM capable of following task instructions zero-shot. We find including a new language in the multitask fine-tuning mixture to be the most effective method to teach BLOOMZ a new language. We conclude that with sufficient training data language adaptation can generalize well to diverse languages. Our code is available at https://github.com/bigscience-workshop/multilingual-modeling.

1 Introduction

BLOOM’s 176B-parameter model covers only 46 languages, while retraining it is prohibitively expensive. This work studies adapting BLOOM to eight new languages and benchmarks approaches for improving zero-shot prompting under resource constraints.

1 Introduction

  • BLOOM has 176B parameters but covers only 46 languages, and retraining it is prohibitively expensive.The authors note that even high-resource languages such as Korean and Russian are excluded.Source: §1 · p. 1
  • The study adapts BLOOM to eight new languages using at most 100K samples per language and evaluates zero-shot prompting on NLU tasks.The languages vary in script, family, and word order; the benchmark includes continued pretraining, MAD-X, and (IA)3.Source: §1 · p. 1

1.1 Our Contributions

  • The paper investigates adaptation scaling for billion-parameter models and benchmarks parameter-efficient methods, compute trade-offs, and adaptation-data size.It describes this as the first exploration of scaling effects for language adaptation under resource constraints.Source: §1.1 · p. 2
  • The authors recommend adapters over continued pretraining for BLOOM models with at least 3 billion parameters and report benefits from monolingual adaptation and adding languages to BLOOMZ’s multitask mixture.BLOOMZ is BLOOM finetuned on a multitask mixture in languages seen during pretraining.Source: §1.1 · p. 2

2 Related Work

Prior language-adaptation research has explored continued pretraining, language-specific adapters, and sparse parameter updates, often with masked language models. Work on prompting has found mixed results across settings, while multilingual prompt methods also include approaches without downstream gradient updates.

Language Adaptation

  • Language adaptation methods include continued pretraining, target-language adapters, and training a sparse subset of model parameters.These approaches aim to transfer pretrained knowledge at lower computational cost than retraining from scratch.Source: §2 · p. 2
  • Unsupported tokenizer scripts have been addressed by adding an embedding layer; prior work also found transliteration can boost non-Latin-language performance.In multilingual machine translation, adding a target language was reported to be harder than adding a source language.Source: §2 · p. 2
  • A study extending XLM-R to 30 languages on token-level classification found continued pretraining most promising, but its cost grows with model size.The present work reports that continued pretraining does not necessarily improve prompting for larger models.Source: §2 · pp. 2–3

Multilingual Prompting

  • Prompt-based learning and standard finetuning have yielded different winners across low-resource XNLI and Indonesian-dialect sentiment settings.Prompt finetuning performed better for XNLI in cited studies, whereas standard finetuning performed better for sentiment prediction.Source: §2 · p. 3
  • Multilingual prompting has also been studied through multitask prompt training and in-context learning, without downstream gradient updates.Prior work reports cross-lingual and cross-task benefits from multitask training and success with English or translated templates.Source: §2 · p. 3

3 Experimental settings

The experiments adapt BLOOM models to eight languages absent from BLOOM’s pretraining and evaluate zero-shot prompting across multilingual NLU tasks. They compare adaptation strategies, training settings, and multilingual-model baselines.

3.1 BLOOM pretrained models

  • BLOOM models from 560 million to 7.1 billion parameters, pretrained on ROOTS with 46 natural languages and 13 programming languages, form the experimental base.The models use a decoder-only Transformer and byte-level BPE tokenizer.Source: §3.1 · p. 3

3.2 New Languages

  • The eight new languages are German, Bulgarian, Russian, Greek, Turkish, Thai, Korean, and Guarani, spanning different families and including scripts not shared with BLOOM’s supported languages.The set includes six XNLI languages unsupported by BLOOM, Korean for comparison with prior work, and low-resource Guarani.Source: §3.2 · p. 3

3.3 Language Adaptation Strategies

  • The study compares continued pretraining, MAD-X adapters, and (IA)3; the latter rescales Transformer activations through learnable vectors.MAD-X uses language and invertible adapters, with the invertible adapter addressing mismatch between original and new language vocabularies.Source: §3.3 · p. 3

3.4 Language Adaptation Setting

  • Adaptation uses up to 100K monolingual samples and 25K training steps; Guarani uses 30K sentences from parallel corpora because OSCAR has around 100 samples.The tokenizer is not retrained; embeddings are trainable for continued pretraining and frozen with invertible adapters for MAD-X and (IA)3.Source: §3.4 · p. 4

3.5 Tasks and Prompt Templates

  • Zero-shot prompting covers five multilingual NLU tasks across inference, commonsense reasoning, anaphora resolution, and paraphrasing, reusing translated XGLM templates without task-specific finetuning.Templates are translated using automatic translation APIs.Source: §3.5 · p. 4

3.6 Baselines

  • The study compares adapted BLOOM with reported state-of-the-art-prompting multilingual generative models and reports original BLOOM without adaptation; XGLM, mGPT, and mT0 have seen all new languages except Guarani during pretraining.Source: §3.6 · p. 4

4 Results and Discussion

Language adaptation improves zero-shot prompting across unseen languages, but the strongest strategy depends on model size. The results also show that adaptation data, adapter capacity, and retaining instruction-tuned behavior shape performance.

4.1 Zero-shot Prompting Performance

  • Beyond 3B parameters, adapters outperform continued pretraining; at 560M, continued pretraining performs best, while adapted BLOOM matches mGPT on several XNLI tasks and beats XGLM and mT0 on German PAWS-X and Russian XWinograd.MAD-X gives better average prompting, while (IA)3 has a slight training-efficiency edge for larger models; BLOOM’s results also vary by task and do not depend on script or language family.Source: §4.1 · pp. 5–6

4.2 Perplexity

  • Lower Russian held-out perplexity does not guarantee stronger prompting: larger continued-pretraining models underperform adapter counterparts on XWinograd and XNLI.Perplexity decreases with model size, even as XWinograd performance drops for both strategies from 1.1B to 1.7B parameters.Source: §4.2 · p. 6

4.3 Connection to Language Independent Representation

  • Russian sentence-retrieval accuracy rises sharply after adaptation, but continued pretraining peaks at 560M while MAD-X retrieval improves with model size.The authors hypothesize that continued pretraining on limited language data can move larger models’ representations farther from English.Source: §4.3 · p. 7

4.4 Amount of Language Adaptation Data

  • Prompting performance correlates positively with adaptation-data size; below 100K samples, Russian XNLI and Turkish XCOPA can fall below non-adapted BLOOM, while other tasks are less affected.The authors estimate that effective adaptation needs around 100 million tokens of the new language.Source: §4.4 · pp. 7–8

4.5 Adapters’ Capacity

  • More MAD-X adapter parameters correlate with better prompting performance, achieved by lowering the adapter bottleneck’s reduction factor.A smaller reduction factor means a larger adapter parameter count.Source: §4.5 · p. 8

4.6.1 Adding Language Support through Unlabeled Data

  • On German XNLI, adapting BLOOMZ with monolingual OSCAR data reduces median accuracy from about 38.5% to 33%, while using BLOOM’s language adapters preserves prompting ability.The authors suggest language adaptation on free-form monolingual text can erase prompting ability gained through multitask instruction tuning.Source: §4.6.1 · p. 8

4.6.2 Adding Language Support through Instruction Tuning

  • Adding Russian to the full xP3 instruction-tuning mixture improves the best prompt on XNLI and XStoryCloze, whereas Russian-only finetuning yields only tiny XStoryCloze gains.The authors conclude that new-language instruction tuning requires diverse tasks in other languages.Source: §4.6.2 · p. 9

5 Conclusion

The paper compares language-adaptation strategies for extending BLOOM to new languages and finds that adapter-based methods work best for larger models under resource constraints. It also finds that including new languages in BLOOMZ’s multitask finetuning is effective.

5 Conclusion

  • Adapter-based strategies best adapt larger BLOOM models for prompting under low-resource settings, while including new languages during BLOOMZ instruction tuning is most effective.The paper also investigates adaptation-data size and adapter capacity, and relates adaptation to instruction tuning.Source: §5 · p. 9

6 Limitations

The limitations identify gaps in vocabulary adaptation, low-resource language coverage, and evaluation scope. They also qualify the experimental settings and the range of adaptation strategies examined.

6.1 Vocabulary and Embedding Adaptation

  • The study did not explore vocabulary or embedding adaptation, although byte-level tokenization handles unseen scripts and their tokenization may be suboptimal.The paper identifies token adaptation’s quality effects and initialization of new-token embeddings as future research questions.Source: §6.1 · p. 9

6.2 Parameter-Efficient Finetuning Strategies

  • Computational constraints limited the parameter-efficient methods considered, leaving prompt tuning, ladder side-tuning, and combinations with adapters for future study.The authors note that combining methods may improve prompting performance.Source: §6.2 · p. 9

6.3 Low-Resource Languages

  • Only Guarani represents a truly low-resource language in the study, and such languages may lack the data needed for effective adaptation.The paper cites 100 million tokens as needed for effective adaptation to prompt in a new language.Source: §6.3 · pp. 9–10

6.4 Generative Tasks

  • Because evaluation covers only natural language understanding, the findings may not generalize to generation; monolingual adaptation can also cause catastrophic forgetting of seen languages.The paper therefore cautions that adapted models are unsuitable for multilingual generative tasks such as machine translation.Source: §6.4 · p. 10

6.5 Experimental Settings

  • The experiments used sequence length 1024 rather than 2048, but evaluation tasks were no longer than 1024 tokens and post-hoc 2048-token results aligned with the findings.The paper also did not adapt the 176-billion-parameter BLOOM or BLOOMZ models because of prohibitive computational costs.Source: §6.5 · p. 10

A Authors’ Contributions

  • Zheng-Xin Yong led the project, coordinated resources and experiments, and wrote most of the paper; contributors also implemented methods and ran adaptation and evaluation experiments.The author-contribution section assigns additional roles in advising, conceptualization, and BLOOMZ experiments.Source: A Authors’ Contributions · pp. 14–15

B How does Language Independent Representation changes with Model Sizes

  • For known languages, intermediate layers achieve almost perfect sentence retrieval accuracy across model sizes, unlike the first and last layers and some very low-resource languages.The study reports no significant retrieval-accuracy differences across model sizes for languages observed during pretraining.Source: B How does Language Independent… · p. 15

C Batch Sizes

  • Batch size 8 gives the best performance-compute trade-off; performance rises quickly up to 8 and more slowly thereafter.Source: C Batch Sizes · p. 15

D Composable Sparse-Finetuning

The C-SFT experiments examine sparse language-specific parameter finetuning for prompting. Longer training improved German XNLI performance, but computational costs constrained further experiments.

D Composable Sparse-Finetuning

  • C-SFT finetunes language-specific and task-specific sparse parameter masks; this study trained only the language-specific mask and froze output embeddings and layer normalization.Source: D Composable Sparse-Finetuning · p. 15
  • Initial C-SFT experiments improved sentence retrieval but did not enable prompting; C-SFT also required full-model finetuning and twice the GPU RAM of continued pretraining.Longer training steps improved prompting performance, but computational constraints prevented further experiments.Source: D Composable Sparse-Finetuning · p. 15
  • German XNLI prompting performance improved from 33.01% to 35.97% with 50K total C-SFT language-adaptation steps.Source: D Composable Sparse-Finetuning · p. 16
  • The authors stopped further C-SFT experiments because of computational constraints.Source: D Composable Sparse-Finetuning · p. 16

E Korean PAWS-X

Models perform poorly on Korean PAWS-X, and the authors judge its prompt template ineffective; on German XNLI, MAD-X adapters yield the best prompting performance among tested parameter-efficient methods.

E Korean PAWS-X

  • All models perform poorly on Korean PAWS-X against the 50% random-classifier baseline, and the authors judge its prompt template ineffective.XGLM also performs poorly on Korean PAWS-X with English templates.Source: E Korean PAWS-X · p. 16, Figure 13 · p. 17

F Prompt Templates

G Other Parameter-Efficient Finetuning Strategies

H Language Adaptation Experimental Setup Details

Adapter placement and training choices shape the reported adaptation results: last-layer adapters benefit most, while the study used constrained training and hyperparameter-search settings.

H Language Adaptation Experimental Setup Details

  • Training used 25,000 steps and around 204 million tokens; compute constraints limited runs to one per adaptation and precluded extensive hyperparameter search.Different hyperparameter sets caused around 1∼2% difference in XNLI accuracy.Source: H Language Adaptation Experimental… · p. 18

I Number of Tokens for Language Adaptation Data

J Placement of Adapters

K Ablations

The ablations indicate that adaptation gains depend mainly on Transformer-block adaptation and on transferring knowledge from a pretrained model.

K Ablations

  • Invertible adapters improve NLI prompting only for German, Bulgarian, and Turkish; adaptation gains mainly result from adapting Transformer blocks.Source: K Ablations · p. 19
  • Without pretraining, adapted BLOOM behaves like a random classifier on XNLI, supporting the role of knowledge transfer from pretrained models.Source: Figure 17 · p. 19, K Ablations · p. 19

L Catastrophic Forgetting

Continued pretraining leads to catastrophic forgetting on English XNLI across model sizes; Table 7 describes natural- and programming-language distributions in BLOOM’s ROOTS pretraining data.

L Catastrophic Forgetting

M Pretraining Languages Existing in BLOOM

N Post-Hoc Experiments

With a sequence length of 2048, adapter-based strategies still outperform continued pretraining for Thai and Greek. The artifacts section records licenses for models and datasets and identifies a table of BLOOM-seen languages.

O Artifacts

N Post-Hoc Experiments

  • With sequence length 2048, adapter-based strategies still outperform continued pretraining for Thai and Greek.The BLOOM-7.1B adaptations used 12,500 steps, halving the steps to preserve the number of tokens seen when sequence length doubled.Source: N Post-Hoc Experiments · p. 19, Figure 19 · p. 20

O Artifacts

  • OSCAR adaptation data are CC0 1.0 Universal, while Guarani data are MIT-licensed.Source: O Artifacts · p. 20
  • The listed evaluation datasets have varied licenses, from Attribution-NonCommercial 4.0 for XNLI to free use for any purpose for PAWS-X.KLUE-NLI and AmericasNLI use Attribution-ShareAlike 4.0; XCOPA uses Attribution 4.0; XStoryCloze uses MIT.Source: O Artifacts · p. 20
  • Table 7 provides information about languages seen during BLOOM pretraining.Source: Table 7 · p. 20

Related papers

Loading 2212.09535v3…