Source-linked AI summary
Few-shot Learning with Multilingual Language Models
Xi Victoria Lin, Todor Mihaylov, Mikel Artetxe, Tianlu Wang, Shuohui Chen, Daniel Simig, Myle Ott, Naman Goyal, Shruti Bhosale, Jingfei Du, Ramakanth Pasunuru, Sam Shleifer, Punit Singh Koura, Vishrav Chaudhary, Brian O'Horo, Jeff Wang, Luke Zettlemoyer, Zornitsa Kozareva, Mona Diab, Veselin Stoyanov, Xian Li
TL;DR
English-dominated training may limit how well generative language models generalize across languages in few-shot settings. The paper trains multilingual generative models on diverse-language data and evaluates zero- and few-shot learning across tasks. Its 7.5B-parameter model achieves state-of-the-art few-shot results across more than 20 languages, while retaining English-performance and prompting limitations.
Problem
Multilingual few-shot learning capabilities are less well understood because generative models have primarily been developed for English and GPT-3 contains only 7% non-English text.
Method
The authors train four multilingual generative models up to 7.5B parameters on 500B tokens spanning 30 diverse languages and evaluate prompted zero- and few-shot learning across multilingual NLU, translation, and English tasks.
Results
The 7.5B-parameter model sets a new few-shot state of the art in more than 20 languages across commonsense reasoning, NLI, and machine translation, outperforming comparable GPT-3 models on multilingual tasks.
Takeaways & Limitations
Strong multilingual few-shot performance can arise through cross-lingual transfer using English prompts and non-English demonstrations, although multilingual pre-training trades off some English performance.
Takeaways & Limitations
The pre-training data may be biased by filtering and inadvertent over-sampling of some low-resource languages, and the model underperforms English-centric models on eight English tasks.
Abstract
from arXiv · showhide
Large-scale generative language models such as GPT-3 are competitive few-shot learners. While these models are known to be able to jointly represent many different languages, their training data is dominated by English, potentially limiting their cross-lingual generalization. In this work, we train multilingual generative language models on a corpus covering a diverse set of languages, and study their few- and zero-shot learning capabilities in a wide range of tasks. Our largest model with 7.5 billion parameters sets new state of the art in few-shot learning in more than 20 representative languages, outperforming GPT-3 of comparable size in multilingual commonsense reasoning (with +7.4% absolute accuracy improvement in 0-shot settings and +9.4% in 4-shot settings) and natural language inference (+5.4% in each of 0-shot and 4-shot settings). On the FLORES-101 machine translation benchmark, our model outperforms GPT-3 on 171 out of 182 directions with 32 training examples, while surpassing the official supervised baseline in 45 directions. We conduct an in-depth analysis of different multilingual prompting approaches, showing in particular that strong few-shot learning performance across languages can be achieved via cross-lingual transfer through both templates and demonstration examples. Finally, we evaluate our models in social value tasks such as hate speech detection in five languages and find it has limitations similar to comparable sized GPT-3 models.
1 Introduction
The paper studies multilingual few-shot learning to address the limited understanding of such capabilities in models whose training data and deployment are primarily English-focused. It trains multilingual generative models and evaluates them across languages and tasks, finding strong cross-lingual performance alongside English-language trade-offs.
- Motivation: Multilingual few-shot learning capabilities are less well understood despite GPT-3’s limited cross-lingual generalization and the use of multilingual models in fine-tuned settings.GPT-3 contains 7% non-English text, while existing multilingual models are typically fine-tuned with substantial labeled data.
- Approach: The authors train multilingual generative models on 500B tokens spanning 30 diverse languages, up-sampling less-resourced languages, and evaluate multilingual NLU, translation, and English tasks.The study covers zero-shot and in-context few-shot learning.
- Results: 65.4% XStoryCloze 0-shot and 46.3% XNLI 0-shot are achieved by XGLM7.5B, rising to 66.5% and 47.3% respectively with 4 shots.These are reported examples of strong language-completion and inference performance.
- Results: XGLM7.5B establishes a new few-shot machine-translation state of the art across many FLORES-101 language pairs and outperforms GPT-3 of comparable size.The comparison is made against GPT-3 with 6.7 billion parameters.
2 Models and Pre-training Data
XGLM uses decoder-only Transformer language models trained on a large, balanced multilingual corpus. The data pipeline up-samples lower-resource languages, but includes acknowledged sampling and filtering limitations.
- Pre-training Data: The pre-training corpus covers 30 languages from 16 language families, with medium- and low-resource languages up-sampled to counter a naturally skewed distribution.English initially had six times as many tokens as the second-largest language.
- Model Vocabulary: A 250k joint vocabulary is created for all languages through unigram language modeling and SentencePiece.The vocabulary model is trained on 10 million randomly sampled sentences with α=0.3.
- Limitations: Some less-resourced languages were inadvertently over-sampled, and the authors did not ablate this mistake because of the computational cost.They identify optimal language balancing as future work.
- Models: The models are decoder-only causal Transformers with 564M, 1.7B, 2.9B, and 7.5B parameters, trained for up to 500B tokens with 2048-token context.Their architecture is similar to GPT-3, with additional embedding parameters from a larger vocabulary.
- Models: Table 1 reports model size, layer count, and hidden dimension, with models in the same row representing comparable sizes.These fields summarize the model configurations used for comparison.
3 Multilingual In-context Learning
The study evaluates multilingual in-context learning with prompted downstream tasks and compares alternative ways to construct prompts across languages. It emphasizes cross-lingual prompting and demonstrations while noting that prompt quality is difficult to optimize.
- Data Distribution: Figure 1 compares each language’s pre- and post-upsampling shares in XGLM’s data with its share in GPT-3’s data, truncating the y-axis at 10% to show the tail.The figure uses blue, green, and orange for the three distributions.
- Prompting Challenge: Prompt construction is challenging because multilingual tasks require suitable prompts for examples in different languages.The paper frames prompt construction as a key issue in multilingual in-context learning.
- Prompting Approaches: The authors consider native handcrafted prompts, translations of English prompts, and direct cross-lingual prompting with English or another high-resource language.Translated prompts use automatic translation APIs, while non-verbal prompts can be directly reused.
- Cross-lingual Transfer: Cross-lingual demonstrations append examples from another language to the context, offering a parameter-free route for transfer from high-resource to low-resource languages.This approach relies on the multilingual model’s cross-lingual nature.
4 Experiments and Results
The experiments evaluate XGLM’s multilingual zero- and few-shot learning across multilingual tasks, prompting strategies, cross-lingual transfer, machine translation, and English tasks. XGLM performs strongly across many languages and translation directions, while multilingual pre-training leaves a substantial English-task gap and several evaluation limitations remain.
- Evaluation setup: XGLM is evaluated without parameter updates on multilingual tasks spanning commonsense reasoning, anaphora resolution, inference, paraphrasing, machine translation, and social-value tasks.The multilingual evaluation includes XStoryCloze, XCOPA, XWinograd, XNLI, PAWS-X, and additional translation and social-value benchmarks.
- Cross-lingual transfer through templates: English templates perform best on average across languages for XNLI and XCOPA, except for 4-shot XCOPA, where machine-translated templates slightly outperform them.English templates substantially improve Chinese and Hindi on XNLI, with similar trends for Thai and Swahili on XCOPA.
- Cross-lingual transfer through demonstration examples: Most cross-lingual demonstration settings improve target-language XNLI over 0-shot performance, although Bulgarian and Swahili show limited transfer benefits.These settings typically underperform in-language few-shot prompting, and transfer can be canceled when the target uses a better prompt such as English.
- Multilingual task performance: XGLM7.5B outperforms GPT-36.7B by a large margin on average across languages, especially for medium-, low-, and extremely low-resource languages.GPT-36.7B remains stronger on English and some similar or high-resource languages.
- Machine translation: 171 out of 182 FLORES-101 translation directions favor XGLM7.5B over GPT-3, and XGLM surpasses the official supervised baseline in 45 directions.GPT-3 remains strong for a few languages represented in its training data but performs poorly on several less-resourced target languages.
5 Related Work
Prior work established multilingual pretraining and cross-lingual in-context learning, while newer approaches expanded transfer through shared representations, joint training, and larger multilingual models.
- GPT-3-style prompting demonstrated few-shot learning without task-specific parameter updates, extending in-context learning beyond English.
- Cross-lingual in-context learning studies evaluated monolingual and cross-lingual prompts in German, Spanish, and French using GPT-2, GPT-Neo, and T5.
- Multilingual counterparts of BERT, RoBERTa, BART, and T5 were developed, but typically relied on large labeled datasets for downstream fine-tuning.
- Shared subword vocabularies and joint training were proposed to facilitate knowledge transfer across languages.
- BLOOM concurrently scaled multilingual pretraining to 46 languages and 175 billion parameters.
6 Conclusion
The paper introduces XGLM models to study multilingual in-context learning across scales and finds that larger models achieve strong few-shot performance across many languages and tasks.
- XGLM studies in-context few- and zero-shot learning with four multilingual generative models at different scales.
- The 7.5B-parameter model sets a new state of the art for few-shot learning in more than 20 languages across commonsense reasoning, NLI, and machine translation.
- Strong few-shot performance in non-English languages is associated with the models’ highly cross-lingual behavior.
Limitations
The authors identify limitations involving training data, model architecture, evaluation design, English-task performance, and the narrow coverage of social-value evaluations.
- Training Data: The training corpus is static CommonCrawl data with 32.6% English, potentially introducing quality, temporal, genre, and filtering-related biases.
- Performance on English tasks: English-task performance is lower than English-centric models, with the gap potentially reflecting differences in training-data quality and quantity plus the curse of multilinguality.
- Model architecture and training objective: The study uses only decoder-only causal language models, excluding bidirectional masked-language and sequence-to-sequence pretraining objectives.
- Model evaluation via in-context learning: Prompt-based comparisons may be biased because the same minimally engineered prompts can work better for some models than others.
- Evaluation on social value tasks for more languages: Social-value analyses cover only six languages, mostly Romance languages, and available datasets constrain evaluation of hate speech and gender bias.
Ethical Considerations
The paper frames multilingual models as tools for broader language inclusion and more efficient deployment, while examining social-value behavior and documenting implementation transparency.
- Energy and maintenance efficiency: A unified multilingual model can serve many languages and may reduce carbon footprint and energy consumption relative to separate language-specific models.
- Diversity and inclusion: XGLM is presented as a shift away from an Anglo-centric NLP perspective through more balanced language representation and multilingual few-shot capability.
- Social values: Social-value evaluations examine hate speech detection and occupational gender bias, with inconclusive overall results and stronger performance for XGLM on Polish in a scarce-data setting.
- Transparency and Accountability: The release includes a detailed model card and data card to support transparency and accountability.
A.1 Validation Perplexity
The evaluation validates XGLM convergence with in-domain perplexity on CC100-XL, while the multilingual in-context framework represents tasks through prompts, verbalizers, and demonstrations.
- Validation Perplexity: In-domain validation perplexity measures convergence on 30,000 held-out sentences per language from CC100-XL, grouped by resource level.The validation data do not overlap with pre-training data.
- Validation Perplexity: Figure A1 plots XGLM perplexity on the CC100_XL validation set as model size changes.
- Multilingual In-context Learning Formulation: The task formulation combines a task description with demonstration examples in one or more languages.The framework models tasks as D = (P, E), where P is the task description and E contains demonstrations.
- Multilingual In-context Learning Formulation: A prompt uses a cloze-style template with a [Mask] symbol, and a verbalizer maps candidate labels or choices to tokenized strings.The instantiated prompt replaces [Mask] with the verbalized candidate.
- Multilingual In-context Learning Formulation: Few-shot learning concatenates instantiated demonstration prompts with separators before the target prompt.These demonstrations form the prefix of the input used for prediction.
C.1 English Evaluation Tasks
The English evaluation follows established few-shot prompting choices and measures accuracy, while prompting-language choices materially affect multilingual task performance.
- English Evaluation Tasks: All English evaluation tasks use accuracy as the metric.Table A1 lists the English tasks included in the few-shot evaluation.
- English Evaluation Tasks: The English setup follows Brown et al.’s prompts and uses a single newline separator for few-shot examples.Several tasks use task-specific likelihood normalization or suffix scoring.
- English Evaluation Tasks: Few-shot results are averaged across five runs with randomly sampled examples, and inputs are truncated to XGLM’s 2,048-token context length.Only complete demonstration examples are retained after truncation.
- Prompting Choices: Prompting choices substantially affect XNLI and XCOPA performance across languages.English prompts outperform machine-translated prompts on average, while human-translated XNLI prompts generally outperform machine-translated prompts.
- Prompting Choices: Human-translated prompts outperform English prompts for Spanish and German but hurt performance for Chinese and Urdu relative to other prompting choices.The relative strength of prompting approaches varies by language.
D.2 Full Results on Learning from Cross-lingual Demonstrations
Cross-lingual demonstrations and templates can transfer across languages, but their benefits depend on prompting choices and task structure; additional evaluations expose sampling, knowledge, and safety considerations.
- Cross-lingual Demonstrations: Cross-lingual demonstrations often improve target-language XNLI performance when human-translated templates are used.Gains are especially notable for Chinese, Thai, and Urdu, whose human-translated-template zero-shot performance is near random.
- Cross-lingual Demonstrations: English templates improve zero-shot performance for most languages, but demonstrations generally help only when source and target languages match.Template transfer and demonstration transfer typically do not add together.
- Cross-lingual Demonstrations: XStoryCloze shows almost no improvement across language pairs, possibly because implicit multi-sentence reasoning is difficult cross-lingually.Figure A5 reports the difference between 4-shot and 0-shot performance.
- Majority Label Bias: Randomly sampled few-shot XNLI sets can have skewed class distributions, motivating comparison with uniform per-class sampling.XNLI contains entailment, neutral, and contradiction classes.
- Majority Label Bias: Uniform class-distribution training sets produce significantly higher XNLI accuracy than skewed sets in both evaluated shot settings and all languages.The comparison uses 24-shot and truncated 48-shot settings.
- Knowledge and Social Values: XGLM retains non-English factual knowledge above 0.6 across evaluated languages, whereas GPT-3 Curie drops sharply for medium- and low-resource languages.The comparison covers 25 pre-training languages and uses mLAMA precision@1.
E.1.2 Results
The evaluation reports multilingual hate-speech results, occupation-identification results, and supporting dataset documentation. Hate-speech detection remains difficult, while occupation-identification performance varies across languages and models.
- Hate speech detection: In-context hate-speech results across five languages are only slightly better than random and are unstable under prompt changes.Most few-shot results are worse than zero-shot, and one-shot models often copy the example label rather than classify the input.
- Hate speech detection: XGLM7.5B achieves better recall than English-centric models in multilingual hate-speech detection.The XGLM6.7B En-only model has very low zero-shot recall when the language condition is set to the same language.
- Occupation identification: XGLM6.7B En-only achieves the best occupation-identification performance on English and Spanish, while GPT-36.7B performs best on French.XGLM7.5B falls behind both models on all three languages, especially Spanish and French.
- Occupation identification: GPT-36.7B is the most biased model on English and Spanish, whereas XGLM6.7B En-only and XGLM7.5B exhibit the least bias on Spanish and English, respectively.Bias is measured as the average absolute accuracy gap between male and female groups across occupations.
- Documentation: The dataset documentation includes a datacard for the multilingual training data used by XGLM.The dataset is a subset of the curated CC100-XL corpus.
F.3 Composition
The training data consists of raw Common Crawl documents, sampled into training and validation sets and processed to reduce duplication. Its composition includes documented sampling losses, possible sensitive content, and potential individual identifiability.
- Dataset composition: The XGLM training dataset contains 1.74 billion documents consisting of raw text without labels or explicit relationships.The instances are textual documents sampled from Common Crawl snapshots, and fewer than 2% are duplicated.
- Dataset composition: The dataset is split into training and validation sets, with larger validation allocations for high-resource languages than for low-resource languages.High-resource languages use at least 5,000 documents and 30,000 lines for validation; low-resource languages use at least 100 documents and 1,000 lines.
- Data considerations: Ten percent of the Russian sample was lost during internal data transfer, leaving a 26.7% random subset of Russian data from CC100-XL.No ethical review process was conducted.
- Data considerations: The dataset may contain offensive or threatening sentences, and individuals may be identifiable through names or account names despite PII identification tools.The dataset documentation also notes that sensitive information could remain because the filtering lists may not be exhaustive.
- Dataset composition: The corpus was collected from 68 Common Crawl snapshots spanning Summer 2013 to March/April 2020, limiting coverage of recent events such as COVID-19.The data was mined, filtered, and sampled by machines.