Source-linked AI summary
Language Models are Few-shot Multilingual Learners
Genta Indra Winata, Andrea Madotto, Zhaojiang Lin, Rosanne Liu, Jason Yosinski, Pascale Fung
TL;DR
The paper asks whether pretrained language models can support multilingual few-shot intent prediction without parameter updates, addressing the difficulty of obtaining labeled data for non-English languages. It evaluates GPT and T5 models using English examples as context and a confidence-based binary-prediction method, finding that they can predict non-English samples significantly better than random, with larger models and more shots improving performance. The approach is efficient in avoiding training updates, but inference can be slower as the number of few-shot samples grows.
Problem
The paper investigates whether language models can perform multilingual intent prediction with few labeled examples, reducing dependence on costly supervised data for low-resource languages.
Method
The study evaluates GPT and T5 models across four languages without gradient updates, using English contextual examples and confidence-based binary predictions for each candidate class.
Results
Pretrained GPT and T5 models predict non-English samples significantly better than random, while more shots and larger models substantially improve performance.
Takeaways & Limitations
In-context learning can support cross-lingual prediction when target-language training examples are unavailable, demonstrating multilingual capability in pretrained language models.
Takeaways & Limitations
Few-shot inference takes more time per step, creating a potential bottleneck when many few-shot samples are used.
Abstract
from arXiv · showhide
General-purpose language models have demonstrated impressive capabilities, performing on par with state-of-the-art approaches on a range of downstream natural language processing (NLP) tasks and benchmarks when inferring instructions from very few examples. Here, we evaluate the multilingual skills of the GPT and T5 models in conducting multi-class classification on non-English languages without any parameter updates. We show that, given a few English examples as context, pre-trained language models can predict not only English test samples but also non-English ones. Finally, we find the in-context few-shot cross-lingual prediction results of language models are significantly better than random prediction, and they are competitive compared to the existing state-of-the-art cross-lingual models.
1 Introduction
This work investigates whether pretrained GPT and T5 models can perform multilingual few-shot intent prediction without parameter updates. Using English examples as context, it studies cross-lingual prediction and introduces a confidence-based binary-prediction approach for multi-class classification.
- Motivation: Few-shot learning reduces reliance on extensive labeled data, which is costly to collect for low-resource languages.The paper positions multilingual few-shot learning as an alternative to supervised data collection and fine-tuning.
- Study scope: The study evaluates publicly available GPT and T5 models across English, French, German, and Spanish without gradient updates.Results are compared with zero-shot and fine-tuning approaches.
- Cross-lingual setting: Given a few English examples as context, pretrained language models predict both English and non-English test samples.The cross-lingual setup uses a source language different from the target language.
- Method: The proposed multi-class method compares normalized confidence scores for boolean-token predictions across candidate classes.It applies multiple binary predictions rather than placing all examples and classes into one k-way prompt.
- Findings: GPT and T5 models perform significantly better than random prediction, with larger models and more shots producing substantial performance gains.T5 models perform worse than GPT models, possibly because of differences in pretraining strategy.
2 Few-shot Multilingual Learners
The method frames multilingual intent detection as few-shot multiple-choice prediction, using prompts and confidence over boolean token probabilities without gradient updates.
- 2.1 Notation and Tasks: The task uses prompt P = [Dpos, Dneg, Q], combining same-label examples, different-label examples, and a query for language-model inference.
- 2.1 Notation and Tasks: Multilingual intent detection is evaluated in monolingual and cross-lingual settings, with the latter using a source language different from the target language.
- 2.2 Prompt Generation: Multi-class prediction is converted into binary true-or-false decisions for each label, selecting the label with the highest prediction confidence.The approach probes GPT and T5 models with prompts tailored to their different learning objectives.
- 2.2 Prompt Generation: For each sample, the method creates L prompts with randomly drawn positive and negative examples rather than merging all shots into one prompt.Separate prompts allow more shots within model input-token limits.
- 2.3 Maximum Confidence Prediction: The final label is chosen by normalizing true-token scores for each label and selecting the label with the highest confidence.
- 2.2 Prompt Generation: Arranging positive examples before negative examples performs well in the authors’ sample-order comparison.The study compares shuffled examples with prompts that place positive samples first.
3 Baselines
The paper compares few-shot in-context learning with zero-shot, zero-shot cross-task, and fine-tuning baselines using entailment or supervised weight updates.
- 3 Baselines: The evaluation compares few-shot learning with zero-shot, zero-shot cross-task, and fine-tuning approaches.
- 3.1 Zero-shot Cross-Task: Zero-shot cross-task prediction uses an entailment model to score combinations of hypotheses and possible labels.The model selects an entailment score for each hypothesis-label combination.
- 3.1 Zero-shot Cross-Task: The zero-shot cross-task baseline presents natural-language questions asking whether an intent matches a text.
- 3.3 Fine-tuning: Fine-tuning updates pretrained model weights using labeled data and the same few-shot samples used for in-context learning.The authors describe supervised correct labels as the advantage of fine-tuning.
4 Experiments
Experiments evaluate GPT and T5 models on multilingual intent datasets across monolingual and English-to-non-English settings, using accuracy over repeated runs and several baselines.
- 4 Experiments: The datasets include English SNIPS, multilingual MTOP in English, French, German, and Spanish, and Multilingual NLU in English and Spanish.
- 4 Experiments: Performance is measured as mean accuracy and standard deviation across three runs.
- 4 Experiments: The experiments test monolingual few-shot learning in four languages and cross-lingual prediction of non-English queries from English context.
- 4.2 Experiment Settings: The few-shot settings use k ∈ [0, 5, K], where K ≤ 40 is constrained by the model’s maximum input-token limit.Inference uses an NVIDIA Tesla V100 16GB GPU with 16-bit precision.
- 4.2 Experiment Settings: The model set includes four GPT-2 sizes, three GPTNEO sizes, and two T5 sizes ranging from 0.1B to 6B parameters.
- 4.2 Experiment Settings: Baselines include fine-tuned mBERT and XLM-R, zero-shot cross-task XLM-R and BART models, random prediction, and reported state-of-the-art results.
5 Results and Analysis
Few-shot performance improves with larger models and more examples, remaining competitive with fine-tuned systems across monolingual and cross-lingual settings. Results also vary by architecture, sample order, language, and inference cost.
- Model performance: Performance improvements are strongly associated with larger pretrained models, and the best few-shot model is less than 10% behind the fine-tuned model.The performance gap decreases with model size, despite no parameter updates.
- Efficiency: Few-shot inference is more efficient in memory than fine-tuning but takes longer per inference step, creating a bottleneck with many examples.The paper identifies reducing inference time as an open research direction.
- Model comparison: GPT models outperform T5 models across language pairs and datasets, while T5 models are less constrained by maximum input length.Sample ordering substantially improves T5 results but causes GPT performance to decline.
- Sample ordering: Changing few-shot sample order can produce high variance in results.The effect is especially pronounced for T5 models, whose performance improves when examples are grouped by label.
- Language effects: Non-English performance is lower than English performance, but the differences are marginal for French, German, and Spanish.The authors identify these results as a possible indication of usefulness for languages related to English, while noting that further investigation is needed.
- Cross-lingual results: English-context prompts can predict non-English samples in the cross-lingual setting, with MultiNLU GPTNEO-J within around 4.2% of a fine-tuned state-of-the-art baseline.GPTNEO-J is also less than 3% worse than the Translate-Train model; MTOP performance is generally higher for en→es than for en→de or en→fr.
- Ablation study: Performance consistently improves as the number of few-shot samples increases, although zero-shot results can fall below random prediction on MultiNLU.On MTOP, few-shot learning performs better than fine-tuning in the reported experiments.
6 Related Work
Related work covers few-shot prompting and major pretrained language-model families, including encoders, decoder-only models, and encoder-decoder models. These models support multilingual NLP tasks ranging from dialogue and labeling to generation and knowledge grounding.
- Few-shot learning: Few-shot in-context learning uses prompts to induce language models to produce task-appropriate outputs for NLP tasks.Prompt engineering and calibration are central concerns in this line of work.
- Pretrained model families: Pretrained language models include encoder architectures such as BERT, RoBERTa, ELMO, ELECTRA, XLM, and XLM-R.These models are presented as a major strand of recent pretrained-language-model research.
- Pretrained model families: Decoder-only models include GPT, while encoder-decoder models include T5, BART, mT5, and mBART.The cited model families span both monolingual and multilingual pretraining approaches.
- Multilingual applications: Pretrained encoders have been applied to dialogue systems, code-switching sequence labeling, and multilingual speech recognition.These applications target multilingual contextual representations across several NLP and speech tasks.
- Multilingual applications: Pretrained encoder-decoder models have been used for summarization, conversational agents, and knowledge grounding.The cited work extends sequence-generation models beyond classification-oriented applications.
7 Conclusion
The paper demonstrates that GPT and T5 models can perform multilingual and cross-lingual in-context learning without parameter updates. Across four languages, the models predict non-English samples significantly better than random prediction, supporting further study of efficient inference for low-resource tasks.
- Conclusion: GPT and T5 models perform multilingual and cross-lingual in-context learning without parameter updates across four languages.The study presents this as an initial examination of multilingual in-context learning and efficient inference for low-resource tasks.
- Conclusion: The language models predict cross-lingual samples significantly better than random prediction.This result supports the paper’s conclusion that multilingual in-context learning is a promising direction for further research.
A Full k-shot Results
The appendix reports full k-shot results across monolingual and cross-lingual settings for SNIPS, MTOP, and MultiNLU, including accuracy and F1 figures for GPT models.
- Appendix coverage: The appendix presents few-shot monolingual and cross-lingual results over different numbers of samples for SNIPS, MTOP, and MultiNLU.The figures cover English, German, Spanish, and French settings where available.
- Reported metrics: The appendix includes accuracy and F1 results for English, German, Spanish, and French GPT-model evaluations across MTOP, MultiNLU, and SNIPS.The displayed results include both monolingual and English-to-target cross-lingual settings.