PaperlayerView exact revision on arXiv

Source-linked AI summary

LLMs Are Few-Shot In-Context Low-Resource Language Learners

Samuel Cahyawijaya, Holy Lovenia, Pascale Fung

arXiv:2403.16512v5cs.CLcs.AI

TL;DR

LLMs struggle to generalize to low-resource languages, and prior ICL research has focused mostly on relatively higher-resource languages. This study evaluates ICL and X-ICL across language resource levels and tests alternatives including query alignment. It finds label alignment does not help all languages, while query alignment improves alignment quality over label alignment.

  • Problem

    LLMs struggle to generalize to low-resource languages, while language-specific updates risk catastrophic forgetting and require substantial computation.

    Source: §1 · p. 1

  • Method

    The study evaluates ICL and X-ICL across 25 low-resource and 7 relatively higher-resource languages, varying alignment, prompting, label encoding, and exemplar retrieval.

    Source: §1 · p. 2

  • Results

    Label alignment does not work for all studied languages, while query alignment significantly improves alignment quality compared with label alignment; X-ICL remains relevant when target-language MT is unavailable.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    Few-shot context can support low-resource understanding through semantically relevant information that closes target-language gaps and aligns low- and high-resource language semantics.

    Source: §1 · p. 2

    Available downstream datasets for low-resource languages are limited, so the authors call for evaluation across broader tasks, including culturally relevant nuances.

    Source: Limitation · p. 10

Abstract

from arXiv · show

In-context learning (ICL) empowers large language models (LLMs) to perform diverse tasks in underrepresented languages using only short in-context information, offering a crucial avenue for narrowing the gap between high-resource and low-resource languages. Nonetheless, there is only a handful of works explored ICL for low-resource languages with most of them focusing on relatively high-resource languages, such as French and Spanish. In this work, we extensively study ICL and its cross-lingual variation (X-ICL) on 25 low-resource and 7 relatively higher-resource languages. Our study not only assesses the effectiveness of ICL with LLMs in low-resource languages but also identifies the shortcomings of in-context label alignment, and introduces a more effective alternative: query alignment. Moreover, we provide valuable insights into various facets of ICL for low-resource languages. Our study concludes the significance of few-shot in-context information on enhancing the low-resource understanding quality of LLMs through semantically relevant information by closing the language gap in the target language and aligning the semantics between the targeted low-resource and the high-resource language that the model is proficient in. Our work highlights the importance of advancing ICL research, particularly for low-resource languages. Our code is publicly released at https://github.com/SamuelCahyawijaya/in-context-alignment

1 Introduction

The introduction examines whether in-context learning can improve LLM understanding of low-resource languages without parameter updates. It presents query alignment as an alternative to label alignment and studies how language resource level and semantically relevant context shape performance.

1 Introduction

  • LLMs struggle to generalize across languages, especially low-resource ones; language-specific updates risk catastrophic forgetting and require substantial computation.The introduction presents X-ICL as a parameter-free approach, though it still fails to match translate-test baselines, especially for low-resource languages.Source: §1 · p. 1
  • X-ICL performance decay correlates with target-language pretraining-data size, consistent with the authors’ hypothesis about weaker low-resource language representations.The study compares 25 low-resource languages with seven relatively higher-resource languages.Source: §1 · p. 2
  • The study examines variations in alignment information, prompts, label encoding, and exemplar selection to assess their effects on downstream performance.Its findings include that format consistency does not benefit the low-resource languages studied, despite improving performance for higher-resource languages.Source: §1 · p. 2
  • Label alignment undermines performance in most studied languages, while query alignment is a more effective alternative.The study reports that query alignment significantly improves alignment quality compared with label alignment.Source: §1 · p. 2
  • Few-shot context can support low-resource understanding through semantically relevant information: monolingual ICL closes language and domain gaps, while X-ICL and alignment address domain and cross-language semantic gaps.The introduction distinguishes the gaps each approach addresses.Source: §1 · p. 2

2 Related Work

Prior work established few-shot learning and multilingual evaluation with LLMs, explored cross-lingual exemplars and prompting, and documented persistent challenges in low-resource languages.

2.1 In-Context Learning

  • Few-shot ICL uses task-specific formatting and examples, while multilingual LLM studies expanded evaluation across languages and tasks.Belebele extended multilingual evaluation to 122 languages.Source: §2.1 · p. 2

Cross-Lingual In-Context Learning (X-ICL)

  • X-ICL studies examined exemplar-language choices, instructions, semantic retrieval, and label alignment, but Tanwar et al. (2023)’s reported benefit from label alignment diverges from this paper’s findings.Earlier work found random source-language mixtures surprisingly effective, and English instructions improved zero-shot performance across multiple languages.Source: Cross-Lingual In-Context Learning… · p. 3

2.2 LLMs on Low-Resource Languages

  • Evaluations report that multilingual LLMs perform worse on low-resource languages, and models struggle with extremely low-resource languages and code-switching.Cahyawijaya et al. (2023a) compared Indonesian and English with 18 Indonesian indigenous languages; Cahyawijaya et al. (2023b) examined 12 low- and extremely low-resource languages.Source: §2.2 · p. 3

3 Methods

The paper’s X-ICL framework varies alignment, prompt formatting, label configuration, and cross-lingual exemplar retrieval to study their effects across language resource levels.

3 Methods

  • The X-ICL framework studies cross-lingual alignment, prompting choices, label configuration, and exemplar retrieval across language resource levels.These components define the dimensions varied in the methods.Source: §3 · p. 3

3.1 Cross-Lingual Alignment

  • Label alignment prompts mappings from source labels to target labels, whereas query alignment provides translations of sentences similar to the target query while keeping the label set unchanged.Query alignment uses parallel exemplars to align the input distribution; Figure 2 depicts three alignment formats.Source: §3.1 · pp. 3–4, Figure 2 · p. 4

3.2 Cross-Lingual Prompting

  • To vary prompt consistency, alignment-before moves alignment text ahead of exemplars and tabular prompting arranges content in columns; source-only and target-only labels provide alternatives to aligned labels.Tabular prompting requires a labeled parallel corpus or risks disrupting input-output mapping through incorrect labeling; English is the study’s source language.Source: §3.2 · p. 4

3.3 Cross-Lingual Retrieval

  • Cross-lingual retrieval selects labeled source-language exemplars relevant to the target query; translation semantic similarity first matches a parallel target-language sentence, then retrieves source exemplars using its pair.The authors argue direct cross-lingual semantic representations may be poorly aligned for low-resource languages; they call the translation-semantic-similarity method T-ICL.Source: §3.3 · p. 5

4 Experimental Settings

The experiments use few-shot in-context learning with two multilingual LLMs and evaluate X-ICL across low- and higher-resource languages, datasets, and alignment settings.

4.1 Retrieval and In-Context Learning Setup

  • Experiments use 3-shot exemplars with XGLM-7.5B and BLOOM-7B, and multilingual sentence transformers to calculate cross-lingual and monolingual semantic similarity.The prediction label is selected to maximize the prompt’s marginal probability.Source: §4.1 · p. 5

4.2 Languages and Datasets

  • The study covers 25 low-resource languages across Africa, the Americas, and South-East Asia, plus seven relatively higher-resource languages for comparison.The low-resource languages span 13 language families; some are unseen by both models, though related languages may have appeared in pretraining.Source: §4.2 · p. 5
  • Evaluation uses four datasets spanning topic classification, natural language inference, and sentiment analysis, with ICL and parallel alignment datasets drawn from different subsets or datasets.Table 2 lists the Dsrc and Dpara for the evaluation datasets; some are translated into English with NLLB, and some datasets are aligned.Source: §4.2 · p. 5, Table 2 · p. 5
  • Only languages supported by NLLB are included to enable comparison with machine-translation-based approaches, and monolingual semantic-similarity baselines use evaluation train and development sets.The study compares label-alignment variants across higher-resource, African low-resource, and American low-resource languages, and query alignment with and without X-ICL.Source: §4.2 · p. 6, Figure 3 · p. 6, Figure 4 · p. 6

5 Result and Discussion

Across the evaluated approaches, semantic-similarity retrieval improves on random retrieval, while query alignment is more effective than label alignment. Formatting consistency has little benefit for low-resource languages, and X-ICL remains useful when machine translation is unavailable.

5.1 Inferiority of In-Context Label Alignment

  • In-context label alignment improves performance in only ∼11.54% of experiments, while ∼88.46% decrease; source-only labels perform best in most languages.For low-resource African languages, target-only labels perform much worse; the authors conjecture this reflects weak language representation.Source: Figure 5 · p. 6, §5.1 · p. 6, §5.2 · p. 7

5.4 Effect of Format Consistency

  • Formatting consistency correlates with slight performance gains in higher-resource languages, but its low-resource trends are unclear and it does not improve low-resource understanding.The authors identify X-ICL and query alignment as better alternatives for improving low-resource language understanding.Source: Figure 6 · p. 7, §5.4 · pp. 7–8

5.2 In-Context Query Alignment

  • 56.25% of query-alignment experiments improve by ∼10% weighted F1, whereas the remaining 43.75% decline by ∼5% weighted F1.Query alignment improves zero-shot performance on low-resource languages, with marginal improvement in X-ICL, and can substitute for or complement X-ICL.Source: §5.2 · pp. 6–7

5.3 Why Query Alignment Performs Better

  • Query alignment can guide interpretation of the target-language query, whereas label alignment transfers label semantics without guiding query interpretation.When query alignment fails, it behaves like regular X-ICL; failed label alignment instead shifts the label space, which the authors connect to severe degradation.Source: §5.3 · p. 7

5.5 Importance of Cross-Lingual Retrieval

  • Cross-lingual and monolingual semantic-similarity retrieval outperform zero-shot prompting, while translation semantic similarity performs almost on par with zero-shot.Cross-lingual retrieval performs similarly to or slightly below monolingual retrieval and is useful when monolingual task corpora are unavailable.Source: Figure 7 · p. 8, §5.5 · p. 8
  • All tested cross-lingual semantic-similarity models outperform zero-shot, but XLMR STS performs differently by resource level and model choice can substantially affect X-ICL.XLMR STS is on par with other models on average, doing better on higher-resource and worse on low-resource languages.Source: §5.5 · p. 8, Figure 8 · p. 9

5.6 Is X-ICL Effective for low-resource Languages?

  • Semantic ICL and X-ICL outperform random retrieval; translate-test ICL yields the highest improvement when machine-translation quality exceeds a certain standard.X-ICL Semantic performs similarly to ICL Semantic, and the authors recommend choosing ICL or X-ICL based on corpus availability when no MT system exists.Source: §5.6 · pp. 8–9, Figure 9 · p. 9
  • The recommended inference approach depends on available resources: use translate-test ICL with high-quality MT and source-language task data, or query alignment with parallel data when neither MT nor task-specific data is available.When MT exists but task-specific data do not, the authors recommend translate-test; without MT, they recommend ICL or X-ICL according to corpus availability.Source: §5.6 · p. 9

6 Conclusion

The conclusion highlights X-ICL research for low-resource languages, including its alignment, formatting, and retrieval choices. It notes that X-ICL is outperformed by translate-test ICL but remains relevant where target-language MT is unavailable.

6 Conclusion

  • The study systematically investigates X-ICL with LLMs, focusing on low-resource languages.Its analysis identifies label alignment's limitations and proposes query alignment as a more effective alternative.Source: §6 · p. 9
  • Formatting consistency has only marginal impact on low-resource languages, while cross-lingual semantic similarity is significant for X-ICL exemplar retrieval.The conclusion presents these as findings from its analysis of prompting and retrieval approaches.Source: §6 · p. 9
  • Although translate-test ICL outperforms X-ICL, X-ICL remains relevant when no MT model is available for the target language.Source: §6 · p. 9

Ethics Statement

The ethics statement frames ICL and X-ICL as potential approaches for low-resource settings with limited linguistic resources. It also states that the experimental datasets follow their licenses and terms of use.

Ethics Statement

  • The work addresses linguistic-data gaps for low-resource languages, especially where monolingual corpora or MT systems are absent.Source: Ethics Statement · p. 10
  • The statement presents ICL and X-ICL as viable solutions when alternative resources are absent.Source: Ethics Statement · p. 10
  • The datasets used in the experiments follow their licenses and terms of use.Source: Ethics Statement · p. 10

Limitation

The study’s scope is bounded by the available low-resource and high-resource datasets, task coverage, and compute; it covers 32 languages and illustrates its alignment prompts.

Limitation

  • Only three low-resource datasets were used, selected for parallel counterparts and enough high-quality labeled data for ICL and evaluation.The authors leave broader low-resource language coverage for future work.Source: Limitation · p. 10
  • The high-resource comparison uses seven languages from TweetSentimentMultilingual; other high-resource languages remain for future study.MARC and CLS were unavailable, while HatEval covered only English and Spanish.Source: Limitation · p. 10
  • The available downstream datasets are few, motivating future evaluation on broader tasks, especially those capturing culturally relevant nuances.The authors identify limited task coverage as a consequence of the available low-resource-language datasets.Source: Limitation · p. 10
  • Experiments used a single RTX3090 GPU, and larger multilingual models such as Falcon and MPT were not tested because of inference cost and limited compute.The authors hypothesize that their work’s scaling behavior will follow the trend reported in prior work.Source: Limitation · p. 10

A Languages Under Study

C Analysis on Cross-lingual Semantic Similarity

The analysis examines semantic similarity and retrieval, language labels, machine-translation quality, and source-language choice in X-ICL. It reports weaker similarity correlations for many low-resource languages and better scores with English exemplars than with closely related ones in most cases.

C Analysis on Cross-lingual Semantic Similarity

D Language Label

  • The paper provides language-specific label sets for MasakhaNews, TweetSentimentMultilingual, NusaTranslation, and AmericasNLI.The label sets are listed in Tables 4–7, respectively.Source: Table 4 · p. 20, Table 5 · p. 21, Table 6 · p. 21

E Effect of Machine Translation Quality to X-ICL

  • Zero-shot MT performance correlates with translation quality at 0.416 for XGLM and 0.247 for BLOOMZ; few-shot ICL (MT) correlations are 0.102 and 0.238, respectively.The authors note that other factors, including exemplar selection and ICL data quality, may affect few-shot results.Source: E Effect of Machine Translation… · p. 22, Table 8 · p. 22

F Effects of Source Languages

  • English exemplars tend to produce better NusaTranslation scores than Indonesian exemplars in most cases, including in X-ICL.The comparison uses English and Indonesian prompts with corresponding English and Indonesian ICL exemplars.Source: F Effects of Source Languages · p. 23, Figure 13 · p. 23

G Visualization of BLOOM Result

The figures visualize BLOOM-7B1 performance across higher-resource and low-resource language groups for label alignment, query alignment, and alignment versus a non-alignment baseline.

G Visualization of BLOOM Result

  • Figure 14 reports BLOOM-7B1 performance with in-context label alignment, target-only labels, and source-only labels for higher-resource, low-resource African, and low-resource American languages.Source: Figure 14 · p. 24
  • Figure 15 presents BLOOM-7B1 performance with and without query alignment across higher-resource languages, low-resource African languages, and low-resource American languages.Source: Figure 15 · p. 24
  • Figure 6 compares weighted-F1 changes for label and query alignment against the non-alignment baseline; negative values indicate degraded performance.The left panel shows label alignment and the right panel query alignment.Source: Figure 16 · p. 24

H Detailed Per Dataset Results

The appendix supplies per-dataset experiment-result tables for XGLM-7.5B and BLOOM-7B1, alongside figures comparing alignment formats and exemplar retrieval strategies.

H Detailed Per Dataset Results

  • Tables 9–12 report XGLM-7.5B results for TweetSentimentMultilingual, MasakhaNews, NusaTranslation, and AmericasNLI.A dash marks experiments not conducted when a machine-translation system was unavailable.Source: H Detailed Per Dataset Results · p. 24, Table 9 · p. 25, Table 10 · p. 26
  • Figures 17 and 18 show BLOOM-7B1 performance across alignment formats ordered by formatting consistency and in-context retrievals spanning monolingual, cross-lingual, and translation semantic similarity, across four language-resource groups.The groups are higher-resource, low-resource Indonesian, low-resource American, and low-resource African languages.Source: Figure 17 · p. 25, Figure 18 · p. 25
  • Tables 13–16 report BLOOM-7B1 results for TweetSentimentMultilingual, MasakhaNews, NusaTranslation, and AmericasNLI.A dash marks experiments not conducted when a machine-translation system was unavailable.Source: H Detailed Per Dataset Results · p. 25, Table 13 · p. 27, Table 14 · p. 28

Related papers

Loading 2403.16512v5…