Source-linked AI summary
Crosslingual Retrieval Augmented In-context Learning for Bangla
Xiaoqian Li, Ercong Nie, Sheng Liang
TL;DR
Bangla is underrepresented in NLP, and LLM performance in low-resource languages remains limited. The paper applies cross-lingual retrieval-augmented in-context learning to MPLMs for Bangla tasks, using high-resource-language examples in prompts. Its evaluation reports steady improvements over zero-shot MPLM performance, while noting that retrieval augmentation’s effectiveness is tied to model architecture.
Problem
Bangla remains underrepresented in NLP amid limited pretraining corpora and labeled datasets, while LLM performance in low-resource languages is limited.
Source: §1 · p. 1Method
Results
Takeaways & Limitations
The reported results support applying cross-lingual retrieval-augmented in-context learning to MPLMs on the tested Bangla classification and summarization tasks.
Source: §1 · p. 2, §7 · p. 8Effectiveness is tied to model architecture, whose impact across different models remains largely unexplored; specific retrieval datasets and resource constraints are also practical challenges.
Source: Limitations · p. 8
Abstract
from arXiv · showhide
The promise of Large Language Models (LLMs) in Natural Language Processing has often been overshadowed by their limited performance in low-resource languages such as Bangla. To address this, our paper presents a pioneering approach that utilizes cross-lingual retrieval augmented in-context learning. By strategically sourcing semantically similar prompts from high-resource language, we enable multilingual pretrained language models (MPLMs), especially the generative model BLOOMZ, to successfully boost performance on Bangla tasks. Our extensive evaluation highlights that the cross-lingual retrieval augmented prompts bring steady improvements to MPLMs over the zero-shot performance.
1 Introduction
Bangla remains underrepresented in NLP, while multilingual in-context learning has often relied on task instructions or randomly selected prompts rather than semantically aligned examples. This work extends cross-lingual retrieval augmentation to generative multilingual models for Bangla tasks and reports steady improvements over zero-shot MPLM performance.
1 Introduction
- Bangla faces limited pretraining corpora, labeled datasets, and persistent data-duplication issues despite its large speaker population.Source: §1 · p. 1
- Unlike task-level or randomly selected prompts, the approach uses semantically aligned examples from high-resource languages and extends PARC to decoder-only BLOOM and BLOOMZ.Figure 1 presents the PARC pipeline using decoder-only multilingual pretrained language models.Source: §1 · pp. 1–2, Figure 1 · p. 1
- The paper evaluates cross-lingual retrieval-augmented in-context learning on Bangla text classification and summarization, reporting steady improvements over MPLM zero-shot performance.Source: §1 · p. 2
2 Related Work
Related work describes Bangla NLP research and multilingual in-context learning, positioning this study as an extension of cross-lingual retrieval augmentation to decoder-only multilingual LLMs.
2 Related Work
- Bangla is a morphologically rich Indo-Aryan language with dialects, yet digital text resources and annotated corpora remain scarce.Source: §2 · p. 2
- Bangla NLP research spans POS tagging, stemming, lemmatization, named-entity recognition, sentiment analysis, and news categorization, alongside later dataset, model, and LLM-prompting work.Prior work includes POS tagging, stemming and lemmatization, named-entity recognition (Ekbal and Bandyopadhyay, 2007, 2008a), and sentiment analysis.Source: §2 · p. 2
- Multilingual in-context learning has used semantically similar examples and cross-lingual retrieval, but prior studies mainly focused on encoder or encoder-decoder models.Nie et al. (2023) proposed PARC with cross-lingual retrieval samples; this work extends the pipeline to decoder-only multilingual LLMs.Source: §2 · pp. 2–3
3 Methodology
The method retrieves high-resource-language examples for a Bangla input, then uses a prefix prompt to guide an MPLM prediction. Retrieved examples may include labels or use self-prediction when labels are absent.
3 Methodology
- The method targets classification and summarization and uses retrieved content from high-resource languages such as English in a zero-shot setting.For decoder-only models, the model generates the answer directly; encoder models instead use a verbalizer and mask-token prediction.Source: §3 · pp. 3–4
- A cross-lingual retriever maps the Bangla input to a shared embedding space and returns the k most semantically similar high-resource-language examples, with or without labels.When retrieved examples have no label, the method suggests a self-prediction step.Source: §3 · p. 3
- Prompt engineering combines the Bangla input and retrieved pattern in a prefix template P to facilitate the model’s prediction.Source: §3 · p. 3
4 Experiments
The experiments test cross-lingual retrieval-augmented in-context learning for Bangla classification and summarization, against zero-shot and extractive baselines. They use multilingual models and retrieve English-language examples for the main method.
4 Experiments
- The study evaluates classification and summarization, calling its approach with k retrieved samples the main method.The experiments focus on these two task types.Source: §4 · p. 4
4.1 Baselines
- Zero-shot prediction uses the input-filled template without cross-lingual context, while Lead64 summarizes by taking the input’s first 64 tokens.Lead64 is used for summarization only.Source: §4.1 · p. 4
4.2.1 Classification
- Classification uses Vio-Lens violent-incident comments and SentNoB sentiment labels; ETHOS supplies an English sentence pool whose inconsistent labels are self-predicted.The English Sentiment Analysis training set also serves as a labeled sentence pool for the experiments.Source: §4.2.1 · p. 4
4.2.2 Summarization
- Summarization uses XL-Sum’s 1.35 million article-summary pairs; the main-method prompt requests a concise summary in the input’s language, unlike the zero-shot prompt.The supplied passages describe the dataset and prompt templates, not a separate retrieval corpus for summarization.Source: §4.2.2 · p. 4
4.3 Models
- The model set includes BLOOM, BLOOMZ, mBERT, and mT5, while a multilingual sentence transformer retrieves semantically similar examples in a 768-dimensional vector space.Retrieval uses k=1 and 3 for classification and k=1 for summarization.Source: §4.3 · p. 5
5 Results
Retrieval-augmented English prompts improve classification F1 for Bloomz-3b and mBERT, while summarization outcomes vary by model and retrieval setting. Template-level results show stable classification gains, but the analysis notes sensitivity to retrieval in summarization.
5.1 Results of classification tasks
- With k=3 retrieval-augmented English prompts, Bloomz-3b F1-scores improve by 5% and 10% across the two classification tasks, while mBERT gains 8% and 7%.Table 1 covers F1-scores for the two tasks under Bangla zero-shot and retrieval-augmented prompting.Source: Table 1 · p. 5, §5.1 · p. 5
- Across micro, macro, and weighted F1, Bloomz-3b and mBERT improve with retrieval prompts; Bloomz-3b zero-shot more often confuses non-violence and neutral, while mBERT’s confusions are more balanced.The authors suggest generation may struggle more with minority classes than masked prediction.Source: §5.1 · p. 5
5.2 Results of summarisation task
- LEAD-64 performs well across summarization metrics, beating zero-shot mT5-base but trailing the Bloomz models in that setting.The authors interpret LEAD-64’s performance as evidence that an article’s opening can often provide an informative summary.Source: §5.2 · p. 5
- Zero-shot mT5-base has the lowest summarization scores, while Bloomz-3b slightly edges Bloomz-1b1, especially on bigram capture (R-2).The supplied Table 3 caption identifies Rouge scores for Bangla summarization but provides no cell values.Source: §5.2 · p. 5, Table 3 · p. 6
- With k=1 retrieval, summarization performance drops for mT5-base and both Bloomz models, with the sharper decline reported for Bloomz-3b.The authors suggest retrieved samples may add noise or distract the models, but state this as a possible explanation.Source: §5.2 · pp. 5–6
5.3 Analysis and Discussion
- Figure 3 reports deviations of main-method performance from zero-shot on the Vio-Lens test set for Bloomz-3b and mBERT using k=1 and k=3 demonstrations, based on 8 and 6 templates, respectively.Source: Figure 3 · p. 7
- The discussion highlights model and resource trade-offs: larger models may demand more computation, and more retrieval examples do not always improve performance.It also notes that stronger semantic and sentence-structure understanding may help classification of nuanced categories.Source: §5.3 · pp. 6–7
6 Ablation Study
Retrieval-prompt gains were stable across templates, while English prompts had the strongest overall macro average F1 among the language templates tested. Hindi retrieval data did not improve overall performance over the English-prompt method.
6.1 The Stability across Templates
- Across different templates, retrieval-augmented English prompts stably improved F1 over the Bangla zero-shot baseline for both Bloomz-3b and mBERT.mBERT showed greater average F1 improvement than Bloomz-3b.Source: §6.1 · p. 7
6.2 Impact of Bangla and Hindi Prompt Template
- English prompts had the highest macro average F1; Bangla prompts had the lowest, despite some category-specific gains.Hindi prompts improved precision and recall for some categories, including Neutral, but remained below English prompts in macro average F1.Source: §6.2 · pp. 7–8
6.3 Impact of Hindi sentence pool
- Hindi retrieval data improved mBERT’s retrieval of Neutral content in some comparisons, but neither mBERT nor Bloomz-3b outperformed the English-prompt method overall.Both models continued to show low Neutral recall and F1 regardless of sentence pool.Source: §6.3 · p. 8
7 Conclusion
The paper applies cross-lingual retrieval-augmented in-context learning to Bangla classification and summarization using BLOOM and BLOOMZ. Its experiments report superior classification F1 scores and identify cross-lingual retrieval as a significant contributor to performance.
7 Conclusion
- The approach was tested on two classification tasks and one summarization task using BLOOM and BLOOMZ for Bangla.The conclusion characterizes the method as cross-lingual retrieval-augmented in-context learning.Source: §7 · p. 8
- The experiments report superior F1 scores for classification, and further analysis identifies cross-lingual retrieval as a significant contributor to model performance.Source: §7 · p. 8
- The authors propose extending this work to more underrepresented languages and to tasks such as question answering or machine translation.They describe the approach as a foundation for further study of cross-lingual retrieval and in-context learning in low-resource languages.Source: §7 · p. 8
Limitations
The paper identifies limited exploration across model architectures, availability of retrieval datasets, and resource constraints as limitations.
Limitations
- The impact of retrieval augmentation across model architectures remains largely unexplored, while language-specific retrieval datasets and resources pose practical challenges.Source: Limitations · p. 8
A Appendix
The appendix lists results tables for prompt-template F1 scores on Vio-Lens and for Hindi-retrieval results on SentNoB.
A Appendix
- Table 5 covers F1-score results for eight prompt templates on the Vio-Lens test using BLOOMZ-3b.Source: Table 5 · p. 13
- Table 6 covers F1-score results for six prompt templates on the Vio-Lens test using mBERT.Source: Table 6 · p. 14
- Table 7 covers BLOOMZ-3b and mBERT results on the SentNoB test using a Hindi retrieval corpus.Source: Table 7 · p. 15