Cross-Lingual Retrieval Augmented Prompt for Low-Resource Languages

1 Introduction

• LRLs are under-represented in MPLM pretraining, and scarce domain- or task-specific annotations make fine-tuning difficult.The pretraining-finetuning paradigm has a high demand for domain-specific labeled data.Source: §1 · p. 1
• Prompt-based learning reformulates inputs and can add retrieved semantically similar examples as context without parameter updating.Earlier approaches discussed here focus on monolingual retrieval-augmented prompts.Source: §1 · p. 2
• PARC retrieves semantically similar cross-lingual sentences from HRL corpora to improve zero-shot transfer across three classification tasks.The study examines labeled and unlabeled HRL corpora, prompt counts, HRL choice, and robustness to retrieval methods and MPLMs.Source: Figure 1 · p. 1, §1 · p. 2

2 Related Work

• Cross-lingual retrieval can use dense sentence representations, including multilingual sentence transformers that produce semantically meaningful embeddings.The cited work describes both sparse bag-of-words and dense neural representations for retrieval.Source: §2 · p. 2
• Earlier retrieval-augmented prompting sampled semantically similar examples, but prior work focused on monolingual retrieval.The cited approaches used discrete prompts or, in one case, continuous soft prompts with supervised training.Source: §2 · p. 2
• Prior multilingual prompt-learning studies included finetuning or prompt tuning, whereas this work applies prompting to LRLs without parameter adjustment.The paper describes its cross-lingual retrieval-augmented discrete prompting as, to the authors’ knowledge, the first investigation of its kind.Source: §2 · pp. 2–3

3 Methodology

• The cross-lingual retriever ranks HRL sentences by semantic similarity to the LRL input, and PARC can retrieve from labeled or unlabeled corpora.The retrieved sentence is the k-th most similar HRL sentence to the LRL input.Source: §3 · p. 3
• A prompt pattern turns the retrieved HRL sentence into a cloze form, and a verbalizer maps class labels to words that fill its mask.The resulting labeled HRL sentence forms cross-lingual context for the prompted LRL input.Source: §3 · p. 3
• PARC concatenates the retrieved cross-lingual context with the prompted LRL input before passing the combined input to the MPLM.The MPLM predicts masked-token probabilities and selects the class whose verbalizer receives the highest probability.Source: §3 · p. 3
• In the unlabeled scenario, the retrieved HRL sample’s label is generated through self-prediction using prompt-based classification without cross-lingual context.In the labeled scenario, the HRL sample’s corpus label is used.Source: §3 · pp. 3–4
• PARC retrieves the K most similar HRL samples and uses the summed verbalizer probabilities to choose a class; concatenating contexts performed worse.The aggregation strategy is called Bag-of-Retrieval (BoR).Source: §3 · p. 4

4.1 Datasets

• The evaluation uses Amazon Reviews for binary sentiment analysis, AG News for topic classification across World, Sports, Business, and Tech, and XNLI for three-way entailment, neutral, or contradiction inference.The test set contains 10 low-resource languages, defined as languages with WikiSize below 7.Source: §4.1 · p. 4
• English task-specific labeled training sets serve as the retrieval pool, while English test sets are machine-translated into target languages for parallel evaluation.Translation quality and cross-lingual performance are checked against original Swahili and Urdu XNLI sets.Source: §4.1 · pp. 4–5

4.2 Baseline

• Baselines include majority guessing, random retrieval, direct prompting, and finetuning on retrieved high-resource examples.The translation baseline is not applicable to these machine-translated test sets; a different translation-baseline experiment is reported in the Appendix.Source: §4.2 · p. 5

4.3 Models

• A multilingual sentence transformer retrieves by cosine similarity, and multilingual BERT performs the cloze-style classification.The retriever is based on XLM-R and trained on parallel data from 50+ languages; mBERT was trained on Wikipedia in 104 languages.Source: §4.3 · p. 5

5 Results

• 4.6%, 10.4%, and 0.4% are PARC’s unlabeled gains over Direct on Amazon Reviews, AG News, and XNLI; labeled gains are 15.1%, 31.3%, and 2.7%.The Table 1 results are averaged across 10 evaluation LRLs with k=1; PARC also beats MAJ, Direct, and Random on all three tasks in both settings.Source: §5 · p. 5, Table 1 · p. 6
• 0.3%, 9.7%, and 1.3% are PARC-labeled’s gains over Finetune on Amazon Reviews, AG News, and XNLI, respectively.The paper considers this comparison fair because Finetune uses labels from retrieved examples.Source: §5 · p. 5
• PARC-labeled noticeably outperforms PARC-unlabeled, and the authors suggest more powerful MPLMs or better patterns might further improve results.The authors attribute the limit on self-prediction to mBERT’s capabilities; they note that NLI improves only slightly compared with topic categorization.Source: §5 · pp. 5–6
• On AG News, PARC beats Direct for every evaluated language in both settings, though retrieval sensitivity varies by language.Urdu and Uzbek gain less, while Javanese gains more; increasing k helps up to 30 except in Telugu and Swahili, which peak at 20.Source: §5 · p. 6

6.1 Effect of k

• Adding one retrieved sample causes an abrupt accuracy increase in labeled and unlabeled settings; labeled performance rises through k=20 and then levels off.Beyond k=1, unlabeled performance shows no clear improvement except for AGNews, while XNLI is less affected by k than the two single-sentence tasks.Source: §6.1 · p. 6

6.2.1 Measurement of Language Similarity

• Language similarity averages cosine similarities across five LANG2VEC feature vectors, while pretraining corpus size is represented by log2 Wikipedia size in MB.The five features are syntax, phonology, phonological inventory, language family, and geography.Source: §6.2 · p. 6, §6.2.1 · p. 7

6.2 Effect of Languages

• Table 2 reports AG News topic-categorization accuracy across 10 LRLs, with k, majority baseline, average accuracy, English retrieval, and BoR specified.The passage identifies the table’s scope and setup but does not provide its cell values.Source: Table 2 · p. 7

6.2.2 Correlation Analysis

• Performance is significantly positively correlated with language similarity and target-language pretraining size, but not source-language size.The analysis covers Amazon review performance in labeled and unlabeled settings; Table 3 reports correlations between performance and the three features.Source: §6.2.2 · p. 8, Table 3 · p. 8

6.3.1 Retriever and MPLM

• Even mBERT+pooling outperforms the Direct baseline on average in both settings; stronger retrievers and XLM-R yield higher accuracy in reported comparisons.The authors suggest that more powerful MPLMs could further improve PARC performance.Source: §6.3.1 · pp. 8–9, Table 4 · p. 8

6.3.2 Unseen Languages

• PARC is also reported effective for topic categorization on nine unseen languages, which were not included in the MPLM’s pretraining corpora.Table 5 presents results for these unseen languages on AG News.Source: Table 5 · p. 8, §6.3.2 · p. 9

6.4 Zero-shot Setting

• With a finetuned MPLM, PARC fails to outperform Direct; without finetuning, PARC performs better in both settings, and PARC-LB also outperforms its finetuned counterpart.The comparison concerns an experiment using English and Telugu on Amazon reviews.Source: §6.4 · p. 9, Table 6 · p. 9

6.5 Qualitative Analysis

• In a Telugu review example, adding two more English prompts changes an initially incorrect prediction to the correct positive class.The prediction remained incorrect after adding only one positive English sample.Source: §6.5 · p. 9, Table 7 · p. 9

• PARC retrieves semantically similar cross-lingual sentences from high-resource-language corpora to augment prompts for zero-shot learning on low-resource languages.The pipeline is evaluated on three classification tasks with parallel test sets across 10 low-resource languages.Source: §7 · p. 9
• PARC performs better than baselines in both unlabeled and labeled settings across three classification tasks and 10 low-resource languages.Source: §7 · p. 9
• Robustness results with other cross-lingual retrievers or multilingual pretrained language models suggest PARC may apply to a wider range of scenarios.Source: §7 · p. 10

Limitations

• PARC is not applicable to unseen languages with new scripts, and its retrieval phase requires more time and computational resources, making inference less efficient.Source: Limitations · p. 10

A Effect of Translations

• The translation analysis compares machine-translated and original XNLI test sets for Swahili and Urdu and uses the original sets as translation-quality gold standards.The passage says the analysis suggests machine-translated test sets are useful proxies for evaluating cross-lingual performance on low-resource languages.Source: A Effect of Translations · p. 13, Table 8 · p. 13

B Language Features

• The language-features table lists features for the 10 evaluated low-resource languages, including similarity to English and log Wikipedia size.The passage identifies SIM as the score for language similarity and WikiSize as the log value of Wikipedia size in MB.Source: B Language Features · p. 13

C Case Study

• The case study compares predictions with no retrieval information, one retrieved sample, and three retrieved samples for Amazon Review and AG News.Source: C Case Study · p. 13

• Detailed experimental results for Amazon reviews, AG News, and XNLI are provided in Tables 12, 13, and 14, respectively.Source: §D.1 · p. 13
• Table 9 lists language features for the 10 low-resource languages evaluated in the experiments.Source: Table 9 · p. 13
• Table 10 provides PARC examples for the Amazon Review task.Source: Table 10 · p. 13

• Table 16 reports detailed data for correlation analysis of language similarity, high- and low-resource pretraining-data sizes, and cross-lingual performance in unlabeled and labeled settings.Source: §D.2 · p. 13

D.3 Complete Results for Robustness Analysis

• Table 15 presents AG News topic-categorization results and defines k, the majority baseline, average accuracy, retrieval language, and retrieval strategies.BoR means Bag of Retrieval, and CONC means Concatenation.Source: Table 15 · p. 17
• Table 16 details 50 source-target language pairs for correlation analysis using Amazon review performance with k = 1.The analysis considers language similarity and source- and target-language pretraining data size, in unlabeled and labeled setups.Source: Table 16 · p. 18
• Table 17 reports results for different retriever and MPLM combinations on Amazon reviews (k = 5), AG News (k = 1), and XNLI (k = 3).The passage identifies the analysis scope but does not provide the table’s result values.Source: §D.3 · p. 13, Table 17 · p. 19
• Table 18 gives an overview of the three tasks’ test sets, with size indicating the number of samples for each low-resource language.Source: Table 18 · p. 19