MINERS: Multilingual Language Models as Semantic Retrievers

1 Introduction

• Prior multilingual embedding analysis in MTEB was limited to bitext mining, while code-switching retrieval and semantic-retrieval benchmarking remained underexplored.Existing code-switching evaluations focused on fine-tuning, and prior semantic-similarity work had not explored generative LMs as context providers.Source: §1 · p. 1
• MINERS is a benchmark for evaluating multilingual LMs as semantic retrievers across 200+ languages, 11 encoder LMs, and 11 generative LMs.It covers bitext retrieval, retrieval-based classification, and ICL classification without fine-tuning.Source: §1 · pp. 1–2
• MINERS is described as adaptable and scalable across models, consolidating their scores to compare task performance and reveal different approaches’ strengths and weaknesses.Source: §1 · p. 2
• MINERS analyzes performance across monolingual, cross-lingual, and code-switching settings and varies the number of retrieved samples.The analysis examines how sample quantity affects retrieval effectiveness.Source: §1 · p. 2
• The paper compares retrieval methods with conventional fine-tuning and reports comparable performance without training.Retrieval uses pretrained models for semantic retrieval tasks.Source: §1 · p. 2

2.1 Motivation

• The benchmark covers high- and low-resource languages, unseen languages, and code-switching, and is designed to be extendable, reproducible, and inference-only.Its stated aspects are language diversity, usefulness across three tasks, and efficiency.Source: §2.1 · p. 2

2.2 Tasks

• Bitext retrieval searches for semantically similar parallel samples; retrieval-based classification votes over retrieved training labels, while ICL classification supplies retrieved examples as few-shot context.For ICL, predictions use normalized class probabilities when available, or direct label prediction otherwise.Source: §2.2 · pp. 2–3

2.3 Settings

• The evaluation settings are monolingual, code-switching, cross-lingual, and cross-lingual code-switching.Cross-lingual evaluation uses one source language and the remaining languages as targets; cross-lingual code-switching evaluates code-switched data in that context.Source: §2.3 · pp. 3–4

2.4 Datasets

• The benchmark includes 11 datasets—seven multilingual and four code-switching—covering parallel and classification data for the retrieval and ICL tasks.The generative models use a unified English prompt template for ICL.Source: §2.4 · p. 4

2.5 Models

• The evaluation covers encoder LMs and commercial embedding APIs, generative LMs and commercial APIs, alongside DistFuse ensembles that merge models’ label-distribution distance scores through a linear combination to improve retrieval results.The reported two-model DistFuse setting uses LaBSE and E5LARGE.Source: §2.5 · pp. 4–5

3.1 Bitext Retrieval

• DistFuse improves bitext retrieval by up to 2.63% over LaBSE and 1.72% over OpenAI-Embedv3, and performs strongly across settings.DistFuse (2) is identified among the top performers; LaBSE ranks highest among open-source models. XLM-R and Glot-500 struggle in bitext retrieval but perform better in retrieval-based classification, while code-switching remains challenging.Source: §3.1 · p. 5

3.2 Retrieval-based Classification Results

• Cohere-Embedv3 API leads models by 1.95% on average, with LaBSE close behind at 1.15%; retrieval also outperforms fine-tuned models in XL and CS without tuning.Merged model scores boost prediction accuracy, especially in Mono and XL, while CS and XL-CS performance remains lower than API models.Source: §3.2 · pp. 5–6

3.4 Performance Dynamics Over k

• Increasing k consistently improves bitext retrieval and retrieval-based classification performance (Figure 2).A larger k provides more bilingual pairs for bitext retrieval and more contextual examples for label prediction through majority voting.Source: Figure 2 · p. 6, §3.4 · p. 7

3.3 ICL Classification Results

• In one-shot ICL, BLOOMZ 3B, with 6× more parameters, beats the top BLOOMZ 560m model by 2.21%, but increasing complexity lowers CS and XL-CS performance.GPT-3.5 Turbo and GPT-4o outperform other models, particularly in CS and XL-CS; possible prior exposure to these datasets is unclear.Source: §3.3 · p. 6

4.1 Model Representation

• E5LARGE forms sample-ID clusters across languages, while XLM-RBASE groups more by language; both separate classes, with clearer separation for E5LARGE.The representation patterns align with bitext retrieval results for XLM-RBASE.Source: §4.1 · p. 7

4.2 Samples Relevance

• As retrieval percentiles decrease, ICL performance improves consistently across all three BLOOMZ models on NusaX.The figure defines lower percentiles as more semantically relevant samples; the authors emphasize retrieving highly relevant examples.Source: §4.2 · p. 7, Figure 4 · p. 8

4.3 Compute Efficiency

• Retrieval-based classification relies mainly on forward-pass embeddings and is more efficient than fine-tuning, while ICL can incur higher inference costs.For very large LMs, ICL inference can exceed fine-tuning cost; as training data grows, fine-tuning complexity eventually surpasses ICL inference.Source: §4.3 · p. 8

4.4 Bitext Retrieval is Unsymmetrical

• Bitext retrieval is asymmetric: non-English-to-English retrieval tends to outperform the reverse direction.Table 5 reports F1@1 for two source-to-target language directions.Source: §4.4 · p. 8, Table 5 · p. 8

Dense Retrieval via LM

• Dense retrieval research has developed contrastive and alignment objectives, while late interaction and in-batch negatives offer other ways to estimate relevance.Examples include Siamese contrastive learning, SimCSE loss, and late interaction.Source: §5 · p. 8

Semantic Retrieval for NLP Tasks

• The cited passage states that retrieving labels using semantic retrieval has proven beneficial for classification, and reports that Bari et al. (2021) enhance accuracy with cross-lingual few-shot nearest-neighbor retrieval.Source: §5 · p. 8
• MINERS adopts an efficient retrieval approach supporting multiple models and extends it to ICL for broader utility and accessibility.The approach builds on Winata et al. (2023a).Source: §5 · p. 9

• MINERS evaluates multilingual LMs on bitext retrieval, semantic-search classification, and retrieval-augmented classification contexts.The benchmark assesses retrieval robustness across more than 200 languages.Source: §6 · p. 9
• Retrieving semantically similar vector representations achieves performance comparable to state-of-the-art fine-tuned approaches without fine-tuning.Source: §6 · p. 9
• The paper examines representation mechanisms to inform more efficient and accurate label retrieval methods.Source: §6 · p. 9

• Ensemble techniques such as DistFuse and target-language prompts are proposed as avenues to improve ICL classification performance.Source: Limitations · p. 9
• Only one prompt template was tested, and results could differ with other prompts.Exploring varied prompts is deferred to future work.Source: Limitations · p. 9
• The evaluation covers a select set of models and datasets, leaving broader comparative coverage for future work.The authors describe this as an initial assessment phase.Source: Limitations · p. 9

• The stated evaluation aims are transparency and fairness, with attention to implications for diverse multilingual communities.Source: Ethical Considerations · p. 9

A.1 Baselines

• Baselines include literature SOTA results, random and majority classifiers, and fine-tuned XLM-RBASE, with task-specific benchmark datasets.Source: §A.1 · p. 13

A.2.1 Preprocessing

• FIRE 2020 labels are consolidated into common categories, while MASSIVE label underscores are replaced with spaces.Source: §A.2.1 · pp. 13–14

A.2.2 Statistics

• Dataset statistics are reported in Table 6, with equal language counts used for NollySenti bitext retrieval and no LinCE SA statistics because its test set was unused.Source: §A.2.2 · p. 14, Table 6 · p. 15

A.3 Languages Under Study

• Cross-lingual source and target language pairs are listed with each dataset’s original language-code conventions preserved.Source: §A.3 · p. 14, Table 7 · p. 15

A.4 LM Sources

• Evaluated models include open-source encoder and generative LMs, alongside GPT-3.5 Turbo, GPT-4, GPT-4o, and Cohere-Embedv3 API models.Source: §A.4 · p. 14

A.5 LM Inference

• Inference runs use 8-bit quantization and test k=1, 5, and 10 retrieved samples; ICL uses zero-shot and one-shot settings with either label-probability scoring or direct generation.The chosen ICL method depends on whether next-token logits are accessible for the model.Source: §A.5 · p. 14

A.6 Hyper-parameters

• Inference and fine-tuning comparisons use separately specified hyperparameters, and fine-tuning omits warmup steps with a linear scheduler.Source: §A.6 · p. 14

B.1 LM Representation Visualization

• In a 200-sample NusaX visualization, LaBSE and Cohere-Embedv3 align samples across languages and group them by class, consistent with E5 models.Figure 5 identifies sample ID, language, and class as the color encodings across its panels.Source: Figure 5 · p. 13, §B.1 · pp. 14–15

B.2 Retrieved Samples

• Four of five retrieved labels were accurate, whereas XLM-RBASE retrieved samples had only two of five labels correct, indicating differences in sample quality and label accuracy across models.Source: §B.2 · p. 15

C.1 Bitext Retrieval Results

• Bitext retrieval performance generally trends upward as the number of retrieved samples k increases.Table 13 reports results for each dataset and model.Source: §C.1 · p. 16

C.2 Retrieval-based Classification Results

• Indo-European languages achieve the highest SIB-200 accuracies, while Latin-script results are generally highest but have the greatest standard deviation.Afro-Asiatic, Austroasiatic, and Sino-Tibetan families show the greatest standard deviations; Nkoo, Olck, Tibt, and Tfng scripts show the lowest performance.Source: §C.2 · p. 16, Table 14 · p. 21, Table 15 · p. 22
• Inference and fine-tuning baseline hyperparameters are documented separately for the classification evaluations.Source: Table 10 · p. 17, Table 9 · p. 17

C.3 ICL Classification Results

• Table 16 reports ICL classification results for Mono, XS, CS, and XS CS settings.Source: §C.3 · p. 17, Table 16 · p. 23

D Prompt Examples

• The ICL prompts use label options for direct prediction and omit them when predicting from label probabilities.Example prompts are provided for LaBSE and E5LARGE with k=1.Source: D Prompt Examples · p. 17, Table 17 · p. 24, Table 18 · p. 25

E DistFuse

• DistFuse weight tuning explores only a few combinations for its two- and three-model variants because of time constraints.The tested combinations vary α, β, and γ weights.Source: E DistFuse · p. 17