PaperlayerView exact revision on arXiv

Source-linked AI summary

MLAIRE: Multilingual Language-Aware Information Retrieval Evaluation Protocal

Youngjoon Jang, Seongtae Hong, Hyeonseok Moon, Heuiseok Lim

arXiv:2605.07249v1cs.IR

TL;DR

Existing multilingual retrieval evaluation largely treats semantically equivalent passages as equally relevant, although users and RAG systems may depend on query-language evidence. MLAIRE addresses this by evaluating controlled multilingual pools with language-aware metrics and failure diagnostics, finding that standard metrics do not reliably indicate query-language preference across 31 retrievers.

  • Problem

    Existing MLIR metrics treat semantically equivalent passages as equally relevant regardless of language, so they do not measure whether retrievers prioritize query-language evidence.

  • Method

    MLAIRE constructs controlled pools with semantically equivalent passages across languages and evaluates semantic retrieval alongside LPR, Lang-nDCG, and a 4-way failure decomposition.

  • Results

    Across 31 retrievers, standard retrieval metrics do not reliably indicate query-language preference, with semantic strength and language preservation showing distinct trade-offs.

  • Takeaways & Limitations

    Language-aware diagnostics should complement standard semantic retrieval metrics when equivalent multilingual evidence is available.

  • Takeaways & Limitations

    LPR is not a universal utility measure for bilingual, code-switching, or settings where non-query-language information is preferable.

Abstract

from arXiv · show

Multilingual Information Retrieval is increasingly important in real-world search settings, where users issue queries over mixed-language corpora. Existing evaluations mainly reward language-agnostic semantic relevance, treating relevant passages equally regardless of language. Yet retrieval utility also depends on the language of the retrieved passages: users may prefer results they can read and verify in the query language, and query--passage language mismatch can complicate downstream grounding and answer verification in Retrieval-Augmented Generation systems. To evaluate this language-aware dimension, we introduce MLAIRE, a Multilingual Language-Aware Information Retrieval Evaluation protocol that disentangles cross-lingual semantic retrieval from query-language preference. MLAIRE constructs controlled pools with parallel passages across languages, enabling measurement of semantic retrieval accuracy and query-language preference when equivalent translations are available. We propose language-aware metrics, including Language Preference Rate (LPR) and Lang-nDCG, together with a 4-way decomposition separating semantic and query-language preference failures. Evaluating 31 dense, sparse, and late-interaction retrievers, we show that standard metrics obscure distinct behaviors: semantically strong retrievers may return correct content in a non-query language, while retrievers with stronger query-language preference may retrieve less semantically relevant passages.

1 Introduction

MLAIRE addresses a gap in multilingual retrieval evaluation: semantic relevance does not reveal whether systems prioritize passages written in the query language. It introduces controlled evaluation and language-aware diagnostics to expose this distinction and its practical relevance.

  • Motivation: Standard semantic relevance metrics do not distinguish query-language passages from equally relevant translations.They can show whether the right content was retrieved, but not whether the query-language version was prioritized.
  • Motivation: Figure 1 compares average nDCG with Language Preference Rate (LPR) across MLAIRE retrievers.PPLX-Embed-4B and BGE-M3 show strong retrieval performance but lower LPR than mE5-large, whereas BM25 and OpenSearch-NSE show high LPR with weaker nDCG.
  • Motivation: Retrieved-language choice affects users’ ability to read, verify, and act on evidence and can affect downstream RAG answer generation.Query–passage language mismatch requires cross-lingual interpretation while the generator answers in the user’s language.
  • Contribution: MLAIRE constructs candidate pools with semantically equivalent passages across languages, making query-language preference directly observable.It reports conventional retrieval metrics alongside LPR, Lang-nDCG, and a 4-way rank-1 decomposition.
  • Contribution: The evaluation covers 31 retrievers spanning dense, sparse, and late-interaction architectures.The study reports systematic mismatches between semantic retrieval quality and query-language preference.

2 Background

MLIR evaluation traditionally treats semantically equivalent passages as equally relevant regardless of language, leaving query-language preference unmeasured. The paper connects this gap to user-facing search and RAG, where language mismatch can affect usability and answer behavior.

  • MLIR evaluation: Existing MLIR metrics assess semantic relevance without distinguishing language versions of the same content.Recall and nDCG can identify the right content but cannot show whether the query-language passage was prioritized.
  • Why language matters: Query-language preference matters because retrieved-language choice affects whether users can read, verify, and act on a passage.Prior multilingual production-search studies report that users prefer results written in their query language.
  • Why language matters: In RAG, query–context language mismatch requires the generator to interpret cross-lingual evidence while answering in the user’s language.Reported effects include degraded answer correctness and reduced preservation of the expected response language.
  • RAG motivation: A controlled XQuAD experiment compares English and query-language gold passages using Qwen2.5-7B-Instruct across 12 query languages.It measures answer accuracy and language coherence, defined as answering in the query language.
  • RAG motivation: These observations motivate evaluating whether retrievers prioritize query-language evidence when equivalent relevant passages are available.

3 MLAIRE

MLAIRE creates controlled multilingual candidate pools containing equivalent passages, then measures both semantic retrieval and query-language preference. Its metrics and failure decomposition distinguish language-aware ranking behavior from ordinary relevance.

  • Dataset design: MLAIRE pairs each query with semantically equivalent relevant passages in multiple target languages.This exposes both cross-lingual semantic retrieval and preference for query-language evidence.
  • Dataset design: Each content group contains passages expressing the same underlying content across languages, with the query-language passage designated as the preferred relevant passage.
  • Dataset design: The evaluation uses Belebele, XQuAD, and MLQA, which provide parallel or partially parallel multilingual QA passages.Belebele contains 122 language variants, XQuAD covers 12 languages, and MLQA covers 7 languages with partial parallelism.
  • Language-aware metrics: LPR measures how often the highest-scoring passage within the target content group is written in the query language.It compares the query-language version with its cross-lingual alternatives.
  • Language-aware metrics: Lang-nDCG assigns higher relevance to passages matching both the target content group and the query language.Its grading gives 3 to query-language target passages, 2 to cross-lingual target passages, and 0 otherwise.
  • Failure analysis: The 4-way top-1 decomposition separates perfect, lang_fail, sem_fail, and both_fail outcomes.Aggregating these categories identifies whether errors reflect language mismatch, semantic mismatch, or both.

4 Experimental Setup

The experiments evaluate a broad set of multilingual retrievers across dense, sparse, and late-interaction paradigms. Models vary substantially in architecture and scale, while retrieval is performed independently over each MLAIRE dataset.

  • Retriever pool: The study evaluates 31 publicly available multilingual retrievers across dense, sparse, and late-interaction paradigms.The pool includes models from 100M to 8B parameters, plus sparse and late-interaction baselines.
  • Retriever paradigms: Dense retrievers independently encode queries and passages into fixed-dimensional vectors scored by cosine similarity.The evaluated families include multilingual-e5, bge-m3, gte, snowflake-arctic, nomic-embed, embeddinggemma, jina, Qwen3-Embedding, llama-nemotron, and pplx-embed.
  • Retriever paradigms: Sparse retrieval includes a subword lexical BM25 baseline and a neural sparse model scored by inner product.BM25 uses XLM-RoBERTa tokenization with k1=1.2 and b=0.75.
  • Retriever paradigms: Late-interaction retrieval uses token-level vectors and MaxSim scoring for jina-colbert-v2 and LFM2-ColBERT-350M.Passages are indexed with PLAID for scalable evaluation.
  • Evaluation procedure: Each model–dataset pair retrieves top-k passages from the dataset’s full corpus, with k set above the maximum number of relevant passages.The study uses k=20 for MLQA and XQuAD, and k=200 for Belebele.

5 Results and Analysis

Across 31 retrievers, semantic retrieval quality and query-language preference emerge as distinct behavioral axes. Results show trade-offs across retrieval paradigms, query languages, training supervision, and structured language-group failures.

  • Main Results: 31 retrievers show that standard semantic retrieval quality does not reliably predict query-language preference.The paper reports weakly negative nDCG–LPR associations across MLQA, XQuAD, and Belebele.
  • Paradigm-Level Patterns: Retrieval paradigms exhibit different trade-offs: multilingual-e5 has stronger LPR despite lower nDCG, while jina-colbert-v2 has competitive semantics and LFM2-ColBERT preserves query language more strongly.Sparse systems also show high LPR but lower standard nDCG than the strongest dense and late-interaction retrievers.
  • Role of Training Dataset: Training-data language composition plausibly shapes the trade-off between semantic alignment and language preservation.Cross-lingual relevance supervision favors cross-language alignment, whereas same-language contrastive training is consistent with stronger LPR.
  • Top-1 Failure Decomposition: Strong semantic retrieval can return relevant passages in the wrong language, producing high combined perfect and lang_fail outcomes for Qwen3-Embedding-8B and jina-colbert-v2.Standard nDCG counts semantically relevant wrong-language passages as successful.
  • Top-1 Failure Decomposition: High query-language preference can coexist with weaker semantic retrieval: BM25 and OpenSearch-NSE rarely produce lang_fail but show larger sem_fail and both_fail proportions.Their language preservation can partly reflect lexical anchoring toward same-language passages.
  • Query-Language Variation in LPR: Query-language preference varies substantially across languages: Qwen3-Embedding and jina-colbert-v2 show low LPR for several languages but higher LPR for others.The multilingual-e5 family, BM25, and LFM2-ColBERT remain near the LPR ceiling for most XQuAD languages.
  • Directional Query-Language Mismatch: LPR failures are structured by linguistic affinity, with wrong-language selections often concentrated within the same macro language group.Figure 4 maps query-language groups to retrieved document-language groups, while model-specific attractions can also cross group boundaries.

6 Conclusion

MLAIRE separates semantic retrieval quality from query-language preference and shows that these dimensions can diverge across multilingual retrievers.

  • 6 Conclusion: Across 31 retrievers, standard retrieval metrics do not reliably indicate whether models retrieve evidence in the query language.Dense, sparse, and late-interaction retrievers exhibit distinct trade-offs between semantic coverage and language alignment.
  • 6 Conclusion: MLAIRE uses top-1 failure decomposition, per-language LPR analysis, and macro-group transition analysis to characterize non-query-language retrieval.The analyses show that such retrieval is often structured rather than random.
  • 6 Conclusion: The protocol evaluates multilingual retrievers by both what they retrieve and whether their behavior aligns with users’ language expectations.This frames query-language preference as a complementary dimension alongside semantic relevance.

A Limitations

The limitations concern MLAIRE’s interpretation, controlled dataset construction, metric sensitivity, motivating RAG analysis, and broader fairness implications.

  • Scope of query-language preference: LPR and Lang-nDCG treat query-language passages as preferred when equivalent passages exist, so they are complementary diagnostics rather than universal utility measures.Cross-lingual evidence may be equally useful or preferable for bilingual, code-switching, or non-query-language-rich settings.
  • Dataset scope: The benchmark’s parallel QA-style datasets may not represent open-domain mixed-language corpora with varied document properties and native authorship.The paper calls for evaluation on ad hoc collections with native relevance judgments and naturally occurring mixed-language pools.
  • Partial parallelism for MLQA: MLQA’s partial parallelism yields fewer competing translations, which can affect both LPR and Lang-nDCG.Belebele and XQuAD are fully parallel, whereas MLQA is only partially parallel.
  • Considerations for new metrics: LPR can increase through language-identification or surface-form cues without improving semantic retrieval.The paper therefore pairs LPR with standard nDCG, Lang-nDCG, and the top-1 4-way decomposition.
  • RAG motivation experiment: The RAG experiment is motivational and does not exhaustively vary retrieval depth, chunking, prompts, decoding, or answer normalization.Future work should test retrieved rather than gold passages, multiple generators, and broader decoding configurations.
  • Fairness and language bias: Query-language preference overlaps with fairness but is not identical: high LPR can still return semantically weak evidence, while cross-lingual retrieval may benefit some users.MLAIRE is intended to diagnose trade-offs rather than enforce monolingual retrieval.

B Ethical Considerations

The ethical considerations emphasize representativeness, context-sensitive use of LPR, and the trade-off between accessible same-language evidence and broader cross-lingual coverage.

  • Representativeness: Benchmark coverage remains uneven across language families, scripts, and resource levels, so results are controlled measurements rather than comprehensive equity evidence.High-resource languages and translated content may be overrepresented relative to natural multilingual search.
  • Responsible use of LPR: LPR is most appropriate when users are expected to read or verify evidence in the query language, not as a standalone deployment objective.Operational use should combine it with semantic metrics and application-specific needs.
  • Fairness trade-offs: Optimizing query-language preference can improve accessibility for same-language users but penalize systems intentionally retrieving cross-lingual evidence for broader coverage.MLAIRE is therefore diagnostic rather than a single optimization target.
  • Positive impact: MLAIRE can help developers assess whether retrievers provide evidence users can read and verify in multilingual search and RAG systems.This supports accessibility and transparency when interpreted alongside application needs.

D Infrastructure and Reproducibility

The experiments used specified hardware and isolated software environments, with runtime varying by retriever architecture and dataset size.

  • Infrastructure: All experiments ran on a single workstation with four NVIDIA RTX A6000 GPUs and 1 TB of system RAM.Each GPU has 48 GB of memory.
  • Software environments: The evaluator used Python 3.10 and PyTorch 2.8 with CUDA 12.1 across three isolated virtual environments.Pinned requirements avoided dependency collisions while sharing the same evaluation core.
  • Runtime: The reported experiments required approximately 50 GPU-hours and under 1 CPU-hour for the BM25 baseline.The largest cost came from encoding and indexing the 122-language Belebele corpus.

E Language Classification

The paper classifies languages by resource level and groups Belebele variants into broad macro language groups for directional analysis.

  • Resource-level classification: Resource-level classification maps SIB-200 classes 0–2 to low-resource, class 3 to mid-resource, and classes 4–5 to high-resource languages.
  • Resource-level classification: Languages without a Joshi-style class use the NLLB-200 high/low-resource distinction as a fallback.
  • Macro language groups: The directional analysis groups 122 Belebele language variants into ten macro language groups.
  • Macro language groups: These groups aggregate languages by broad family, script, and regional affinities for visualizing group-level non-query-language retrieval patterns, not as a strict linguistic taxonomy.

F Recall and Lang-Recall

Recall measures multilingual semantic retrieval, whereas Lang-Recall isolates recovery of the query-language passage. Their divergence varies by retriever and is largest on Belebele because its relevant groups contain many language variants.

  • Metric definitions: Recall@k counts semantically relevant passages in the top k regardless of language, while Lang-Recall@k checks whether the query-language version appears there.
  • High-LPR retrievers: 85.17 versus 43.97 on MLQA and 99.87 versus 72.59 on XQuAD are multilingual-e5-large’s Lang-Recall@20 and standard Recall@20 values, respectively.
  • High-LPR retrievers: 98.56 versus 13.94 on XQuAD are BM25’s Lang-Recall@20 and standard Recall@20 values, respectively.
  • Lower-LPR retrievers: Lower-LPR retrievers such as pplx-embed-v1-4b show smaller Recall–Lang-Recall gaps, consistent with distributing retrieval capacity across equivalent passages in multiple languages.
  • Largest gap in Belebele: Belebele exhibits the largest gap because each query can have up to 122 semantically relevant language variants, enlarging the standard relevant set.
Loading 2605.07249v1…