PaperlayerView exact revision on arXiv

Source-linked AI summary

Evaluating Large Language Models for Cross-Lingual Retrieval

Longfei Zuo, Pingjun Hong, Oliver Kraus, Barbara Plank, Robert Litschko

arXiv:2509.14749v1cs.CLcs.IR

TL;DR

Cross-lingual LLM reranking has lacked systematic large-scale evaluation, particularly beyond translation-based first-stage retrieval. This study compares multilingual retrievers and LLM rerankers across CLIR settings. It finds that bi-encoders can improve gains, translation benefits diminish with stronger rerankers, and instruction-tuned pairwise models perform competitively with listwise rerankers.

  • Problem

    Large-scale comparisons of LLM rerankers for CLIR are lacking, and prior LLM-based CLIR work relies on machine translation to bridge the language gap during retrieval.

    Source: §1 · p. 1

  • Method

    The study evaluates passage- and document-level CLIR by comparing multilingual bi-encoders and MT-based lexical retrieval alongside listwise and pairwise LLM rerankers.

    Source: §1 · p. 2, §5 · p. 9

  • Results

    The evaluation finds that multilingual bi-encoders and hybrid retrieval can yield further reranking gains, document-translation benefits diminish with stronger listwise rerankers, and instruction-tuned pairwise rerankers perform competitively with listwise models.

    Source: §5 · p. 9

  • Takeaways & Limitations

    The results identify retriever choice, translation, reranking strategy, document length, and language-resource disparities as factors associated with CLIR reranking performance.

    Source: §5 · p. 9

    The evaluation uses a fixed set of LLMs, may be affected by translation quality, and is limited to zero-shot reranking without fine-tuning or domain adaptation.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Multi-stage information retrieval (IR) has become a widely-adopted paradigm in search. While Large Language Models (LLMs) have been extensively evaluated as second-stage reranking models for monolingual IR, a systematic large-scale comparison is still lacking for cross-lingual IR (CLIR). Moreover, while prior work shows that LLM-based rerankers improve CLIR performance, their evaluation setup relies on lexical retrieval with machine translation (MT) for the first stage. This is not only prohibitively expensive but also prone to error propagation across stages. Our evaluation on passage-level and document-level CLIR reveals that further gains can be achieved with multilingual bi-encoders as first-stage retrievers and that the benefits of translation diminishes with stronger reranking models. We further show that pairwise rerankers based on instruction-tuned LLMs perform competitively with listwise rerankers. To the best of our knowledge, we are the first to study the interaction between retrievers and rerankers in two-stage CLIR with LLMs. Our findings reveal that, without MT, current state-of-the-art rerankers fall severely short when directly applied in CLIR.

1 Introduction

Cross-lingual retrieval commonly uses machine translation, but translation adds latency and may be unavailable or unreliable for low-resource languages. This study evaluates LLM reranking beyond translation-dependent setups, comparing first-stage retrievers and reranking approaches across CLIR settings.

1 Introduction

  • CLIR retrieves documents in a language different from the query; machine translation has been a dominant way to convert the task into monolingual retrieval.Translation increases query latency and can be unavailable or unreliable for low-resource languages, with errors adversely affecting retrieval.Source: §1 · p. 1
  • Figure 1 distinguishes query-translation and document-translation monolingual routes from original-document cross-lingual routes that omit translation in retrieval and reranking.The paper summarizes its systematic evaluation and releases code and resources for reproducibility.Source: Figure 1 · p. 1, §1 · p. 2
  • The study moves beyond MT-dependent CLIR evaluation by comparing state-of-the-art bi-encoders with MT-based lexical retrieval and examining translation effects on reranking.It also compares listwise and pairwise LLM rerankers on CLIR.Source: §1 · p. 2
  • The evaluation asks how dense bi-encoders and BM25 perform as first-stage retrievers, how retriever–reranker interactions vary by language-resource level, and how reranking strategy and document length matter.These are the study’s four stated research questions.Source: §1 · p. 2

2 Related Work

The section situates the study in multi-stage retrieval and LLM reranking, identifying limited cross-lingual evidence and describing how this work broadens comparisons across retrievers, reranking strategies, and language settings.

2.1 Multi-Stage Retrieval

  • Multi-stage retrieval first filters candidates with a fast retriever, then reranks them for precision; prior work finds first-stage quality affects reranking and smaller, stronger candidate sets can suffice.LLM rerankers are costly over full candidate pools, motivating multi-stage retrieval.Source: §2.1 · p. 2
  • The study presents a large-scale comparison of LLM-based retrievers and rerankers across retrieval paradigms and language pairs.Earlier first stages commonly used BM25 with document translation, while newer work employs bi-encoders.Source: §2.1 · p. 2

2.2 Reranking Approaches with LLMs

  • Listwise LLM rerankers rank multiple documents in one prompt, but their cross-lingual performance remains poorly understood despite open-source models distilled from closed-source systems.RankGPT uses instruction-tuned LLMs for passage reranking; open-source examples include RankVicuna, RankZephyr, and Rank-without-GPT.Source: §2.2 · p. 2
  • This study extends pairwise reranking to cross-lingual retrieval with multilingual decoder-only LLMs and compares it with listwise rerankers across language-resource settings and non-English query pairs.Prior evaluation covered English-to-African language pairs with translation-based first-stage retrieval; the experiments adopt pairwise and listwise strategies rather than pointwise reranking.Source: §2.2 · pp. 2–3

3 Evaluation Framework

The framework evaluates first-stage retrieval and LLM reranking across translation-based and original-language routes. It uses CLEF and CIRAL datasets, multiple retriever and reranker types, and translated documents to compare cross-lingual and monolingual settings.

3 Evaluation Framework

  • The framework compares translation-based lexical retrieval with multilingual bi-encoders, listwise and pairwise rerankers, and the impact of translation on reranking.The experiments are designed to quantify whether first-stage retrieval gains carry through to reranking.Source: §3 · p. 3

3.1 Datasets

  • CLEF 2003 covers nine European-language pairs with 60 parallel queries, while CIRAL Test Set A covers English queries and Hausa, Somali, Swahili, and Yoruba passages.CIRAL passages average 139 whitespace-delimited tokens, compared with 274 for CLEF documents.Source: §3.1 · p. 3

3.2 Multi-Stage Pipeline: Experimental Setup

  • The first stage uses BM25 with translated documents and five bi-encoders retrieving up to 100 documents; second-stage experiments include listwise RankZephyr and RankGPT models and pairwise Llama-3.1-8B-Instruct and Aya-101.The pairwise models differ in language coverage across the evaluated languages.Source: §3.2 · pp. 3–4

3.3 Document Translation Setup

  • For monolingual comparisons, documents are translated with NLLB-200-1.3B; sentence-level translation is used to improve accuracy and reduce repetitions relative to whole-document translation.Table 2 reports CLEF MAP for first-stage retrieval and listwise reranking, with OG, QT, and DT distinguishing original-language, query-translated, and document-translated settings.Source: §3.3 · p. 4, Table 2 · p. 5

4 Results and Discussion

First-stage retriever performance varies by dataset, and reranking gains depend on the retriever, language setting, and reranking approach. Pairwise and listwise LLM rerankers show different strengths, while document-length effects depend on the model and retrieval setup.

4.1 First-Stage Retrieval

  • On CLEF, NV-Embed-v2 leads the bi-encoders at 0.323 MAP, while on CIRAL M3 leads and BM25-DT trails most bi-encoders.The best-performing first-stage ranker differs considerably by dataset.Source: §4.1 · p. 5

4.2 Second-Stage Reranking

  • On average, listwise rerankers improve their input rankings, but only RankGPT4.1 improves across all language pairs on both datasets.RankZephyr and RankGPT3.5 improve NV-Embed-v2 rankings for most CLEF language pairs, with less pronounced gains over BM25-DT.Source: §4.2 · p. 6
  • Reranking gains track first-stage quality: M3 exceeds BM25-DT by +0.110 MAP on CIRAL, with larger reranking gains than the smaller CLEF retrieval gap.The translation benefit also diminishes with stronger rerankers: on CLEF, RankGPT4.1 gains only +0.003 MAP from document translation, versus +0.026 for RankZephyr and +0.011 for RankGPT3.5.Source: §4.2 · p. 6
  • On CLEF with NV-Embed-v2, Llama-3.1-8B-Instruct scores 0.371 with and 0.385 without translation, outperforming RankZephyr and RankGPT3.5.Aya-101 also outperforms both listwise models on translated documents.Source: §4.2 · p. 7
  • On CIRAL with M3, Aya-101 beats RankZephyr and RankGPT3.5 only on original-language documents, while GPT4.1 outperforms both pairwise models.The pairwise rerankers are instruction-tuned LLMs without additional reranking-data post-training, unlike the listwise models used here.Source: §4.2 · p. 7
  • All three rerankers improve document-translated rankings on both datasets, but original-language reranking sometimes underperforms its first-stage ranking.This suggests reranking may introduce additional noise in some cross-lingual settings.Source: §4.2 · p. 8
  • Even state-of-the-art rerankers remain far from the oracle upper bound, especially in cross-lingual settings, despite translation narrowing the gap.The highest realized improvements still fall short of expectations on both datasets.Source: §4.2 · p. 8

4.3 Impact of Document Length

  • RankZephyr performs best at 128 tokens and drops at 256, while Llama-based pairwise rerankers are more stable or improve with longer inputs under dense retrieval.Optimal input length is task- and model-dependent and may require careful tuning; the authors hypothesize pairwise comparisons can use longer-document context more effectively.Source: §4.3 · p. 8

5 Conclusion

The evaluation finds that multilingual bi-encoders and hybrid retrieval can improve cross-lingual reranking over lexical MT-based retrieval, while translation benefits diminish with stronger listwise rerankers. Instruction-tuned pairwise rerankers are competitive with listwise rerankers, and performance is sensitive to document length and language-resource disparities.

5 Conclusion

  • Multilingual bi-encoders and hybrid retrieval can improve cross-lingual reranking over lexical MT-based retrievers.The study also finds that document-translation benefits diminish with stronger listwise rerankers.Source: §5 · p. 9
  • Instruction-tuned pairwise rerankers perform competitively with listwise rerankers such as RankZephyr.Source: §5 · p. 9
  • Reranking performance is sensitive to document length and language-resource disparities.Source: §5 · p. 9
  • The study calls for more robust approaches to cross-lingual reranking without relying on machine translation.Source: §5 · p. 9

Limitations

The evaluation is limited to a fixed set of LLMs, translation quality may confound results, and experiments use zero-shot reranking without fine-tuning or domain adaptation.

Limitations

  • The evaluation covers a fixed set of LLMs that may not represent the full diversity of available open-source or commercial models.Source: Limitations · p. 9
  • Translation quality remains a potential confounder when comparing document and query translation strategies.Source: Limitations · p. 9
  • Experiments are restricted to zero-shot reranking without fine-tuning or domain adaptation, which may underestimate downstream potential.Source: Limitations · p. 9

A Prompt Template for Listwise and Pairwise Reranking

The appendix presents the prompt templates used for listwise and pairwise LLM reranking, drawing on prior implementations for each approach.

A Prompt Template for Listwise and Pairwise Reranking

B Query and Document Translation

The study compares query and document translation across CLEF and CIRAL, using document translation to form second-stage candidate pools because it performs better. Translation effects vary by benchmark and stage: query translation is less effective for low-resource African languages, while document translation has a larger reranking effect on CIRAL than CLEF.

B Query and Document Translation

  • Query translation is less effective on CIRAL than CLEF, where queries translated into higher-resource languages have more reliable translation and lexical overlap.On CIRAL, translated queries can be short or low-quality and struggle to match document content.Source: B Query and Document Translation · p. 13
  • Document translation has a more pronounced reranking effect on CIRAL, where short documents support more accurate, less noisy translation and English text better matches LLM training.Source: B Query and Document Translation · p. 13
  • The original-language setting is less problematic on CLEF, where languages are better supported and longer, descriptive queries are easier to interpret, narrowing the gap with document translation.Source: B Query and Document Translation · p. 13

C Hybrid Retrieval Experiments

Hybrid retrieval produces mixed oracle results across CLEF and CIRAL, while listwise and pairwise reranking results show different gains from translation. Pairwise rerankers improve over reranking NV-Embed-v2 and M3 results, with the best model differing by dataset.

Hybrid retrieval

  • The hybrid approach reranks fused top-100 rankings using reciprocal rank fusion, and the analysis is limited to open-source LLMs.For a fair comparison, the fused ranking is capped at 100 documents because the union of the two top-100 sets can exceed 100.Source: C Hybrid Retrieval Experiments · p. 13
  • Hybrid retrieval raises CLEF's oracle MAP to 0.611, above NV-Embed-v2's 0.586 and BM25-DT's 0.560; on CIRAL, its oracle falls below M3, though its MAP is 0.403.The passage attributes the cross-dataset difference partly to the larger bi-encoder versus lexical-retriever gap on CIRAL: 0.392 vs. 0.282, compared with 0.323 vs. 0.308 on CLEF.Source: C Hybrid Retrieval Experiments · pp. 13–14

Listwise reranking

  • RankZephyr's CLEF MAP rises from 0.342 to 0.384 without translation and from 0.368 to 0.410 with translated documents; on CIRAL, it rises from 0.327 to 0.390 and from 0.407 to 0.495, respectively.The authors report these gains as consistent with Section 4 and say retrieval improvements translate to reranking improvements.Source: C Hybrid Retrieval Experiments · p. 14

Pairwise reranking

  • For pairwise rerankers, translation's improvements diminish; Llama-3.1-8B-Instruct and Aya-101 outperform reranking only NV-Embed-v2 and M3 results, with Llama-3.1-8B (DT) best on CLEF and RankZephyr (DT) on CIRAL.Source: C Hybrid Retrieval Experiments · p. 14

D Model Information

The study uses retrievers with different training and language coverage, alongside RankZephyr, whose training corpus is limited to monolingual English data.

Retriever models

  • RepLlama is fine-tuned from Llama-2-7B with LoRA on the MS MARCO Passage Ranking training split for one epoch.The passage identifies the training split as the Bajaj et al. (2018) dataset.Source: D Model Information · p. 15
  • E5 is initialized from XLM-RoBERTa-large and trained on a mixture of multilingual datasets covering 100 languages; NV-Embed-v2 is built on decoder-only Mistral-7Bv0.1.Source: D Model Information · p. 15

Reranker models

  • RankZephyr is based on Zephyr-7B-ß, distilled from RankGPT3.5 and RankGPT4, and trained on monolingual English data.The distillation uses 100K training queries from RankGPT3.5 and 5K queries from RankGPT4.Source: D Model Information · p. 15
Loading 2509.14749v1…