PaperlayerView exact revision on arXiv

Source-linked AI summary

LAMAR: An Open Language-Aware Multilingual Alignment Reranker

Seongtae Hong, Youngjoon Jang, Jungseob Lee, Seungyoon Lee, Heuiseok Lim

arXiv:2607.22042v2cs.IR

TL;DR

Existing multilingual rerankers do not consistently prioritize documents matching the query language among semantically equivalent candidates, despite document language affecting answer generation. LAMAR jointly models semantic relevance and language coherence, achieving the strongest controlled language-coherence results, competitive benchmark performance, and best results with retrieved candidates.

  • Problem

    It remains unclear whether multilingual rerankers prioritize query-language documents among semantically equivalent multilingual candidates, although document language can affect answer generation.

  • Method

    LAMAR uses English-anchored relevance distillation followed by preference alignment to jointly preserve semantic relevance and prioritize documents written in the query language.

  • Results

    LAMAR achieves the best overall language-coherence performance, remains competitive on multilingual reranking benchmarks, and records the highest scores across reported metrics with retrieved candidate sets.

  • Takeaways & Limitations

    LAMAR accounts for language coherence while retaining strong semantic reranking performance across multilingual benchmarks and practical retrieval settings.

Abstract

from arXiv · show

In multilingual retrieval augmented generation pipelines, an embedding model can retrieve relevant documents written in multiple languages, which are subsequently reranked before answer generation. However, it remains unclear whether existing multilingual rerankers consider document language when ordering semantically relevant candidates. Our analysis shows that these rerankers do not consistently prioritize documents written in the same language as the query when semantically equivalent documents are available across languages, even though document language can affect answer generation. We release LAMAR, a language aware multilingual cross encoder trained to account for both semantic relevance and language coherence. LAMAR first uses English anchored relevance distillation to establish consistent relevance scoring across multilingual inputs and then applies preference alignment for language coherence to encourage documents written in the same language as the query to receive higher rankings while retaining semantic relevance. In a controlled experiment designed to assess language coherence, LAMAR achieves the best performance overall and across all languages examined individually. LAMAR also remains competitive on established multilingual reranking benchmarks. In practical retrieval settings, LAMAR achieves the best results across all reported metrics when reranking candidates retrieved in the first stage. These results demonstrate that LAMAR accounts for language coherence while achieving strong performance on general multilingual reranking benchmarks.

1 Introduction

Multilingual RAG must balance semantic relevance with language coherence because document language can affect answer generation. LAMAR addresses this through two-stage training and achieves strong language-aware and general multilingual reranking results.

  • Motivation: Answer generation can vary with document language even when retrieved documents are semantically relevant, motivating language coherence in multilingual RAG.The paper constructs an evaluation using semantically equivalent documents across languages to examine this effect.
  • Problem: Existing multilingual rerankers do not consistently prioritize semantically equivalent documents written in the query language, especially for English queries.Documents in other languages are often ranked above English documents for English queries.
  • LAMAR: LAMAR jointly models multilingual semantic relevance and language coherence through English-anchored relevance distillation followed by preference alignment.The first stage calibrates relevance scores across multilingual input pairs, while the second encourages language-coherent ranking.
  • Results: Compared with 13 multilingual rerankers, LAMAR more effectively prioritizes query-language documents on parallel multilingual oracle candidate sets.It also achieves competitive performance on established multilingual reranking benchmarks and consistent semantic reranking across languages.
  • Results: LAMAR records the highest scores across all reported metrics when reranking first-stage candidate sets in practical retrieval settings.This result complements its performance on oracle candidate sets and established multilingual reranking benchmarks.

2 Problem Analysis

Existing multilingual rerankers optimize semantic relevance without consistently prioritizing documents in the query’s language, despite language coherence affecting suitability and QA performance. A diagnostic evaluation on parallel XQuAD queries and documents across twelve languages exposes this inconsistency and motivates language-aware reranking.

  • Problem Setup: Multilingual rerankers can treat semantically equivalent documents in different languages as equally valid candidates under relevance-centered training.In multilingual RAG, document language can affect a candidate’s suitability for the query.
  • Diagnostic Evaluation Design: The diagnostic evaluation uses parallel XQuAD queries and gold documents across twelve languages to isolate language coherence from overall reranking effectiveness.The languages are Arabic, German, Greek, English, Spanish, Hindi, Romanian, Russian, Thai, Turkish, Vietnamese, and Chinese.
  • Top-1 Language Selection: Evaluated rerankers do not consistently rank the semantically equivalent gold document written in the same language as the query first.Because each oracle candidate set contains semantically equivalent gold documents, the comparison focuses on the language of the top-ranked document.
  • Same-Language Document Rank: Same-language selection varies substantially across query languages, including variation in the mean rank of the query-language document across the full ranking.For Jinareranker-v3, the mean rank is below 2.0 for some query languages, while the passage reports substantial cross-language variation.
  • Implication: The findings motivate rerankers that preserve semantic relevance while prioritizing documents in the same language as the query.Query-document language consistency is associated with higher QA performance, while existing rerankers do not consistently prioritize the query-language gold document.

3 LAMAR

LAMAR is a language-aware multilingual cross-encoder reranker trained in two stages to calibrate semantic relevance across languages and promote language coherence. Its preference-alignment stage raises the ranking of documents in the query language while preserving semantic relevance.

  • LAMAR encodes each query–document pair with a cross-encoder to produce a scalar relevance score.
  • Training stages: English-anchored relevance distillation transfers teacher scores from English anchor pairs to semantically corresponding multilingual pairs.This places cross-language query–document relations in a consistent semantic relevance space.
  • Training stages: Preference alignment combines group ranking and language-coherence losses to align parallel-candidate preferences.The objective preserves positive-over-negative ordering while modeling preference for documents written in the query language.
  • Group ranking loss: ADR-MSE ranks positive documents above negative documents without imposing a relative order on semantically equivalent parallel documents.It assigns relevance labels 1 to positive documents and 0 to negative documents, with higher-ranked documents receiving greater weight.
  • Language-coherence loss: Softplus score differences incorporate language preference for semantically corresponding pairs in both positive and negative groups without a hard margin.

4 Experimental Setup

LAMAR is evaluated using multilingual and English training data, standardized two-stage training, broad multilingual reranker baselines, and two evaluation settings designed to assess language-aware behavior and multilingual reranking.

  • Datasets: Training uses MMARCO, translated MIRACL triplets, and RLHN, combining multilingual parallel data with English examples.MMARCO links each query to one positive document, MIRACL adds one negative document, and RLHN contains English queries with multiple positive and negative documents.
  • Training data: The final training sets contain 6.7M instances in Stage 1 and 8.6K instances in Stage 2.Stage 1 uses English teacher pairs with corresponding multilingual student pairs.
  • Implementation: LAMAR is initialized from bge-m3-retromae and trained for one epoch per stage on eight NVIDIA RTX A6000 GPUs.Training uses sequence length 8,192, warmup ratio 0.1, AdamW, linear decay, and bf16 precision; Stage 1 uses Qwen3-Reranker-4B as teacher.
  • Baselines: The comparison includes publicly available multilingual rerankers spanning bge, jina, gte, Qwen3, Llama-Nemotron, Contextual, Zerank, and Prism model families.Listed baselines include Qwen3-Reranker-0.6B/4B and Prism-Qwen3.5-Reranker-0.8B/2B/4B.
  • Benchmarks: Evaluation uses an oracle reranking setting for language-aware behavior and a second setting for multilingual reranking.The oracle setting uses parallel subsets from XQuAD and BELEBELE, which provide parallel queries and documents across multiple languages.

5 Experimental Results

LAMAR achieves strong language-coherent reranking in controlled multilingual evaluations while remaining competitive on general semantic benchmarks. Its two-stage training preserves semantic effectiveness and improves language preference through explicit preference alignment.

  • Overall Performance: LAMAR achieves the best performance on parallel oracle subsets of XQuAD and BELEBELE, directly evaluating language coherence while controlling for semantic relevance.These subsets contain semantically equivalent passages across languages, with same-language documents designated as relevant.
  • Consistency Across Languages: LAMAR records the highest nDCG@1 in all six shared languages on both benchmarks and remains stable across languages.Other rerankers show lower scores and larger cross-language gaps, particularly for English queries.
  • General Multilingual Reranking: LAMAR achieves an average score of 86.84 on five MTEB multilingual reranking benchmarks, the second-highest overall result.Its 0.6B parameters perform comparably to substantially larger rerankers.
  • General Multilingual Reranking: LAMAR records an average score of 69.5 on MIRACL, the second-highest average, while remaining competitive across linguistically diverse languages.The evaluation uses a multi-monolingual setting with relevance assessed separately within each language.
  • Ablation Analysis: Removing LLC substantially degrades language-coherence performance, whereas removing Lrank improves language-coherence evaluation but harms general multilingual reranking effectiveness.Stage 1 establishes a consistent semantic relevance scale, while the full two-stage model preserves semantic reranking and adds language preference.

6 Related Work

Related work shows that multilingual RAG must handle retrieval and generation across languages, while document and context language can influence ranking and answer quality. Multilingual reranking research has pursued efficient transfer, zero-shot and translated supervision, listwise training, systematic evaluation, and increasingly multilingual pretraining and long-context modeling.

  • Multilingual RAG systems retrieve information and generate answers across languages.
  • Retrievers may favor particular document languages despite comparable semantic information, and retrieved-context language can affect generation quality and evidence use.
  • Multilingual reranking has explored parameter-efficient transfer, zero-shot large-language-model reranking, translated supervision, listwise training, and systematic pairwise and listwise evaluation.
  • Recent models have advanced multilingual reranking through multilingual pretraining, long-context modeling, and supervision from large language models.

7 Conclusion

This work finds that existing multilingual rerankers do not consistently favor documents matching the query language, even when semantically equivalent alternatives exist. It introduces LAMAR to address language coherence in multilingual reranking.

  • Findings: Existing multilingual rerankers do not consistently prioritize documents written in the same language as the query.This analysis considers semantically equivalent documents available across multiple languages.
  • Findings: Document language can affect answer generation in multilingual retrieval-augmented generation evaluation.
  • LAMAR: LAMAR is a language-aware multilingual cross encoder introduced to account for language coherence in reranking.It is trained through English-anchored relevance distillation.

A Training Dataset Details

The section describes the training datasets’ sources, language coverage, and public links, then explains how instances are synthesized for LAMAR’s two training stages. It covers English-anchored relevance distillation and preference alignment for language coherence.

  • Dataset sources and coverage: Training datasets are described by their sources, language coverage, and public links.The section also includes a language-coverage table.
  • Instance synthesis: Instance synthesis is presented separately for English-anchored relevance distillation and preference alignment for language coherence.These are LAMAR’s two training stages.
  • Construction procedures: The section specifies data-construction algorithms and dataset-specific synthesis settings for both training stages.These settings follow the discussion of dataset language coverage and public links.

A.1 Datasets and Language Coverage

LAMAR is trained using MMARCO, RLHN, and MIRACL, spanning multilingual aligned pairs and English retrieval triplets. MMARCO provides aligned query–positive document pairs across 14 languages, while RLHN supplies query–positive–negative examples.

  • Training datasets: LAMAR is trained on MMARCO, RLHN, and MIRACL.These datasets provide the training data used in the experiments.
  • Language coverage: 14 languages are covered by the processed MMARCO query–positive document pairs.MMARCO is a multilingual version of the MS MARCO passage ranking dataset.
  • Training datasets: RLHN is an English retrieval training collection with query, positive-document, and negative-document triplets.Each triplet supports modeling the semantic relation among its components.
  • Language coverage: The language coverage of MMARCO, RLHN, and MIRACL is summarized in Table 7.The passage identifies Table 7 as the summary of coverage across the three datasets.

A.2 Data Construction for English-Anchored Relevance Distillation … B.1 Language-Coherence Evaluation

The paper constructs multilingual distillation and preference-alignment data to preserve query–document identities, calibrate relevance across languages, and promote language coherence. The supplied passages also indicate that detailed BELEBELE and XQuAD results are reported separately.

  • A.2 Data Construction for English-Anchored Relevance Distillation: English-anchored distillation pairs an English teacher input with a student input expressing the same underlying query–document relation.
  • A.2 Data Construction for English-Anchored Relevance Distillation: Teacher and student inputs preserve both query identity i and document identity j, including documents selected from other query examples.
  • A.2 Data Construction for English-Anchored Relevance Distillation: 120 distillation pairs are produced per triplet by combining original positive and negative documents with four documents sampled from distinct examples.
  • A.3 Data Construction for Preference Alignment for Language Coherence: Preference-alignment data are synthesized listwise by assigning source and target languages to each multilingual triplet.
  • A.3 Data Construction for Preference Alignment for Language Coherence: For Stage 2, R=3 assignments cycle through all 2,601 source–target language pairs formed by 51 languages, with each pair appearing three or four times.
  • B.1 Language-Coherence Evaluation: The detailed BELEBELE results are provided in the language-coherence evaluation section.
  • B.1 Language-Coherence Evaluation: The detailed XQuAD results are provided in the language-coherence evaluation section.

B.2 Multilingual Reranking Evaluation

The multilingual reranking evaluation reports results across BELEBELE, XQuAD, XGLUE, HUME, MLDR, and Wiki retrieval benchmarks. Metrics include nDCG@1, nDCG@10, MRR@10, and Main-nDCG@10, measured across languages.

  • BELEBELE results are reported across languages using nDCG@1, nDCG@10, and MRR@10.
  • XQuAD retrieval results are reported across languages using nDCG@1, nDCG@10, and MRR@10.
  • XGLUE and HUME retrieval results are reported across languages using nDCG@10.
  • MLDR retrieval results are reported across languages using Main-nDCG@10.
  • Wiki retrieval results are reported across languages using nDCG@10.
Loading 2607.22042v2…