PaperlayerView exact revision on arXiv

Source-linked AI summary

Improving Semantic Proximity in Information Retrieval through Cross-Lingual Alignment

Seongtae Hong, Youngjoon Jang, Jungseob Lee, Hyeonseok Moon, Heuiseok Lim

arXiv:2604.05684v1cs.IR

TL;DR

The paper addresses whether multilingual retrievers maintain cross-lingual retrieval quality when English and the query language coexist in the document pool. It introduces multi-reference evaluations and a JSD-plus-InfoNCE training strategy, reporting substantially improved alignment and retrieval from 2.8K samples while reducing language disparities.

  • Problem

    Conventional CLIR settings may inadequately assess cross-lingual alignment, allowing irrelevant English documents to outrank relevant documents in the query language.

  • Method

    The paper introduces a two-language multi-reference scenario and Max@R, then combines JSD-based embedding alignment with InfoNCE retrieval training.

  • Results

    Using 2.8K samples, the method substantially improves cross-lingual alignment and retrieval while maintaining or improving conventional CLIR performance.

  • Takeaways & Limitations

    The proposed scenario and metric expose language bias and alignment problems that conventional settings overlook, while the training strategy reduces these disparities.

  • Takeaways & Limitations

    The experiments and evaluations primarily center on English, leaving language pairs that do not involve English insufficiently examined.

Abstract

from arXiv · show

With the increasing accessibility and utilization of multilingual documents, Cross-Lingual Information Retrieval (CLIR) has emerged as an important research area. Conventionally, CLIR tasks have been conducted under settings where the language of documents differs from that of queries, and typically, the documents are composed in a single coherent language. In this paper, we highlight that in such a setting, the cross-lingual alignment capability may not be evaluated adequately. Specifically, we observe that, in a document pool where English documents coexist with another language, most multilingual retrievers tend to prioritize unrelated English documents over the related document written in the same language as the query. To rigorously analyze and quantify this phenomenon, we introduce various scenarios and metrics designed to evaluate the cross-lingual alignment performance of multilingual retrieval models. Furthermore, to improve cross-lingual performance under these challenging conditions, we propose a novel training strategy aimed at enhancing cross-lingual alignment. Using only a small dataset consisting of 2.8k samples, our method significantly improves the cross-lingual retrieval performance while simultaneously mitigating the English inclination problem. Extensive analyses demonstrate that the proposed method substantially enhances the cross-lingual alignment capabilities of most multilingual embedding models.

1 INTRODUCTION

The paper argues that conventional CLIR settings inadequately assess cross-lingual alignment because multilingual retrievers can favor unrelated English documents over relevant documents in the query language. It introduces a two-language evaluation scenario and training strategy to analyze and mitigate this bias.

  • Motivation: Conventional CLIR evaluation typically uses a document pool in one language different from the query language, limiting assessment of multilingual representation capability.The paper contrasts this with multilingual retrieval from integrated collections containing multiple languages.
  • Problem: English-prioritization bias can rank irrelevant English documents above relevant documents written in the query language.The paper also reports substantial performance variation across query languages.
  • Evaluation: The paper defines a two-language coexistence scenario and introduces Max@R to evaluate cross-lingual alignment, retrieval performance, and language preference.This setup is designed to expose issues that conventional settings may not measure adequately.
  • Method: The proposed strategy combines JSD for cross-lingual embedding-distribution alignment with InfoNCE for query-document retrieval.It jointly optimizes alignment and retrieval using only 2.8K samples.
  • Results: The method substantially improves cross-lingual alignment and retrieval while maintaining or improving performance in monolingual settings, including conventional CLIR.The reported improvements also mitigate language-related retrieval problems.

2 PRELIMINARY

The paper frames CLIR and MLIR as retrieval over collections whose languages differ from or span the query language, then motivates a multi-reference setup for deeper alignment analysis. Figure 2 compares models across languages and reports Max@R gaps.

  • Evaluation Gap: Existing evaluation settings assess basic cross-lingual or shallow multilingual retrieval but can miss inaccurate alignment and language bias.The paper therefore considers a multi-reference setup containing semantically equivalent documents in multiple languages.
  • Multi-Reference Setup: The proposed setup uses a two-language document pool and evaluates whether all semantically relevant documents rank highly regardless of query or document language.The experimental environment is constructed from a fully parallel multilingual dataset.
  • Experimental Comparison: Figure 2 compares four multilingual embedding models across Arabic, Chinese, Spanish, Thai, and Vietnamese in conventional CLIR and Multi scenarios.The displayed values indicate Max@R gaps.

3 MULTI-REFERENCE IN CROSS-LINGUAL INFORMATION RETRIEVAL

The multi-reference scenario and Max@R metric expose language-dependent instability and extreme retrieval ranks that conventional CLIR can overlook. The analysis therefore reveals practical cross-lingual alignment problems requiring more rigorous evaluation.

  • Problem Definition: Insufficient cross-lingual alignment can prevent relevant documents in both languages from appearing at the top of retrieval results.The paper introduces a metric to quantify this problem when two languages coexist in the document pool.
  • Max@R Metric: Max@R evaluates how far retrieval must proceed before all parallel ground-truth documents for a query are found.It is designed for multi-reference scenarios where existing MAP, MRR, and NDCG@k do not measure complete reference retrieval directly.
  • Max@R Metric: A lower Max@R means all relevant reference documents appear within a smaller top-ranked portion of results.Thus, lower values indicate more efficient and higher-performing retrieval.
  • Observed Disparities: 597.9: multilingual-e5 shows this Max@R difference between Chinese and English queries in the Multi scenario.The disparity indicates amplified query-language differences under the proposed evaluation setting.
  • Observed Disparities: 12.12, 37.38, and 35.04: gte-multilingual’s Max@R for Spanish, Arabic, and Thai queries respectively shows substantial target-language instability.The paper describes the Arabic and Thai values as nearly a threefold increase over Spanish.
  • Observed Disparities: 650.95: multilingual-e5 reaches this Max@R for Chinese queries, requiring review of hundreds of documents to find all relevant documents.The result illustrates why conventional CLIR may fail to capture practical alignment and retrieval problems.
  • Implications: The proposed scenario and metric reveal critical cross-lingual alignment and retrieval issues overlooked by conventional evaluation settings.The findings motivate more realistic evaluation across varied cross-lingual conditions.

4 METHODS

The method combines distribution-level semantic embedding alignment with contrastive retrieval optimization. JSD aligns cross-lingual embedding distributions, while InfoNCE improves query–document similarity and separates related from unrelated examples.

  • Unified training objective: The training objective jointly minimizes JSD-based alignment loss and InfoNCE contrastive loss.This combines distribution-level cross-lingual alignment with query–document retrieval optimization.
  • Semantic embedding alignment: Standard similarity objectives can produce equal cosine scores while leaving embedding distributions substantially misaligned.The method therefore targets the underlying representation geometry rather than retrieval similarity alone.
  • Semantic embedding alignment: JSD aligns English and target-language document embeddings at the distribution level to improve cross-lingual semantic alignment.The approach transforms semantic embedding vectors into probability distributions and minimizes their Jensen–Shannon divergence.
  • Retrieving objective: InfoNCE maximizes similarity between English queries and corresponding target-language passages while minimizing similarity with unrelated examples.Negative examples are queries from other instances in the same batch, and relevance is measured by cosine similarity.

5 EXPERIMENTS

Experiments evaluate the proposed alignment strategy across multilingual retrieval scenarios, datasets, languages, and embedding models. The method improves cross-lingual retrieval, reduces language-performance disparities, preserves monolingual performance, and benefits from combining JSD with InfoNCE.

  • Experimental scenarios: Three scenarios evaluate retrieval of parallel ground-truth documents, same-language-filtered documents, and monolingual documents.The Multi-1 setting forces retrieval of the opposite-language counterpart, while the Multi setting retains parallel documents in both languages.
  • Datasets and metrics: Fully parallel multilingual benchmarks provide matching question-document pairs across languages for evaluating the scenarios and Max@R.The experiments use fully parallel datasets as a prerequisite for cross-lingual comparisons.
  • Main results: The proposed method consistently improves retrieval across evaluated language pairs and metrics, including Complete@10 performance for non-English queries.These improvements are reported across all considered languages and models.
  • Main results: 6.89%p → 1.77%p and 4.45%p → 0.12%p are the reported language-performance gap reductions for jina-embeddings-v3 on XQuAD and Belebele.The reductions quantify decreased disparity between English and target-language queries.
  • Main results: 650.95 to 23.10 is the reported Max@R improvement for Chinese queries with multilingual-e5-base on XQuAD.The lower Max@R indicates relevant passages are positioned nearer the top in the multi-reference setting.
  • Case study: NDCG@1 improves across language pairs in Multi-1 for both English and target-language queries.The authors present Multi-1 as a more rigorous assessment of cross-lingual semantic alignment.
  • Case study and ablation: Monolingual performance is largely preserved or modestly improved, while removing either JSD or InfoNCE harms alignment or retrieval effectiveness.The ablation results support complementary roles for distribution alignment and query-document relevance learning.
  • Ablation study: Directly aligning output-representation distributions outperforms document-level similarity enhancement across base models.The comparison distinguishes the proposed distribution-alignment approach from LNCEpsg.

6 RELATED WORKS

Related work addresses cross-lingual retrieval through shared embedding spaces, knowledge transfer, parallel-corpus alignment, and fine-grained evaluation tasks. These studies motivate explicit alignment objectives and more nuanced assessments of multilingual representations.

  • Cross-lingual retrieval: Existing CLIR studies bridge semantic gaps through cross-lingual embedding spaces and knowledge-transfer approaches that reduce query-document distances.Unsupervised methods also seek to reduce reliance on translation resources.
  • Embedding alignment: Explicit alignment research uses parallel corpora and alignment objectives, while contextual studies evaluate fine-grained transfer through parsing and token-level retrieval tasks.These approaches emphasize sentence-level and contextual embedding alignment.

7 CONCLUSION

The paper introduces an evaluation scenario and Max@R to expose semantic misalignment and language disparities in multilingual retrieval. Its JSD–InfoNCE training strategy improves cross-lingual retrieval, reduces language bias, and remains stable in monolingual settings.

  • Conclusion: A new evaluation scenario and Max@R metric reveal semantic misalignment and language disparities not observable in existing scenarios.The scenario is designed to investigate cross-lingual alignment more rigorously.
  • Conclusion: The training strategy combines Jensen-Shannon Divergence for embedding alignment with InfoNCE for cross-lingual retrieval performance.Together, the losses target semantic proximity and retrieval effectiveness.
  • Conclusion: The method mitigates linguistic misalignment and language bias while improving cross-lingual retrieval and reducing performance disparities across languages.The conclusion reports these outcomes as the central empirical findings.
  • Conclusion: The method demonstrates stable performance in monolingual settings.This extends the reported benefits beyond cross-lingual retrieval scenarios.

ETHICS STATEMENT

The study acknowledges risks from machine-translated training data, including distorted cultural nuances and biases that may affect some linguistic or cultural groups.

  • Machine-translated training data may distort subtle cultural nuances or introduce biases, producing inaccurate or unintended outcomes for certain groups.The paper acknowledges these risks while stating that it expects the research benefits to outweigh them.

REPRODUCIBILITY STATEMENT

The paper is designed for full reproducibility, documenting experimental setup, training information, and computational resources across the main text and appendices.

  • Experimental setup details are provided in Section 5.1, while training information and computational-resource requirements appear in Appendix C.The stated materials include datasets, implementation specifics, hyperparameters, and hardware requirements.

A LIMITATIONS

The study’s evaluation is mainly centered on English and two-language settings, and its training data relies on machine translation. It also reports experiments using multilingual QA data and fixed experimental controls.

  • The evaluation primarily centers on English, limiting coverage of language pairs that do not involve English.The authors intentionally focus on English because it is widely used and high-resource, and because they assume models may be biased toward it.
  • The experimental design focuses mainly on two-language settings, although real-world multilingual contexts often involve more than two languages.The authors retain simpler cross-lingual settings because existing models still struggle in them.
  • The training dataset is built with language-model-based machine translation, which may miss subtle linguistic nuances and cultural contexts compared with human translation.The paper describes this approach as practical and efficient for constructing large-scale multilingual datasets.
  • The evaluation uses XQuAD, a professionally translated parallel QA dataset spanning 13 languages, converted into retrieval tasks.XQuAD provides one-to-one correspondence between multilingual question-answer pairs.
  • Experiments use one training epoch, batch size 32, a linear scheduler with 0.15 warm-up, AdamW, and bfloat16 mixed precision.Model-specific initial learning rates are also reported in the hyperparameter description.
  • Training uses two NVIDIA A100 80GB GPUs with AMD EPYC 7513 processors, while evaluation uses a single accelerator.
  • All reported results come from a single run with fixed random seed 42 controlling data shuffling, batch composition, and sampling order.The fixed seed is used for both baselines and the proposed model to control the comparison.

D EXTENDED RESULTS FOR ADDITIONAL LANGUAGES

The appendix extends experiments to five additional languages and reports further comparisons for Multi-1, Multi, and Mono scenarios.

  • Extended experiments cover German, Greek, Hindi, Romanian, and Turkish across three retrieval scenarios.These languages were not included in the paper’s main body.
  • Figure 4 compares NDCG@1 across additional language pairs in the Multi-1 scenario, using each specified language as the query language.
  • Table 4 reports performance comparisons for the remaining language pairs under the Multi scenario.
  • Table 5 reports Mono-scenario performance for five additional languages with jina-embeddings-v3 using English and target-language queries.
Loading 2604.05684v1…