Source-linked AI summary
Mr. TyDi: A Multi-lingual Benchmark for Dense Retrieval
Xinyu Zhang, Xueguang Ma, Peng Shi, Jimmy Lin
TL;DR
Dense retrieval models may not generalize reliably to non-English, out-of-distribution retrieval, motivating a dedicated benchmark. The paper constructs Mr. TYDI for mono-lingual retrieval in eleven typologically diverse languages and evaluates zero-shot baselines including mDPR. Although mDPR is weaker than BM25 alone, its relevance signals improve sparse-dense hybrids, while the dataset’s judgments are not exhaustive.
Problem
Dense retrieval generalizes poorly to out-of-distribution inputs, including retrieval in languages different from those used for training.
Method
The paper constructs Mr. TYDI, a benchmark for mono-lingual retrieval in eleven typologically diverse languages, and evaluates BM25, zero-shot mDPR, and a sparse-dense hybrid.
Results
Even poor dense retrieval results provide valuable relevance signals in a sparse-dense hybrid, improving BM25 results.
Takeaways & Limitations
Mr. TYDI provides a resource for studying multilingual mono-lingual retrieval and the complementary behavior of dense and sparse relevance signals.
Takeaways & Limitations
Mr. TYDI uses sparse, non-exhaustive relevance judgments, with roughly one good answer per question.
Abstract
from arXiv · showhide
We present Mr. TyDi, a multi-lingual benchmark dataset for mono-lingual retrieval in eleven typologically diverse languages, designed to evaluate ranking with learned dense representations. The goal of this resource is to spur research in dense retrieval techniques in non-English languages, motivated by recent observations that existing techniques for representation learning perform poorly when applied to out-of-distribution data. As a starting point, we provide zero-shot baselines for this new dataset based on a multi-lingual adaptation of DPR that we call "mDPR". Experiments show that although the effectiveness of mDPR is much lower than BM25, dense representations nevertheless appear to provide valuable relevance signals, improving BM25 results in sparse-dense hybrids. In addition to analyses of our results, we also discuss future challenges and present a research agenda in multi-lingual dense retrieval. Mr. TyDi can be downloaded at https://github.com/castorini/mr.tydi.
1 Introduction
Dense retrieval learns query and passage representations for nearest-neighbor ranking, but its robustness outside training conditions is limited. Mr. TYDI targets this issue for mono-lingual retrieval in non-English languages and introduces a benchmark with zero-shot baselines.
- Dense retrieval: Dense retrieval trains separate query and passage encoders whose vectors are compared for nearest-neighbor search.This bi-encoder design is used by systems such as DPR, ANCE, and ColBERT.
- Generalization challenge: Out-of-distribution inputs can differ in passage domain, query type, input relationship, or combinations of these factors.The concern is whether supervised representations remain robust and generalizable beyond their training data.
- Generalization challenge: Zero-shot evaluations found BM25 more effective overall, while dense models trained on one dataset can fail on another.The comparison motivates studying dense retrieval robustness across corpora, queries, and tasks.
- Mr. TYDI: Mr. TYDI evaluates mono-lingual retrieval in eleven diverse languages, such as Bengali questions against Bengali documents.It focuses on non-English mono-lingual retrieval rather than cross-lingual retrieval.
- Mr. TYDI: The paper contributes Mr. TYDI, zero-shot BM25 and mDPR baselines, and initial findings for a research agenda in multilingual dense retrieval.mDPR is a multilingual adaptation of DPR, and the paper reports that its relevance signals can improve BM25 in sparse-dense hybrids.
2 Background and Related Work
Existing resources do not directly support open-retrieval evaluation of mono-lingual non-English search: some provide candidate passages, others target cross-lingual retrieval, and others use unsuitable corpora. Mr. TYDI extends TYDI toward this evaluation setting.
- Motivation: The authors motivate a new resource because existing datasets do not provide the required retrieval setting or language-specific evidence.The rationale is summarized through comparisons with TYDI, XOR-TYDI, MKQA, and prior multilingual test collections.
- TYDI: TYDI resembles machine reading comprehension because candidate passages come from a single top-ranked Wikipedia article rather than a large retrieval corpus.Mr. TYDI is framed as an open-retrieval extension requiring search over a much larger corpus.
- XOR-TYDI: XOR-TYDI targets cross-lingual retrieval of answers in English Wikipedia, not mono-lingual retrieval in the question’s language.Its construction translates non-English questions into English and annotates English passages.
- Existing-resource limitations: TYDI-derived resources cannot support the desired evaluation when they lack ground truth for intermediate retrieval or use English answer passages.These constraints prevent evaluating searches such as Finnish questions over Finnish Wikipedia.
- MKQA: MKQA uses translated, cross-lingually aligned questions and retrieval-independent answer annotations, which may limit natural-language and typological coverage.The authors argue that Mr. TYDI fills an evaluation gap by preserving language-specific questions and focusing on typologically diverse languages.
- Existing-resource limitations: Older multilingual test collections often use diverse source corpora, making cross-language generalization difficult for the research questions considered here.The paper contrasts these collections with TYDI’s comparable Wikipedia corpora.
3 Mr. TYDI
Mr. TYDI is an open-retrieval extension to TYDI for monolingual ad hoc retrieval across language-specific Wikipedia collections. It derives relevance judgments from TYDI while acknowledging that those judgments are sparse and non-exhaustive.
- Dataset and task: Mr. TYDI extends TYDI into open-retrieval evaluation by ranking passages from the same-language Wikipedia collection for each question.The task focuses on first-stage retrieval, with retrieved passages intended for possible downstream answer extraction.
- Corpus construction: The corpus uses TYDI’s raw Wikipedia dumps and passage-level granularity, adding article titles to retrieval units for isolated understanding.
- Questions and judgments: Questions begin with TYDI items having non-empty answer passages; TYDI’s development set serves as test data, and 20% of original training questions form a new development set.
- Questions and judgments: Relevant passages are imputed from TYDI annotations, while unjudged passages are treated as non-relevant under standard information-retrieval assumptions.Because TYDI assessed only the top-ranked article, some relevant passages may remain unidentified.
- Questions and judgments: The judgments are far from exhaustive, so additional judgments may change absolute scores even if they are expected not to alter findings qualitatively.The paper relates this sparse-judgment issue to MS MARCO and other information-retrieval evaluations.
- Evaluation: Results are evaluated with reciprocal rank and recall at depth 100, measuring ranking quality and an upper bound on downstream reader effectiveness.The choice k = 100 follows conventions in question-answering research.
4 Baselines
The baselines compare traditional sparse retrieval, zero-shot multilingual dense retrieval, and a score-fused hybrid. The dense model is adapted from DPR with multilingual BERT, while the hybrid combines BM25 and mDPR scores.
- BM25: BM25 provides a strong traditional information-retrieval baseline, using language-specific Lucene analyzers where available and whitespace analysis for Telugu and Swahili.
- mDPR: mDPR adapts DPR to monolingual non-English retrieval by replacing BERT with multilingual BERT while retaining the original training procedure.It is trained on English Natural Questions and applied zero-shot to passages and questions in all eleven languages.
- mDPR: The zero-shot setup tests dense-retrieval generalizability without target-task exposure, making the default-parameter comparison fair to BM25.
- Sparse–Dense Hybrid: The sparse–dense hybrid fuses normalized BM25 and mDPR scores as ssparse + α · sdense after retrieving 1000 hits from each method.
- Results: Table 2 reports BM25 with default and tuned parameters, mDPR, and the sparse–dense hybrid on the Mr. TYDI test set.A dagger marks significant improvements over tuned BM25 using a paired t-test with p < 0.01.
5 Results and Analysis
Across eleven languages, mDPR generally trails tuned BM25, yet combining sparse and dense scores significantly improves MRR@100 in nine languages. The analyses show that dense retrieval's weaknesses often arise from missed relevant passages, while hybrid gains come from recall, ranking, or both.
- High-Level Findings: Tuned BM25 parameters yield at most minor MRR@100 and recall improvements for most languages, except Telugu.Tuned BM25 therefore serves as the competitive baseline for subsequent comparisons.
- High-Level Findings: mDPR underperforms BM25 across all languages except English in the zero-shot setting.The authors associate this result with poor robustness across languages and sensitivity to differences between NQ and TYDI questions.
- High-Level Findings: The sparse–dense hybrid significantly exceeds tuned BM25 in MRR@100 for nine of eleven languages, with Swahili and Telugu as exceptions.On average, the hybrid is around eight MRR@100 points and five recall points above tuned BM25.
- High-Level Findings: In Arabic, mDPR reaches 71% of BM25's MRR@100 while the hybrid improves over BM25 by 34%.Figure 2 normalizes each language's effectiveness to tuned BM25 and relates mDPR performance to hybrid gains.
- High-Level Findings: The hybrid's gains correlate positively with relative mDPR effectiveness, although even weak mDPR can provide useful relevance signals.In Thai, mDPR reaches only 32% of tuned BM25 while the hybrid yields an 18% relative gain; Swahili and Telugu do not improve over tuned BM25.
- Components of Effectiveness: For Arabic, Japanese, and Korean, mDPR and BM25 rank similarly when a relevant passage is retrieved, but mDPR has lower recall.For other languages, BM25 shows both better recall and better ranking; the analysis separates these failure modes.
- Components of Effectiveness: Hybrid gains in Bengali, Indonesian, Swahili, and Telugu come mostly from higher recall, while Thai gains come mainly from better ranking.For the remaining languages, the hybrid improves both recall and ranking.
6 Future Work
Mr. TYDI establishes zero-shot baselines while opening a broad research agenda on multilingual dense retrieval, hybrid behavior, language effects, and transformer properties.
- 6 Future Work: Zero-shot mDPR is ineffective alone, but its relevance signals complement BM25 through better recall versus better ranking.The complementarity varies across languages, and its underlying causes remain unexplained.
- 6 Future Work: Future experiments should examine language-specific fine-tuning, multilingual training choices, machine-translation augmentation, and complementary task data.The paper deliberately omits language-specific fine-tuning results, although preliminary experiments suggest such techniques help.
- 6 Future Work: Mr. TYDI can probe multilingual transformers by relating cross-language effectiveness differences to queries, corpus sizes, pretraining data, and typological properties.The authors hypothesize that inherent transformer properties may contribute to variation across languages.
7 Conclusion
The paper introduces Mr. TYDI and evaluates BM25, mDPR, and sparse–dense hybrids for multilingual mono-lingual retrieval. Its conclusion is that dense retrieval generalizes poorly out of distribution, yet even weak dense signals can improve hybrid retrieval and support future work on equitable multilingual information access.
- 7 Conclusion: Mr. TYDI benchmarks mono-lingual retrieval in eleven typologically diverse languages and supports zero-shot experiments with BM25, mDPR, and sparse–dense hybrids.The dataset is intended as a resource for exploring dense retrieval on out-of-distribution multilingual inputs.
- 7 Conclusion: Dense retrieval techniques do not generalize well to out-of-distribution input, but poor dense results still provide valuable relevance signals in sparse–dense hybrids.The conclusion frames hybrid retrieval as the key positive finding despite weak standalone dense effectiveness.
- 7 Conclusion: Mr. TYDI provides a starting point for developing techniques that improve dense-retrieval robustness and generalizability across diverse languages.The authors connect this research direction with more equitable information access for non-English speakers.