Source-linked AI summary
mMARCO: A Multilingual Version of the MS MARCO Passage Ranking Dataset
Luiz Bonifacio, Vitor Jeronymo, Hugo Queiroz Abonizio, Israel Campiotti, Marzieh Fadaee, Roberto Lotufo, Rodrigo Nogueira
TL;DR
Many languages lack reliable labeled IR resources, while available datasets often favor traditional retrieval techniques. The paper creates and evaluates a 13-language machine-translated version of MS MARCO, finding that multilingual fine-tuning improves zero-shot effectiveness and that translation quality is associated with retrieval performance.
Problem
Many languages lack reliable labeled IR resources, while available datasets often favor traditional retrieval techniques.
Method
The authors machine-translate MS MARCO into 13 languages and fine-tune monolingual and multilingual rerankers and a multilingual dense-retrieval model.
Results
Multilingual models fine-tuned on mMARCO outperform models fine-tuned only on English MS MARCO in zero-shot evaluation on Mr. TyDi.
Takeaways & Limitations
mMARCO provides a feasible mechanism for addressing labeled-data scarcity, and translation quality is associated with retrieval effectiveness.
Takeaways & Limitations
The authors leave evaluation on languages unseen during language-model pretraining or fine-tuning for future work.
Abstract
from arXiv · showhide
The MS MARCO ranking dataset has been widely used for training deep learning models for IR tasks, achieving considerable effectiveness on diverse zero-shot scenarios. However, this type of resource is scarce in languages other than English. In this work, we present mMARCO, a multilingual version of the MS MARCO passage ranking dataset comprising 13 languages that was created using machine translation. We evaluated mMARCO by finetuning monolingual and multilingual reranking models, as well as a multilingual dense retrieval model on this dataset. We also evaluated models finetuned using the mMARCO dataset in a zero-shot scenario on Mr. TyDi dataset, demonstrating that multilingual models finetuned on our translated dataset achieve superior effectiveness to models finetuned on the original English version alone. Our experiments also show that a distilled multilingual reranker is competitive with non-distilled models while having 5.4 times fewer parameters. Lastly, we show a positive correlation between translation quality and retrieval effectiveness, providing evidence that improvements in translation methods might lead to improvements in multilingual information retrieval. The translated datasets and finetuned models are available at https://github.com/unicamp-dl/mMARCO.
1 Introduction
Multilingual IR lacks reliable labeled resources, while existing datasets often favor traditional retrieval methods. The paper addresses this gap by creating mMARCO, a machine-translated multilingual version of MS MARCO, and evaluating models trained on it.
- Research gap: Many languages lack reliable labeled datasets for training and evaluating neural IR models.Creating new labeled resources is expensive, motivating automatic translation as an attainable alternative.
- Research gap: Existing training and evaluation datasets often favor traditional bag-of-words techniques, disadvantaging neural models.
- Research gap: Multilingual models are often evaluated on limited monolingual datasets, despite evidence of zero-shot transfer across languages.
- Contribution: mMARCO translates MS MARCO into 13 languages to provide a multilingual resource for training and evaluating IR models.The dataset is motivated by MS MARCO’s transfer-learning capabilities and use in evaluating deep learning models.
- Contribution: The authors fine-tune monolingual and multilingual models for dense retrieval and reranking, then evaluate multilingual models in a zero-shot setting.They report superior effectiveness over models fine-tuned only on the original English dataset.
2 Related Work
Related work covers multilingual pretrained models, dataset translation, and multilingual IR resources. It positions mMARCO against existing collections that are often small, domain-specific, or unevenly distributed across languages.
- Multilingual models: Multilingual pretrained models such as mBERT, XLM, XLM-R, mT5, and mMiniLM have improved cross-lingual task performance.
- Dataset translation: Prior studies have translated datasets either to use English-based models or to create additional multilingual training data.
- Multilingual IR resources: IR initiatives including TREC, CLEF, FIRE, and NTCIR provide annotated multilingual collections, but many are too small for fine-tuning large transformer models.
- Multilingual IR resources: Existing IR collections often draw from specific domains, which can harm model generalization.
- Comparison with Mr. TyDi: Mr. TyDi differs from mMARCO in corpus composition and language-level training-example distribution.Mr. TyDi uses Wikipedia, whereas mMARCO uses diverse web passages.
3 Methodology
The methodology translates MS MARCO into 13 languages, constructs multilingual training data, and evaluates reranking and dense-retrieval models on passage ranking. It compares open-source and commercial translation processes and uses zero-shot evaluation across languages.
- Dataset: MS MARCO provides 8.8M passages, approximately 530k queries, and at least one human-selected relevant passage per query.
- Dataset: MS MARCO is suitable for translation because its labels are invariant to sentence structure and token ordering.This avoids the annotation-preservation problems common when translating span- or token-labeled datasets.
- Translation: The authors use Helsinki open-source translation models and Google Translate, selecting 13 target languages based on language prominence and translation-pair availability.
- Translation: Helsinki translation splits passages into sentences, translates them, and reassembles them, whereas Google Translate processes passages in batches capped at 5000 characters.
- Training data: The multilingual training set uses equal proportions from nine languages, with query, positive passage, and negative passage kept in the same language.This setup supports fine-tuning in one language and zero-shot evaluation in another.
- Experimental setup: Mr. TyDi passages exceeding model input limits are segmented into overlapping windows for mColBERT and mT5 evaluation.
4 Results
The results show that translated multilingual training supports strong multilingual and zero-shot retrieval, while dense retrieval mitigates translation-induced lexical mismatch. Distilled multilingual reranking remains competitive with much larger models, and translation quality is positively associated with retrieval effectiveness.
- Main results: 0.14 average R@1000 drop affects BM25 on non-English translated datasets, likely because independently translated queries and passages create lexical mismatches.The mismatch can prevent BM25 from retrieving relevant passages and subsequently penalize rerankers.
- Main results: 0.193 and 0.168 higher R@1000 values for mColBERT than BM25 occur in German and Dutch, respectively, indicating dense retrieval mitigates lexical mismatch.mColBERT achieved higher R@1000 than BM25 for all languages.
- Main results: MT5 and MMINILM achieve the same English MRR@10, although MMINILM has 107 million versus MT5’s 580 million parameters.The result supports the effectiveness of MMINILM’s distillation despite its substantially smaller size.
- Main results: 0.08 and 0.09 average MRR@10 drops occur for MMINILM and MT5, respectively, on rows (2)–(14) compared with English results.Rows (10)–(14) are zero-shot evaluations on languages excluded from multilingual finetuning.
- Zero-shot results: mT5 finetuned on mMARCO outperforms BM25 in all Mr. TyDi languages and exceeds English-only mT5 on average, including languages absent from mMARCO.The absent languages include Bengali, Finnish, Japanese, Korean, and Swahili.
- Ablation study: Multilingual finetuning remains close to monolingual performance on English and Portuguese, while MT5 slightly outperforms the monolingual model on Portuguese.The best Portuguese result uses English-Portuguese finetuning; finetuning on more than two languages performs lower.
5 Conclusions and Future Work
The paper introduces mMARCO, a 13-language translated version of MS MARCO, and releases the dataset, models, and translation code for multilingual IR research. Its findings support multilingual finetuning, translation-based data expansion, and lightweight distilled reranking models.
- mMARCO translates MS MARCO into 13 languages and provides a multilingual IR resource for training and evaluation.
- The released benchmarks include monolingual and multilingual Transformer-based models evaluated on the translated datasets.
- Multilingual models finetuned on multilingual datasets achieve competitive results compared with monolingual approaches.
- Translating datasets is presented as a feasible mechanism for addressing labeled-data scarcity in multilingual information retrieval.
- MINILM remains competitive with a much larger model when finetuned in the same way.