PaperlayerView exact revision on arXiv

Source-linked AI summary

MMTEB: Massive Multilingual Text Embedding Benchmark

Kenneth Enevoldsen, Isaac Chung, Imene Kerboua, Márton Kardos, Ashwin Mathur, David Stap, Jay Gala, Wissam Siblini, Dominik Krzemiński, Genta Indra Winata, Saba Sturua, Saiteja Utpala, Mathieu Ciancone, Marion Schaeffer, Gabriel Sequeira, Diganta Misra, Shreeya Dhakal, Jonathan Rystrøm, Roman Solomatin, Ömer Çağatan, Akash Kundu, Martin Bernstorff, Shitao Xiao, Akshita Sukhlecha, Bhavish Pahwa, Rafał Poświata, Kranthi Kiran GV, Shawon Ashraf, Daniel Auras, Björn Plüster, Jan Philipp Harries, Loïc Magne, Isabelle Mohr, Mariya Hendriksen, Dawei Zhu, Hippolyte Gisserot-Boukhlef, Tom Aarsen, Jan Kostkan, Konrad Wojtasik, Taemin Lee, Marek Šuppa, Crystina Zhang, Roberta Rocca, Mohammed Hamdy, Andrianos Michail, John Yang, Manuel Faysse, Aleksei Vatolin, Nandan Thakur, Manan Dey, Dipam Vasani, Pranjal Chitale, Simone Tedeschi, Nguyen Tai, Artem Snegirev, Michael Günther, Mengzhou Xia, Weijia Shi, Xing Han Lù, Jordan Clive, Gayatri Krishnakumar, Anna Maksimova, Silvan Wehrli, Maria Tikhonova, Henil Panchal, Aleksandr Abramov, Malte Ostendorff, Zheng Liu, Simon Clematide, Lester James Miranda, Alena Fenogenova, Guangyu Song, Ruqiya Bin Safi, Wen-Ding Li, Alessia Borghini, Federico Cassano, Hongjin Su, Jimmy Lin, Howard Yen, Lasse Hansen, Sara Hooker, Chenghao Xiao, Vaibhav Adlakha, Orion Weller, Siva Reddy, Niklas Muennighoff

arXiv:2502.13595v4cs.CLcs.AIcs.IR

TL;DR

Existing text-embedding benchmarks provide limited coverage across languages, domains, and tasks. MMTEB expands evaluation through a large collaborative benchmark and computationally efficient benchmark construction, finding that smaller multilingual models can outperform 7B models in highly multilingual or low-resource settings while retaining similar rankings in a faster English benchmark.

  • Problem

    Existing text-embedding benchmarks have limited coverage across languages, domains, and tasks, leaving multilingual and low-resource evaluation insufficiently broad.

  • Method

    MMTEB combines over 500 multilingual evaluation tasks with collaborative quality control, inter-task-correlation downsampling, hard-negative retrieval sampling, and embedding reuse to construct broad and efficient benchmarks.

  • Results

    Smaller XLM-R Large-based models outperform 7B LLM-based embeddings in highly multilingual or low-resource settings, while zero-shot MTEB(eng, v2) correlates 0.90 with MTEB(eng, v1).

  • Takeaways & Limitations

    MMTEB enables broad multilingual embedding evaluation with benchmark variants that reduce computational demands while preserving useful model comparisons.

  • Takeaways & Limitations

    The benchmark is skewed toward high-resource languages, and human-translated datasets may introduce bias toward English or translated content.

Abstract

from arXiv · show

Text embeddings are typically evaluated on a limited set of tasks, which are constrained by language, domain, and task diversity. To address these limitations and provide a more comprehensive evaluation, we introduce the Massive Multilingual Text Embedding Benchmark (MMTEB) - a large-scale, community-driven expansion of MTEB, covering over 500 quality-controlled evaluation tasks across 250+ languages. MMTEB includes a diverse set of challenging, novel tasks such as instruction following, long-document retrieval, and code retrieval, representing the largest multilingual collection of evaluation tasks for embedding models to date. Using this collection, we develop several highly multilingual benchmarks, which we use to evaluate a representative set of models. We find that while large language models (LLMs) with billions of parameters can achieve state-of-the-art performance on certain language subsets and task categories, the best-performing publicly available model is multilingual-e5-large-instruct with only 560 million parameters. To facilitate accessibility and reduce computational cost, we introduce a novel downsampling method based on inter-task correlation, ensuring a diverse selection while preserving relative model rankings. Furthermore, we optimize tasks such as retrieval by sampling hard negatives, creating smaller but effective splits. These optimizations allow us to introduce benchmarks that drastically reduce computational demands. For instance, our newly introduced zero-shot English benchmark maintains a ranking order similar to the full-scale version but at a fraction of the computational cost.

1 INTRODUCTION

MMTEB addresses limited multilingual, domain, and task coverage in existing text-embedding benchmarks through a broad, open collaborative expansion. It targets accessibility for low-resource communities while extending evaluation across languages, domains, and capabilities.

  • Motivation: Existing text-embedding benchmarks lack broad coverage across languages, domains, and tasks.Many existing benchmarks focus on specific domains or narrower evaluation settings.
  • Motivation: MMTEB was created through large-scale open collaboration to broaden coverage, especially for low- to mid-resource languages.The collaboration included contributors from diverse linguistic backgrounds.
  • MMTEB: MMTEB comprises more than 500 tasks across 10 categories and over 250 languages, spanning domains including fiction, social media, medicine, and programming.It also includes instruction following, long-document embedding, reasoning, and cross-lingual retrieval tasks.
  • Accessibility: The benchmark is designed for accessibility because extensive embedding evaluation can require substantial computational resources.The paper highlights the co-occurrence of limited compute and low-resource languages and provides public code and leaderboard access.

2 MMTEB CONSTRUCTION

MMTEB combines community quality control, computational optimizations, correlation-based task selection, and benchmark construction procedures to produce broad but more manageable evaluations. The resulting suite supports multilingual, regional, domain-specific, language-specific, and tailored benchmarks.

  • Community construction: Contributors were recruited from industry, low-resource language communities, and academia, with GitHub-based coordination and a point-based authorship system.This structure was intended to support broad applicability and transparent contribution recognition.
  • Quality control: Each submitted task received contributor review and required metadata covering annotation source, dataset source, license, dialects, and citations.Two multilingual models were also run to identify potentially problematic performance patterns for further examination.
  • Quality control: Tasks with suspiciously low, near-perfect, or unusually similar scores were examined for implementation flaws or poor data quality, while difficult tasks could still be retained.Language-familiar contributors were consulted when possible before inclusion decisions.
  • Efficiency: Embedding caching and dataset downsampling reduce evaluation cost by avoiding redundant encoding and processing representative subsets.Clustering bootstrapping encodes a 4% corpus subsample and reuses documents across 10 sets; some tasks encode 100× fewer documents and average speedup is 16.11x.
  • Benchmark construction: Task selection treats task scores as features and removes predictable tasks through inter-task correlation, enabling subsets intended to predict scores on unobserved tasks.The procedure includes an initial scope, iterative refinement, and manual review.
  • Benchmark construction: MMTEB yields multilingual, European, Indic, faster English, code-retrieval, long-document, and language-specific benchmarks, with code for creating tailored suites.The MTEB(Multilingual) construction begins with 550 tasks and filters to 343 tasks covering over 250 languages before further selection.

3 EXPERIMENTAL SETTINGS

The experiments evaluate representative multilingual embedding models across selected multilingual benchmarks using aggregate and category-level metrics. The setup compares models across sizes and architectures while examining parameter count, embedding size, and sequence length.

  • Models: The evaluation includes multilingual LaBSE, MPNet, MiniLM, and multilingual-e5 models spanning varied training data and model sizes.The multilingual-e5 series uses a two-step weak-supervision training approach.
  • Metrics and ranking: Results are reported as average scores across all tasks, per-task-category scores, and scores weighted by task category.Model ranks are computed using the Borda count method, with tournament Borda count used for ties.
  • Evaluation scores: Figure 2 maps mean MTEB(Multilingual) performance against parameter count, with circle size encoding embedding size and color encoding maximum sequence length.Only selected model labels are shown for readability, and overlapping models may reflect similar performance.
  • Benchmarks: Three showcased multilingual benchmarks are MTEB(Multilingual), MTEB(Europe), and MTEB(Indic).They target fully multilingual evaluation and European or Indic language groups with varying resource levels.

4 ANALYSIS AND DISCUSSION

Across multilingual benchmarks, instruction-tuned models generally perform better, while multilingual pre-training is especially advantageous for lower-resource languages. The optimized zero-shot English benchmark preserves rankings while reducing task count and runtime.

  • Instruction-tuned models significantly outperform non-instruction-tuned models, with the largest gains in bitext mining and clustering.The advantage remains pronounced across task categories despite generic prompts and no model-specific prompt tuning.
  • Multilingual-e5-large-instruct generally outperforms the larger e5-mistral-7b-instruct and GritLM-7B, especially for mid-to-low-resource languages.GritLM remains best in class for retrieval on MTEB(Multilingual).
  • Multilingual-e5-large-instruct outperforms Mistral-based models on lower-resource languages in MTEB(Europe) and all languages in MTEB(Indic), despite having substantially fewer parameters.Mistral-based performance decreases and becomes more volatile as native-speaker counts fall.
  • English models perform worse on non-English languages, whereas LaBSE shows relatively high bitext performance.
  • A zero-shot MTEB(eng, v2) ranking correlates with MTEB(eng, v1) at Spearman 0.90 and Pearson 0.96, while reducing the benchmark from 56 to 40 tasks.On an H100, runtime is 3.11 hours for GritLM-7B and 0.81 hours for all-MiniLM-L12.

5 RELATED WORK

Prior embedding benchmarks expanded task and domain coverage but retained limited multilingual coverage, while collaborative research offers a way to broaden data and participation. MMTEB builds on this trajectory.

  • BEIR introduced evaluation using publicly available datasets from diverse information-retrieval tasks and domains.
  • MTEB expanded beyond information retrieval to eight additional task categories and covers 58 tasks across 112 languages.Its multilingual coverage is mainly based on machine-translated tasks or bitext mining.
  • Open research initiatives can stimulate innovation, reduce negative biases, and increase the diversity of data sources.Diverse stakeholder involvement is associated with more ethical, robust, and reproducible research.

6 CONCLUSION

MMTEB is a large collaborative multilingual benchmark that supports multiple language-focused evaluations and computationally lighter benchmark variants. Results show that smaller XLM-R-based models can outperform 7B LLM-based embeddings in multilingual and low-resource settings.

  • MMTEB is an open collaboration comprising more than 500 tasks covering more than 1000 languages.The paper constructs MTEB(Multilingual), MTEB(Indic), and MTEB(Europe) from these additions.
  • Hard-negative retrieval sampling and bootstrapped clustering reduce the number of text samples requiring embedding.
  • 7B LLM-based embedding models achieve state-of-the-art performance on the English benchmark but are outperformed by smaller XLM-R Large-based models in highly multilingual or low-resource settings.

LIMITATIONS

MMTEB expands multilingual coverage but remains uneven and carries limitations in translation bias and collaboration credit assignment. Low-resource languages are underrepresented overall, while human-translated data may favor English-oriented evaluation and training.

  • English leakage: Human-translated datasets can introduce English leakage by transferring labels from English samples to translations.The paper gives SIB200ClusteringS2S as an example where this process may bias evaluation toward English or translated content.
  • Credit assignment: Equal credit for dataset submissions did not reflect differences in contributor effort.Some datasets were readily available, whereas others required reformulation, HTML parsing, and multiple review rounds.
  • Language representation: The benchmark’s language distribution is skewed toward high-resource languages, with low-resource languages better represented in only some task categories.The authors identify bitext-mining and classification as examples of categories with stronger low-resource representation.

ETHICAL CONSIDERATIONS

MMTEB acknowledges that its large-scale evaluation has environmental costs and depends on collaborative datasets with varying quality and provenance. The project tracks emissions, but further work is needed to identify and minimize dataset biases.

  • Environmental impact: The benchmark’s compute requirements create an environmental impact across evaluation tasks.The authors add emissions tracking with codecarbon to measure kilograms of CO2-equivalents and estimate each task’s carbon footprint.
  • Dataset quality: MMTEB datasets vary in data quality and origin because the benchmark is a collaborative project.The paper states that additional efforts are still required to identify and minimize biases in the benchmark datasets.
  • Task evaluation: Classification evaluation repeatedly samples training data and fits logistic regression before calculating performance metrics.The process uses 8–16 samples per label, a maximum of 100 iterations, and 10 repetitions for robustness.
  • Task evaluation: Pair classification embeds two texts, computes a model-specified distance, and selects the best binary threshold for scoring.Supported distance functions include cosine, Euclidean, dot product, and Manhattan distance.
  • Task evaluation: Bitext mining embeds matching sentence pairs and identifies the closest match using cosine similarity.Performance metrics are then reported for the retrieved matches.
  • Task evaluation: Clustering embeds documents, samples repeated subsets, applies K-means, and compares estimated clusters with labels.The revised formulation samples subsets from already embedded documents, allowing one document to be reused across multiple subsets.
  • Task evaluation: Retrieval embeds queries and corpus documents, ranks documents by similarity, and scores results against query–document relevance mappings.Models may embed queries and documents differently depending on the model.
  • Task evaluation: Instruction retrieval adds query-specific relevance instructions to traditional retrieval and evaluates performance with Robustness@10.The instructions specify criteria for document relevance rather than applying generically to the whole dataset.

B.3 NOVEL DATASETS

MMTEB adds diverse multilingual and domain-specific evaluation tasks, including retrieval, reranking, clustering, and instruction-oriented datasets. Its task construction combines novel datasets, multilingual coverage, quality controls, and computationally efficient evaluation practices.

  • PublicHealthQA introduces COVID-19 question-answer retrieval data sourced from CDC and WHO pages collected between 2019-12 and 2020-04.
  • WebLINXReranking introduces HTML reranking based on relevance to web actions such as clicks, hovers, and text insertions.
  • WikiClustering creates a multilingual clustering benchmark from Wikipedia main-topic classifications and currently covers 14 specified languages.
  • WikipediaRetrievalMultilingual and WikipediaRerankingMultilingual use multilingual-LLM-generated questions grounded in Wikipedia articles to build retrieval and reranking datasets.
  • The benchmark defines diverse domains, but domain labels can overlap and the authors do not claim the definition is precise or comprehensive.
  • ClusteringFast evaluates 9 English clustering tasks on 13 models using stratified document subsampling and generally preserves model-ranking agreement with the original approach.The approach often produces smaller V-measure variance, occurring on 5 of 9 datasets.

C.1.2 RETRIEVAL

MMTEB reduces retrieval evaluation cost by retaining representative documents and testing hard-negative subsampling across sparse- and dense-judgment datasets. The analysis finds that at least 100 hard negatives generally preserve useful scores and rankings, motivating a conservative 250-document setting.

  • Retrieval downsampling evaluates sparse NQ and densely judged TREC-COVID datasets across multiple models and hard-negative counts.The analysis uses NQ with one relevant document per query and TREC-COVID with more than 500 relevant documents per query.
  • Manually adding the ground-truth relevant document to hard negatives did not help because most models already ranked it within the top N.
  • NQ model rankings remain stable with one hard negative, whereas TREC-COVID rankings become unstable starting at 100 hard negatives.
  • At 100 or more hard negatives, absolute scores are relatively similar to the original evaluation, while fewer negatives can produce inflated or unstable scores.
  • The retrieval procedure retains the top 250 ranked documents per query using a TREC pooling strategy to reduce collection size while preserving evaluation behavior.

E FULL RESULTS

MMTEB evaluates a large and heterogeneous task collection and makes detailed results available for analysis by language, domain, and task. Its tooling supports inspecting specific model results and comparing model behavior across languages.

  • MMTEB evaluations span more than 500 tasks, with multiple language subsets covering more than 1000 languages.
  • The public versioned results repository provides scores, runtimes, evaluation times, performance metrics, CO2 emissions, and version information by language subset.
  • The mteb package can load and aggregate scores for selected models across English and French retrieval tasks in the Legal domain.
  • Figure 12 relates model rank on MTEB(Multilingual) to the total number of speakers of each language using moving-average trendlines.

F.1 ABSTENTION FOR RETRIEVAL AND RERANKING TASKS

This section introduces model abstention ability as a score-calibration evaluation for Retrieval and Reranking tasks. It measures whether selectively abstaining on low-confidence instances improves performance, using the normalized area under the metric-abstention curve.

  • Abstention evaluation: Model abstention ability is proposed as an additional score-calibration evaluation alongside existing Retrieval and Reranking ranking metrics.The approach uses the implementation of Gisserot-Boukhlef et al. (2024).
  • Abstention evaluation: A model abstains when its confidence c(q, d1, · · ·, dk) falls below threshold τ, with varying thresholds producing different abstention rates.The confidence function operates on one query and k candidate documents.
  • Abstention evaluation: Effective abstention is indicated when the metric score increases as the abstention rate rises.
  • nAUC computation: nAUC summarizes abstention performance as the normalized area under the metric-abstention curve, computed from a confidence function, metric function, and labeled test dataset.The procedure evaluates performance across multiple abstention thresholds, then normalizes the resulting AUC using lower and upper bounds.
  • nAUC computation: The normalization compares observed AUC with a lower bound representing no improvement from abstention and an oracle-based upper bound representing optimal selective retention.The upper bound is obtained by retaining the best instances at each abstention rate using true labels.

H.1 BENCHMARK CREATION

This section describes community-created benchmarks extending MTEB across languages, task categories, and specialized evaluation settings. The collection includes multilingual, regional, English, code, and reasoning-oriented benchmarks, with task inventories documented in accompanying tables.

  • Benchmark landscape: MTEB includes maintained language-specific and multilingual benchmarks, including English, Scandinavian, French, German, and Russian collections.The section refers readers to the benchmark registry for an up-to-date overview.
  • Language-focused benchmarks: Russian was underrepresented in the original benchmark, so the new multilingual benchmark adds Russian datasets covering the main MMTEB tasks.The Russian portion contains 18 datasets across 7 main tasks after data cleaning and automatic filtering.
  • Specialized benchmarks: RAR-b evaluates whether embedding models retrieve correct answers to reasoning questions as top similar results, with and without instructions.It targets reasoning-level understanding and representation of nuanced expressions beyond STS or similar established evaluations.
  • Specialized benchmarks: RAR-b comprises 17 tasks from 15 datasets spanning commonsense, temporal, and spatial reasoning.
  • Regional benchmarks: MTEB(Europe) starts from 420 tasks over European and Schengen-area languages, while MTEB(Indic) is reduced to 23 final tasks after filtering and review.MTEB(Indic) began with 55 tasks and retained 44 after manual filtering before final selection.
Loading 2502.13595v4…