MMTEB: Massive Multilingual Text Embedding Benchmark

• Existing embedding benchmarks tend to focus on specific domains, tasks, or literary forms, while language coverage remains limited, especially for low- to mid-resource languages.Source: §1 · p. 2
• The benchmark was built through open collaboration and systematic dataset tests, metadata requirements, and review, with contributors from diverse linguistic and professional backgrounds.Source: §1 · p. 2
• MMTEB comprises more than 500 tasks across 10 categories and over 250 languages, spanning domains from fiction and social media to medical and programming documentation.Source: §1 · p. 2
• MMTEB integrates benchmarks for instruction following, long-document embedding, reasoning, and cross-lingual retrieval.The cited works include Winata et al. (2021), Weller et al. (2024), Zhu et al. (2024), Xiao et al. (2024a), Su et al. (2024), and Franco-Salvador et al. (2014).Source: §1 · p. 2
• 3.11 hours on an H100 GPU evaluates a 7B model using 2% of the original documents (6% of characters), while retaining sensitivity to model rankings.The authors frame this efficiency as a way to make evaluation more accessible to low-resource communities facing limited computing resources.Source: §1 · p. 3

2.1 OPEN SCIENCE EFFORT

• To broaden domain applicability, MMTEB recruited industry professionals, low-resource language communities, and researchers, using a point-based authorship system modeled on Lovenia et al. (2024).GitHub was used for transparent coordination.Source: §2.1 · p. 3

2.2 ENSURING TASK QUALITY

• Each submitted task was reviewed by at least one main contributor and required metadata on annotation and dataset sources, licensing, dialects, and citations.Two multilingual models were run to flag tasks with scores near random, near-perfect, or roughly similar; contributors familiar with the target language were consulted where possible, and difficult tasks could still be included if the results reflected task difficulty rather than poor quality.Source: §2.2 · pp. 3–4

2.3 ACCESSIBILITY AND BENCHMARK OPTIMIZATION

• Because extensive benchmark evaluation is computationally costly and low-resource communities may face constraints, MMTEB pursues more efficient evaluation.The paper notes that moderately sized LLMs can take up to two days to run on a single A100 GPU on MTEB(eng, v1).Source: §2.3 · p. 4

2.3.1 DOWNSAMPLING AND CACHING EMBEDDINGS

• 16.11x average speedup across clustering tasks preserved model rankings (Average Spearman correlation: 0.96) using bootstrapping with a 4% corpus subsample and 10 sets.For retrieval, TREC pooling retained the top 250 documents per query and sampled 1,000 queries for datasets exceeding that size, reducing the largest collections to at most 250,000 documents while preserving ranking performance. For bitext mining, embedding caching made computations linear in the number of languages, reducing embedded English Flores documents from 410,000 to 1,012.Source: §2.3.1 · p. 4

2.3.2 ENCOURAGING SMALLER DATASET SUBMISSIONS

• Contributors were encouraged to downsample datasets before submission using stratified splits based on target categories, then compare scores before and after downsampling.The stratification was intended to preserve the datasets’ ability to differentiate candidate models.Source: §2.3 · p. 4, §2.3.2 · p. 5

2.3.3 TASK SELECTION

• Task selection treats tasks as prediction features and iteratively removes the most predictable task, using Spearman rank correlation to preserve relative model rankings.The procedure predicts each model’s score on a held-out task from other tasks and can add criteria for task diversity and language representation.Source: §2.3.3 · p. 5

2.4 BENCHMARK CONSTRUCTION

• MMTEB introduces multilingual, regional, language-specific, and domain-specific benchmarks, and provides code for communities and companies to create tailored benchmarks.Examples include MTEB(Multilingual), MTEB(Europe), MTEB(Indic), MTEB(eng, v2), CoIR for code retrieval, and LongEmbed for long-document retrieval.Source: §2.4 · p. 5
• MMTEB’s multilingual benchmark narrows an initial scope of 550 tasks to 343 tasks covering >250 languages, then selects a diverse final set of 131 tasks.MTEB(eng, v2) starts from optimized MTEB(eng, v1) tasks with Spearman correlation of 0.97, p < .0001 (Pearson 0.99, p < .0001), and selects 41 tasks; selection uses intertask correlation and retains tasks to protect diversity, categories, and languages.Source: §2.4 · pp. 5–6

3.1 MODELS

• The study benchmarks multilingual models from LaBSE, MPNet, MiniLM, and e5 alongside Mistral-based 7B models to examine model size and instruction finetuning.Revision IDs, implementations, and prompts are listed in Appendix G; the versioned results repository includes performance, runtime, emissions, and model metadata.Source: §3.1 · p. 6

3.2 EVALUATION SCORES

• Scores cover task-wide averages, task-category scores, and category-weighted averages; model ranks use Borda count, with tournament Borda count for ties.Each task acts as a preference voter when model scores are aggregated.Source: §3.2 · pp. 6–7

3.3 MULTILINGUAL PERFORMANCE

• The showcased benchmarks are fully multilingual MTEB(Multilingual), MTEB(Europe), and MTEB(Indic), which target language settings with varying resource levels.Selected-model performance is presented in Table 2; per-task and cross-domain metrics are in Appendix E.Source: §3.3 · p. 7

4 ANALYSIS AND DISCUSSION

• Narrow-focus multilingual models favor mid/high-resource or European languages, while multilingual-e5-large-instruct leads on lower-resource European and all Indic languages despite having fewer parameters.Mistral-based models’ performance decreases and becomes more volatile as the number of native speakers falls.Source: §4 · pp. 7–9, Figure 3 · p. 9
• Instruction tuning improves performance across task categories, especially bitext mining and clustering; multilingual-e5-large-instruct generally outperforms larger e5-mistral-7b-instruct and GritLM-7B (Conneau et al., 2019).This advantage persists despite many tasks using generic category prompts rather than model-specific prompts. GritLM remains best in class for MTEB(Multilingual) retrieval and outperforms multilingual-e5-large-instruct on MTEB(Code) and MTEB(eng, v2); it also has a higher maximum sequence length.Source: §4 · p. 7, Table 2 · p. 8
• English models perform poorly on non-English languages, while LaBSE has relatively high bitext performance.Source: §4 · p. 9

MTEB(eng, v1) vs. zero-shot MTEB(eng, v2)

• MTEB(eng, v1) and zero-shot MTEB(eng, v2) correlate at Spearman 0.90 (Pearson 0.96); v2 reduces the task count from 56 to 40 and runs in 3.11 hours for GritLM-7B or 0.81 hours for all-MiniLM-L12 on an H100.The smaller English models perform worse on v2, likely because MS MARCO and Natural Questions were removed to make the benchmark zero-shot.Source: §4 · p. 9

Text Embedding Benchmarks

• MMTEB expands prior multilingual embedding benchmarks to over 1,000 languages (250 excluding bitext-mining tasks), 10 embedding tasks, and over 400 datasets.BEIR (Thakur et al., 2021) introduced diverse public IR datasets; MTEB (Muennighoff et al., 2023b) spans 58 tasks and 112 languages, while MIRACL and MINERS (Winata et al., 2024b) cover monolingual retrieval and multilingual semantic retrieval, respectively.Source: §5 · pp. 9–10

Massive Collaborative Projects

• MMTEB extends collaborative benchmark creation by focusing on text embeddings, using a custom point system to incentivize contributions, and handling communication openly via GitHub.The paper situates this effort among collaborative NLP projects spanning model, dataset, benchmark, and other artifact creation.Source: §5 · p. 10

• MMTEB is an open collaboration benchmark with more than 500 tasks across more than 1000 languages, yielding MTEB(Multilingual), MTEB(Indic), and MTEB(Europe).Source: §6 · pp. 10–11
• The authors propose a simple approach to construct further benchmarks; retrieval hard-negative mining and bootstrapping clustering evaluation to reuse encoded documents across sets notably reduce samples needing embedding.Source: §6 · p. 11
• Large (7B) LLM-based embedding models achieve state-of-the-art performance on English, but smaller XLM-R Large-based models outperform them in highly multilingual or low-resource settings, even with prompt-based improvements.Source: §6 · p. 11

English Leakage

• Allowing human translations may bias tasks such as SIB200ClusteringS2S toward English or translated content, potentially encouraging developers to increase such content in pre-training data.The task transfers labels from English samples to their translations.Source: LIMITATIONS · p. 11

Credit Assignment for Large-scale Collaborations

• MMTEB's collaboration point system defined contribution terms but awarded equal points for dataset submissions despite differences in the effort required.Some submissions were readily available, while others required reformulation, HTML parsing, and multiple review rounds.Source: LIMITATIONS · p. 11

Languages Representation

• Despite covering over 250 languages and 500 tasks, MMTEB skews toward high-resource languages, with low-resource languages better represented in bitext-mining and classification.Figure 6 plots the 100 languages with the most tasks and omits English, which has 290 tasks.Source: LIMITATIONS · p. 11, Figure 6 · p. 11

ETHICAL CONSIDERATIONS

• MMTEB tracks task-level carbon emissions with codecarbon and acknowledges that further work is needed to identify and reduce dataset bias.The benchmark combines datasets of varying quality and origin.Source: ETHICAL CONSIDERATIONS · p. 11

A CONTRIBUTIONS

• New datasets, new task categories, annotations, and fixes earned contribution points, with awards varying by contribution type and effort.New datasets earned points for implementation and each added language; annotations earned one point, while fixes typically earned 2–10 points.Source: A CONTRIBUTIONS · pp. 33–34
• Model runs, reviews, writing, and coordination also counted toward coauthorship, which required 10 points.Model runs typically earned one point per model on relevant tasks; reviews earned two points per reviewer, while writing and coordination were scored by relative effort.Source: A CONTRIBUTIONS · p. 34

B OVERVIEW AND CONSTRUCTION OF TASKS

• The appendix surveys existing MTEB tasks, explains task construction from existing datasets, and introduces newly built MMTEB datasets.The overview directs readers to Sections B.1, B.2, and B.3 for these topics, respectively.Source: B OVERVIEW AND CONSTRUCTION OF TASKS · p. 35

B.1 INTRODUCTION TO BENCHMARK TASKS

• MMTEB adds multi-label classification for tasks with multiple labels per document, using repeated downsampled training sets and a K-nearest-neighbours classifier.Training sets contain eight instances per unique label; evaluation uses Accuracy, F1, and LRAP.Source: §B.1 · p. 37

B.2 TASK CONSTRUCTION

• Existing-dataset task construction applies minimal preprocessing, reduces dataset size where applicable, filters for relevance, and tests whether tasks differentiate candidate models.Newly introduced datasets are described separately in Section B.3.Source: §B.2 · p. 38
• Classification tasks were generally trimmed, bitext tasks sized to preserve model rankings, and clustering tasks drawn from labeled corpora, sometimes reformulated hierarchically.Primary metrics include accuracy by default for general classification, average precision for pair classification, F1 for bitext mining, and nDCG@10 for retrieval unless dataset publications specify otherwise.Source: §B.2 · p. 38
• Retrieval adapted QA and news data, instruction retrieval used context-rich FollowIR queries, reranking spanned several domains, and STS drew on multilingual similarity and paraphrase benchmarks.Multi-label datasets used higher-order stratification to balance labels; primary metrics were accuracy for multi-label classification, MAP for reranking, and Spearman correlation for STS.Source: §B.2 · pp. 38–39

B.3 NOVEL DATASETS

• New tasks include COVID-19 PublicHealthQA from CDC and WHO Q&A and FAQ pages collected from 2019-12 to 2020-04, and WebLINXReranking, derived from WebLINX, for reranking HTML elements by relevance to web-agent actions.The actions include clicks, hovers, and text insertions; the paper describes this as the first task to rerank HTML elements for actions in web environments.Source: §B.3 · p. 39
• WikiClustering converts Wikipedia main-topic classifications into multilingual paragraph-to-paragraph clustering, using first paragraphs, removing ambiguous or singleton topics, and sampling 2,048 articles.It is implemented for Bosnian, Catalan, Czech, Danish, Basque, Manx, Ilokano, Kurdish, Latvian, Minangkabau, Maltese, Scots, Albanian, and Walloon.Source: §B.3 · p. 39
• WikipediaRetrievalMultilingual and WikipediaRerankingMultilingual use target-language questions generated from Wikipedia articles to support retrieval and reranking.For the described construction, 1,500 articles from the top 100,000 viewed were sampled; each selected a window of nine consecutive paragraphs, with the middle as positive and the surrounding eight as hard negatives.Source: §B.3 · pp. 39–40

B.4 TASK METADATA

• MMTEB requires task metadata before task inclusion; its domain labels cover topical fields and text types, are not mutually exclusive, and are neither claimed precise nor comprehensive.The domain definition aligns with the Universal Dependencies project.Source: §B.4 · p. 40, §B.4.1 · p. 42

C.1 SPEEDING UP TASKS

• The appendix targets runtime reduction for clustering and retrieval while maintaining relative model rankings, detailing the approach introduced in the main paper.For clustering, documents were sampled by target category, and ranking agreement was evaluated against the original approach.Source: §C.1 · p. 42, §C.1.1 · p. 42

C.1.1 CLUSTERING

• 16.11x average speedup was achieved across clustering tasks while maintaining relative model ordering; e5-large had the largest average speedup at 16.93x.The authors expect the speedup effect to be more pronounced among models with 7 billion or more parameters.Source: §C.1.1 · p. 42, Table 6 · p. 42
• Clustering rankings were compared using mean V-measure scores and Spearman correlation; significant rankings were based on bootstrapped V-measure distributions.The downsampled clustering evaluation used a 4% sample, except for RedditS2S and StackExchange, which used n_samples = 32768.Source: §C.1.1 · p. 42
• ClusteringFast showed generally strong ranking agreement, smaller V-measure spread on 5 of 9 datasets, and the lowest significant Spearman score on Reddit P2P, whose documents were shortest.The study evaluated nine single-level English clustering tasks across 13 models; the downsampling fraction was 4%, except for RedditS2S and StackExchange, where n_samples = 32768.Source: §C.1.1 · pp. 42, 44, Figure 8 · p. 43, Table 7 · p. 44

C.1.2 RETRIEVAL

• The analysis tested ranking fidelity, sparse versus dense judgments, and whether e5-large-v2 alone could supply hard negatives, using NQ, TREC-COVID, and a broad model set.Models were evaluated with 2, 5, 10, 50, 100, 500, or all hard negatives per query; NQ has one relevant document per query and TREC-COVID more than 500.Source: §C.1.2 · pp. 44–45
• Including the ground-truth relevant document among hard negatives made little difference because most models already ranked it in the top N.The analysis found manually including the document offered little help.Source: §C.1.2 · p. 44
• NQ rankings stayed stable with one hard negative, while TREC-COVID rankings became unstable below 100; scores were similar at 100+ negatives, motivating a conservative 250 per query.NQ scores rose toward very high values with few negatives; TREC-COVID scores were more stable but swung more with smaller document sets.Source: §C.1.2 · p. 45, Figure 10 · p. 45

C.2 CODE OPTIMIZATIONS

• Dataset-loading improvements reduced bitext-mining load time by almost a factor of 100, to under a minute, by optimizing file requests and restructuring datasets and code.The issue arose with multilingual and cross-lingual datasets containing many small files, where network-request and gzipped-file handling overhead could make even datasets under 10 MB take hours to load.Source: §C.2 · p. 45
• MTEB deduplicated repeated dataset samples, introduced a second task version to preserve compatibility with existing benchmark scores, and added a local cache to avoid re-encoding samples.MindSmallReranking is given as an example of a dataset with repeated samples.Source: §C.2 · p. 45

D.1 TASKS

• MMTEB’s automatically updated task documentation lists metadata including licenses, task categories, and domains.The documentation provides an overview of implemented tasks.Source: §D.1 · p. 46

D.2 LANGUAGES

• The language overview identifies the top 100 of 1,051 ISO 639-3 languages and their task counts in Table 8.Table 8 lists the languages using ISO 639-3 codes; the supplied passage does not give the individual counts.Source: §D.2 · p. 46, Table 8 · p. 48

D.3 EXAMPLES

• Tables 9 and 10 provide examples for new MMTEB task types, including instruction retrieval; examples for several established task types are referenced to the MTEB paper.The referenced task types include bitext mining, classification, clustering, pair classification, reranking, retrieval, STS, and summarization.Source: Table 9 · p. 48, §D.3 · p. 48

E FULL RESULTS

• Evaluations covered more than 500 tasks, with multiple tasks spanning language subsets that together covered more than 1,000 languages.Source: E FULL RESULTS · p. 48
• Results are available in a public, versioned repository with per-language-subset metrics, run and evaluation times, CO2 emissions, and version numbers.Source: E FULL RESULTS · p. 49
• The mteb package supports loading and aggregating results, including scores for English and French retrieval tasks in the Legal domain.Figure 11 illustrates this example for a set of models.Source: E FULL RESULTS · p. 49, Figure 11 · p. 49

E.1 PERFORMANCE PER NUMBER OF SPEAKERS

• Figure 12 presents model rankings on MTEB (Multilingual) by total language-speaker count, with moving-average trendlines using a window of 10.Source: Figure 12 · p. 50

F.1 ABSTENTION FOR RETRIEVAL AND RERANKING TASKS

• The benchmark adds model abstention ability to existing ranking metrics to assess score calibration in retrieval and reranking (Muennighoff et al., 2023b; Gisserot-Boukhlef et al., 2024).Source: §F.1 · p. 50
• Abstention is determined by whether confidence for a query and its candidate documents falls below a threshold, with effective abstention raising metric scores as abstention rates increase.Source: §F.1 · p. 50
• Evaluation varies thresholds, measures task performance on each resulting subset, and computes the area under the metric-abstention curve.Source: §F.1 · p. 50
• The implementation uses maximum cosine similarity, score standard deviation, or the gap between the top two cosine scores as confidence functions.Source: §F.1 · p. 50
• Normalized AUC scales the model’s curve area between a lower bound where performance stays constant and an upper bound set by a label-informed oracle.The oracle retains the best instances at each abstention rate.Source: §F.1 · pp. 50–51

G MODELS

• Models were run on the same version of each task, although different MTEB versions were used across tasks because debugging and computational costs prevented uniform reruns.Source: G MODELS · p. 51
• Model revisions and prompts are available through the model registry, while JSON results include performance metrics, metadata, emissions, runtime, and the MTEB version used.Source: G MODELS · p. 51

H.1 BENCHMARK CREATION

• Contributed coverage includes language-specific and specialized instruction-retrieval, law, low-resource bitext, and code-retrieval benchmarks; the registry provides an up-to-date overview of maintained benchmarks.Source: §H.1 · p. 51
• The Russian addition contains 18 manually constructed datasets across seven main tasks, addressing the language’s limited representation in earlier benchmarks.Source: §H.1 · pp. 51–52
• RAR-b (Xiao et al., 2024a) evaluates retrieval of reasoning-question answers with and without instructions across 17 tasks and 15 datasets spanning five reasoning types.It assesses whether correct answers are retrieved as top matches and probes representation of nuanced expressions beyond STS and traditional topical IR.Source: §H.1 · p. 52
• The European selection was reduced from 420 initial tasks to 96 through filtering, iterative predictability-based selection, language and category safeguards, and contributor review.Source: §H.1 · p. 52
• The Indic benchmark narrows 55 initial tasks to 44 after manual filtering and 23 in the final reviewed selection.Source: §H.1 · p. 52

H.2 BENCHMARK TASK OVERVIEW

• The task overview identifies listings for MTEB(Multilingual) (Tables 12–13), MTEB(Europe) (14), MTEB(Indic) (15), MTEB(eng, v2) (16), and MTEB(Code) (17).Source: §H.2 · pp. 52–53

H.3 PERFORMANCE ON MTEB(eng, v2)

• Table 18 reports representative-model performance across task categories on MTEB(eng, v2).The supplied passages do not provide its performance values.Source: §H.3 · p. 53, Table 18 · p. 56

H.4 PERFORMANCE ON MTEB(Code)

• Table 19 reports representative-model performance on MTEB(Code), with category metrics omitted because all code-related tasks are retrieval tasks.The supplied passages do not provide its performance values.Source: §H.4 · p. 53, Table 19 · p. 57