Source-linked AI summary
MTEB: Massive Text Embedding Benchmark
Niklas Muennighoff, Nouamane Tazi, Loïc Magne, Nils Reimers
TL;DR
Text embedding evaluation often covers too few tasks to show whether models transfer beyond STS and classification. MTEB addresses this by benchmarking diverse datasets, languages, and embedding tasks at scale. Its results show strong task-dependent variation rather than a universally best model, while open-source tooling and a leaderboard support reproducible comparison.
Problem
Current embedding evaluations rarely cover the breadth of use cases, leaving transferability to tasks such as search and clustering unclear.
Method
MTEB unifies 8 embedding tasks across 58 datasets and 112 languages, evaluates over 30 models, and provides open-source benchmarking software and a leaderboard.
Results
Model performance varies strongly across tasks, with no model achieving state-of-the-art performance on all tasks.
Takeaways & Limitations
MTEB provides broad baselines and evaluation infrastructure intended to make embedding-model selection and future research easier.
Takeaways & Limitations
MTEB has author-acknowledged limitations despite its aim to provide holistic performance reviews.
Abstract
from arXiv · showhide
Text embeddings are commonly evaluated on a small set of datasets from a single task not covering their possible applications to other tasks. It is unclear whether state-of-the-art embeddings on semantic textual similarity (STS) can be equally well applied to other tasks like clustering or reranking. This makes progress in the field difficult to track, as various models are constantly being proposed without proper evaluation. To solve this problem, we introduce the Massive Text Embedding Benchmark (MTEB). MTEB spans 8 embedding tasks covering a total of 58 datasets and 112 languages. Through the benchmarking of 33 models on MTEB, we establish the most comprehensive benchmark of text embeddings to date. We find that no particular text embedding method dominates across all tasks. This suggests that the field has yet to converge on a universal text embedding method and scale it up sufficiently to provide state-of-the-art results on all embedding tasks. MTEB comes with open-source code and a public leaderboard at https://github.com/embeddings-benchmark/mteb.
1 Introduction
MTEB addresses the narrow and inconsistent evaluation of text embeddings by covering diverse tasks and simplifying reproducible comparison. Its results show that model strengths differ by task, so no single embedding solution dominates.
- Motivation: Existing evaluations often focus on STS and classification, leaving transferability to search and clustering unclear.STS is also reported to correlate poorly with other real-world use cases, while differing preprocessing and hyperparameters can confound comparisons.
- Contribution: MTEB covers 8 embedding tasks, 58 datasets, and 112 languages, with open-source software and a leaderboard for accessible evaluation.Adding a model requires less than 10 lines of code.
- Evaluation: Over 30 models are evaluated with additional speed and memory benchmarks to provide a holistic view across open-source and API-accessible systems.The evaluation includes models such as the OpenAI Embeddings endpoint.
- Findings: Different models dominate different tasks, including SimCSE’s strong STS performance but low clustering and retrieval performance.The benchmark is intended to make model selection and future embedding research easier.
2 Related Work
Prior embedding evaluation centered heavily on STS and classifier-based tests, leaving direct comparison tasks such as retrieval and clustering underrepresented. MTEB unifies these and other datasets in a common framework.
- Existing evaluation: SemEval datasets primarily evaluate STS, while SentEval aggregates STS datasets but focuses on classifiers trained on embeddings.These evaluations do not directly test retrieval or clustering.
- MTEB: MTEB unifies SemEval STS11–STS22, BEIR, and datasets from additional embedding tasks in one accessible evaluation framework.This provides a more holistic review of text embedding models.
- Embedding models: Transformer-based embedding models add context awareness, and fine-tuning with contrastive objectives is commonly used for competitive performance.The variety of pretrained transformers creates many candidate embedding models for practitioners to compare.
- Embedding models: MTEB benchmarks both word embedding and transformer models while quantifying the gains from slower context-aware models.This directly incorporates model efficiency into comparison.
3 The MTEB Benchmark
MTEB is designed as a diverse, simple, extensible, and reproducible benchmark spanning multiple embedding tasks, text lengths, and languages. Its datasets support task-specific evaluation while revealing similarities across datasets and task types.
- 3.1 Desiderata: MTEB’s desiderata are diversity, simplicity, extensibility, and reproducibility across 8 tasks, 58 datasets, and 112 languages.Its API accepts models that map lists of texts to consistently shaped vectors, while versioning supports reproducible results.
- 3.2 Tasks and Evaluation: The benchmark includes bitext mining, classification, clustering, pair classification, reranking, retrieval, STS, and summarization.These task types evaluate embeddings through different comparison or downstream scoring procedures.
- 3.2 Tasks and Evaluation: Retrieval embeds queries and corpus documents, ranks documents by cosine similarity, and uses nDCG@10 as its main metric.Several ranking metrics are computed for multiple values of k.
- 3.2 Tasks and Evaluation: STS embeds sentence pairs and evaluates cosine-similarity distances against ground-truth similarities using Pearson and Spearman correlations.Spearman correlation based on cosine similarity is the main metric.
- 3.3 Datasets: MTEB includes sentence-to-sentence, paragraph-to-paragraph, and sentence-to-paragraph settings to vary input length and comparison structure.Sentence-to-paragraph retrieval uses single-sentence queries against long, multi-sentence documents.
- 3.3 Datasets: Across 56 datasets, similar corpora, STS variants, S2S/P2P variants, and scientific datasets form recognizable similarity clusters.Scientific datasets from reranking, retrieval, and clustering can be highly similar despite differing task types.
4 Results
MTEB results show substantial variation across tasks, with different models excelling in retrieval, similarity, classification, clustering, and multilingual settings. Performance generally increases with model size, but model selection also involves speed, embedding size, training data, and task fit.
- Overall results: No model claims state-of-the-art across all seven English tasks, and self-supervised methods remain behind supervised methods for competitive embedding performance.The results show considerable task variability and a persistent gap between self-supervised and supervised methods.
- Scale and multilinguality: Performance strongly correlates with model size, but multilingual results vary: LaBSE leads bitext mining, while multilingual MPNet is strongest overall for classification and STS.SGPT-BLOOM-7B1-msmarco performs well on several languages seen extensively during pre-training, while larger models incur significant cost.
- Classification: ST5 models dominate classification, with ST5-XXL averaging 3% ahead of OpenAI Ada Similarity.The comparison covers most classification datasets.
- Clustering: MPNet is on par with ST5-XXL on clustering despite being almost 50x smaller, while clustering performance may depend on training across diverse datasets.Models fine-tuned on a single NLI dataset may produce incoherent embeddings on unseen topics.
- Retrieval: SGPT-5.8B-msmarco is best on the BEIR subset and full BEIR benchmark, whereas models geared toward STS perform badly on retrieval.Retrieval distinguishes query and document texts, and task-specific model orientation matters.
- STS & Summarization: ST5-XXL has the highest STS performance, while GTR and SGPT-msmarco perform badly on STS, separating retrieval and similarity use cases.The results highlight a bifurcation between asymmetric retrieval embeddings and symmetric similarity embeddings.
5 Conclusion
MTEB provides a broad, open benchmark for text embeddings, covering diverse tasks, languages, and models. Its results show strong task-dependent variation, with no model achieving state-of-the-art performance across all tasks.
- MTEB covers 8 text embedding tasks, up to 15 datasets per task, and 112 languages.
- Open-sourcing MTEB and its leaderboard provides a foundation for further progress in text embeddings.
- Nearly 5,000 experiments on over 30 models establish baselines for future embedding research.
- Model performance varies strongly across tasks, with no model claiming state-of-the-art results on all tasks.
A Datasets
MTEB includes diverse clustering datasets built from scientific papers, StackExchange posts, Reddit posts, and newsgroup articles. Its custom splits test coarse, fine-grained, and multi-scale clustering under varying cluster sizes.
- MTEB clustering datasets include scientific papers, StackExchange posts, Reddit posts, and Twenty Newsgroups articles.
- Scientific-paper clustering uses titles or title-plus-abstract text with human-assigned category labels.
- Main-category splits evaluate coarse clustering, while secondary-category splits evaluate fine-grained distinctions.
- Secondary-category clustering across all main categories tests fine-grained and multi-scale capacity simultaneously.
- Each dataset, split, and strategy samples subsets of labels and articles, producing varying cluster amounts and sizes.
A.2 Classification
The appendix catalogs MTEB datasets across classification, pair classification, bitext mining, reranking, retrieval, STS, and summarization. These datasets span sentiment, intent, paraphrase, translation, ranking, similarity, and summary evaluation settings.
- Classification datasets cover review polarity, counterfactual detection, intents, scenarios, toxicity, sentiment, emotions, and duplicate questions.
- Several classification datasets are multilingual, including AmazonCounterfactual with 4 languages, MassiveIntent and MassiveScenario with 51, and AmazonReviews with 6.
- Pair-classification datasets label question or tweet pairs as duplicates, paraphrases, or non-matches.
- Bitext mining datasets provide aligned translated sentences, including BUCC in 5 languages and Tatoeba in 112 languages.
- Reranking datasets rank related news, scientific papers, or duplicate questions, while STS datasets measure sentence similarity on monolingual and crosslingual pairs.
- MTEB includes publicly available BEIR retrieval datasets such as ArguAna, ClimateFEVER, MSMARCO, NQ, Quora, and SciFact.
B Limitations of MTEB
MTEB aims to provide holistic performance reviews but explicitly acknowledges that the benchmark has limitations.
- MTEB aims to provide holistic performance reviews while acknowledging that the benchmark has limitations.
1. Long document datasets
MTEB includes varied text lengths, but it does not yet cover very long documents. This limits coverage for use cases such as retrieval.
- MTEB includes datasets with multiple text-length settings, including S2S, P2P, and S2P.
- The longest MTEB datasets contain only a few hundred words.
- Very long documents remain absent, despite their relevance to retrieval use cases.
2. Task imbalance
MTEB’s dataset coverage is uneven across tasks and languages, and its scope is limited to natural-language applications. These boundaries affect how broadly its aggregate results can be interpreted.
- Summarization has only one dataset, so average scores are weighted toward tasks with more datasets, especially retrieval, classification, and clustering.
- MTEB includes multilingual classification, STS, and bitext mining, but retrieval and clustering remain English-only.
- MTEB excludes code datasets and therefore does not benchmark code models.
- The benchmark focuses on natural-language applications and leaves text embeddings for other modalities to future work.
F Additional results
Additional results are reported through dataset-level tables, model-link tables, and leaderboard-accessible data. Correlation analyses compare model performance across tasks and architectures.
- Tables 11 onward report results for individual MTEB datasets, with additional JSON results available on the Hugging Face Hub and the leaderboard.
- Figure 6 shows correlation heatmaps for model performance and MTEB tasks; clustering and reranking correlate most strongly, whereas summarization and classification correlate more weakly with other tasks.
- Tables 10–14 provide model links and results for English, multilingual bitext mining, multilingual classification, and multilingual STS.