MTEB: Massive Text Embedding Benchmark

1 Introduction

• Narrow embedding evaluations leave transferability uncertain, while STS poorly correlates with other real-world uses (Neelakantan et al., 2022; Wang et al., 2021).SimCSE (Gao et al., 2021b) and SBERT (Reimers and Gurevych, 2019) evaluate only on STS and classification, leaving search and clustering transferability unresolved.Source: §1 · p. 1
• Separate evaluation pipelines introduce preprocessing and hyperparameter differences that can obscure whether reported improvements reflect better embeddings or favorable evaluation choices.This encourages unvalidated deployment in new use cases or requires additional task-specific reevaluation.Source: §1 · p. 1
• 58 datasets spanning 112 languages and eight embedding tasks make up MTEB.The tasks are bitext mining, classification, clustering, pair classification, reranking, retrieval, STS, and summarization.Source: §1 · p. 1
• Less than 10 lines of code enable evaluation of any embedding model using MTEB’s open-source software.Datasets and the leaderboard are available on the Hugging Face Hub.Source: §1 · p. 1
• No single embedding model performs best across tasks; different models dominate different tasks.The evaluation covers over 30 open-source and API-accessible models, including speed and memory benchmarking. SimCSE (Gao et al., 2021b) performs poorly on clustering and retrieval despite strong STS performance.Source: §1 · pp. 1–2

2.1 Benchmarks

• SentEval (Conneau and Kiela, 2018) lacks retrieval and clustering, while USEB (Wang et al., 2021) omits retrieval and classification.SemEval datasets benchmark semantic textual similarity, whereas BEIR (Thakur et al., 2021) evaluates zero-shot information retrieval. SentEval emphasizes classifier fine-tuning and lacks easy support for transformer embeddings (Reimers and Gurevych, 2019); USEB consists mostly of reranking tasks.Source: §2.1 · p. 2
• MTEB unifies SemEval datasets (STS11 - STS22), BEIR, and datasets from other embedding tasks into an accessible framework for holistic evaluation.This brings datasets from different embedding tasks into a common evaluation framework.Source: §2.1 · p. 2

2.2 Embedding Models

• Transformers (Vaswani et al., 2017) introduce context awareness through self-attention, while SBERT (Reimers and Gurevych, 2019) demonstrates the benefit of additional fine-tuning for competitive embeddings.BERT (Devlin et al., 2018) combines transformer architecture with large-scale self-supervised pre-training and can produce embeddings by averaging. Recent models commonly use supervised contrastive fine-tuning on positive and negative text pairs (Gao et al., 2021b; Wang et al., 2021; Ni et al., 2021b; Muennighoff, 2022).Source: §2.2 · p. 2
• MTEB benchmarks word embeddings and transformers to quantify gains from often slower context-aware models amid uncertainty over which model best serves each use case.The variety of available pre-trained transformers creates a correspondingly large space of potential embedding models.Source: §2.2 · p. 2

3.1 Desiderata

• 58 datasets span eight tasks and 112 languages, including 10 multilingual datasets, under design goals of diversity, simplicity, extensibility, and reproducibility.A simple API accepts models producing consistently shaped text vectors; adding datasets requires one specification file, while new tasks require data-loading and evaluation implementations. Dataset and software versioning support reproducibility, and the paper’s results are available as JSON files.Source: §3.1 · pp. 2–3

3.2 Tasks and Evaluation

• Bitext mining evaluates cosine-nearest sentence matches with F1; classification evaluates logistic regression with accuracy; clustering evaluates mini-batch k-means with v-measure.Bitext matches commonly represent translations, while classification trains on embedded training examples with a maximum of 100 iterations before testing. Clustering uses batch size 32 and k equal to the label count (Pedregosa et al., 2011), with label-permutation-invariant v-measure.Source: §3.2 · p. 3
• Pair classification uses cosine-based average precision, reranking uses MAP, and retrieval uses nDCG@10 as their main evaluation metrics.Pair classification computes several embedding distances and threshold-based classification scores; reranking orders references by query cosine similarity and averages scores across queries. Retrieval ranks corpus documents by query cosine similarity and reuses BEIR datasets and evaluation (Thakur et al., 2021).Source: §3.2 · pp. 3–4
• STS and summarization both use Spearman correlation based on cosine similarity as their main metric.STS compares embedding similarities against continuous sentence-pair labels using Pearson and Spearman correlations. Summarization scores each machine summary by its closest human-summary embedding, then correlates those scores with human assessments.Source: §3.2 · p. 4

3.3 Datasets

• MTEB covers sentence-to-sentence, paragraph-to-paragraph, and sentence-to-paragraph comparisons, including sentence queries against longer retrieval documents.MTEB imposes no input-length limit, leaving truncation to individual models. Several clustering datasets offer sentence-level title comparisons and paragraph-level comparisons containing both titles and content.Source: §3.3 · pp. 4–5
• Dataset similarities reflect shared corpora, related dataset variants, and scientific domains, including similarities across task types and sentence- versus paragraph-level versions.Figure 2 compares averaged ST5-XXL embeddings from 100 samples per dataset using cosine similarity. ClimateFEVER and FEVER score 1 because they share a corpus, while scientific datasets show high similarities across reranking, retrieval, and clustering.Source: Figure 2 · p. 4, §3.3 · p. 5

4.1 Models

• The evaluated models span self-supervised encoders, context-independent word embeddings, supervised transformer encoders and decoders, and a context-aware LSTM model.Encoder training includes retrieval-oriented GTR (Ni et al., 2021b) and STS-oriented ST5 (Ni et al., 2021a); SGPT (Muennighoff, 2022) fine-tunes <0.1% of parameters with weighted-mean pooling, whereas cpt-text (Neelakantan et al., 2022) uses last-token pooling. Word embeddings trade context awareness for speed, while LASER (Heffernan et al., 2022) uses an LSTM trained on parallel data for bitext mining.Source: §4.1 · pp. 6–7

4.3 Efficiency

• Glove leads the maximum-speed cluster, MPNet and MiniLM balance speed and performance, and GTR-XXL, ST5-XXL, or SGPT-5.8B offer task-dependent maximum performance.The highest-performing cluster is slow, and SGPT-5.8B additionally requires substantial embedding storage. Embedding sizes range from 1.2 kB per example for Glove / Komninos to 16.4 kB for SGPT-5.8B.Source: Figure 4 · p. 7, §4.3 · p. 8

4.2 Analysis

• No model achieves state-of-the-art performance across all seven English tasks, and supervised methods retain a substantial advantage over self-supervised methods.Performance strongly correlates with model size, with multi-billion-parameter models dominating most MTEB tasks. Results vary even more at the individual-dataset level.Source: Figure 3 · p. 6, §4.2 · p. 7
• 3% ahead of the best non-ST5 model, OpenAI Ada Similarity, ST5-XXL achieves the highest average classification performance.Despite being almost 50x smaller, MPNet matches ST5-XXL on clustering, possibly benefiting from diverse fine-tuning data; retrieval query embeddings also compete with similarity embeddings, potentially benefiting from larger MSMARCO training data. GTR-XL and GTR-XXL lead pair classification, whose rankings differ significantly from STS despite similar framing.Source: §4.2 · pp. 7–8

4.4 Multilinguality

• LaBSE (Feng et al., 2020) dominates multilingual bitext mining, while LASER2 varies substantially across languages and other models struggle outside their pre-training languages.Evaluation covers 10 multilingual datasets across bitext mining, classification, and STS. Only the default multilingual LASER2 is evaluated for a one-to-one comparison; mixing language-specific variants might reduce its variability, while MPNet, MiniLM, and SGPT-BLOOM-7B1-msmarco perform poorly on unseen pre-training languages.Source: Figure 5 · p. 8, §4.4 · p. 9

4.2 Analysis

• MPNet and MiniLM perform strongly on reranking, but their substantial SciDocsRR advantage over larger models likely reflects training-data overlap (Cohan et al., 2020a).MTEB scores do not adjust for overlap with model-training datasets because controlling contamination is challenging. The authors believe averaging enough datasets makes these effects insignificant.Source: §4.2 · p. 8
• SGPT-5.8B-msmarco leads MTEB’s BEIR subset and full BEIR (Thakur et al., 2021; Muennighoff, 2022), whereas ST5-XXL leads STS.STS-focused models perform poorly on retrieval, while retrieval models perform poorly on STS, reflecting asymmetric query–document versus symmetric similarity use cases (Muennighoff, 2022). The larger BLOOM-based model performs significantly worse, possibly because of multilinguality; on largely symmetric QuoraRetrieval, however, SGPT-5.8B-nli outperforms SGPT-5.8B-msmarco (Muennighoff, 2022).Source: §4.2 · p. 8

4.4 Multilinguality

• Multilingual MPNet performs best overall on classification and STS, outperforming slightly faster multilingual MiniLM on almost all languages, while LASER2 consistently trails other models.SGPT-BLOOM-7B1-msmarco achieves state-of-the-art performance on Hindi, Portuguese, Chinese, and French, languages extensively represented in pre-training, but is not much ahead of cheaper MPNet. It also performs competitively on Russian and Japanese, which unintentionally leaked into pre-training data (Muennighoff et al., 2022).Source: §4.4 · p. 9

5 Conclusion

• 8 text embedding tasks, with up to 15 datasets each and coverage of 112 languages, define MTEB’s benchmark scope.MTEB aims to provide reliable embedding performance estimates; its open-source release and leaderboard provide a foundation for further advances.Source: §5 · p. 9
• Close to 5,000 experiments on over 30 different models established baselines for future text embedding research.The authors describe this as the most comprehensive benchmarking of text embeddings to date.Source: §5 · p. 9
• No model achieved state-of-the-art performance on all tasks, and model performance varied strongly across tasks.Studies of scaling behavior, model efficiency, and multilinguality revealed model characteristics intended to inform research and industry decisions.Source: §5 · p. 9

A Datasets

• MTEB’s dataset inventory includes custom arXiv, bioRxiv, and medRxiv clustering datasets, with task statistics summarized in Table 2.S2S inputs use paper titles; P2P inputs concatenate titles and abstracts. Human-assigned categories supply labels: bioRxiv and medRxiv have unique, single-level categories, while arXiv uses the first assigned category.Source: §A.1 · p. 14, A Datasets · p. 14

A.1 Clustering

• Three arXiv split strategies evaluate coarse main-category clustering, fine-grained clustering within a main category, and fine-grained, multi-scale clustering across main categories.For every dataset, split, and strategy, label subsets and articles are sampled to vary cluster counts and sizes.Source: §A.1 · p. 14
• 199 subreddits supply titles for RedditClustering (Geigle et al., 2021), organized into 25 splits with 10–50 classes each.Each class contains 100–1,000 sentences.Source: §A.1 · p. 14
• 10 splits constitute RedditClusteringP2P, a custom MTEB dataset clustering concatenated titles and posts by subreddit, with 10 and 100 clusters per split.The passage specifies 1,000 to 100,000 posts.Source: §A.1 · p. 14
• 121 StackExchange sites supply titles for StackExchangeClustering (Geigle et al., 2021), organized into 25 splits with 10–50 classes each.Each class contains 100–1,000 sentences.Source: §A.1 · p. 14
• 10 splits constitute StackExchangeClusteringP2P, a custom MTEB dataset using concatenated titles and posts, with 10 to 100 clusters per split.Each split contains 5,000 to 10,000 posts.Source: §A.1 · p. 14
• 10 splits constitute TwentyNewsgroupsClustering, which clusters article titles into 20 newsgroup classes, with 1,000 to 10,000 titles per split.Every split contains all 20 classes.Source: §A.1 · p. 14

A.2 Classification

• Multilingual classification includes AmazonCounterfactual, AmazonReviews, and MassiveIntent/MassiveScenario (FitzGerald et al., 2022).AmazonCounterfactual covers 4 languages, AmazonReviews 6, and MassiveIntent/MassiveScenario 51 each. Labels distinguish counterfactual reviews, review ratings of 0–4 (1–5 stars), 60 intents, or 60 scenarios, respectively.Source: §A.2 · pp. 14–16
• Classification also evaluates movie-review polarity with Imdb, comment toxicity with ToxicConversations, and tweet sentiment with TweetSentimentExtraction.Imdb and ToxicConversations use binary labels; TweetSentimentExtraction distinguishes neutral, positive, and negative sentiment.Source: §A.2 · p. 16

A.3 Pair Classification

• Pair classification tests question duplication with SprintDuplicateQuestions (Shah et al., 2018) and tweet paraphrasing with TwitterSemEval2015 and TwitterURLCorpus (Lan et al., 2017).Each dataset assigns a binary label indicating whether the paired texts are duplicates or paraphrases.Source: §A.3 · p. 16

A.4 Bitext Mining

• 112 languages are represented in Tatoeba (Research), whose sentence-translation pairs support bitext mining alongside BUCC (Zweigenbaum et al., 2016, 2017, 2018).Tatoeba provides sets of 1000 sentences each; BUCC provides approximately 10–70k sentences each for English, French, Russian, German, and Chinese, with annotated translation pairs.Source: §A.4 · p. 16

A.5 Reranking

• Reranking covers similar-question matching, news recommendation, and title-based scientific-paper ranking.AskUbuntuDupQuestions supplies manually annotated similar/dissimilar questions, while StackOverflowDupQuestions targets Java, JavaScript, and Python questions. MindSmall ranks news titles, and SciDocsRR (Cohan et al., 2020b) ranks related papers using titles.Source: §A.5 · p. 16

A.6 Semantic Textual Similarity (STS)

• 11 language pairs in STS17 and 18 in STS22 extend semantic textual similarity evaluation to crosslingual sentence pairs.The STS benchmarks (Agirre et al., 2012, 2013) use scores from 0 to 5 and include image captions, news headlines, and forum text. STS12–STS16 and STSBenchmark are monolingual English benchmarks.Source: §A.6 · pp. 16–17

A.7 Summarization

• Summarization uses SummEval (Fabbri et al., 2020), pairing human annotations with summaries generated by models trained on CNN or DailyMail.The summaries come from recent summarization models.Source: §A.7 · p. 17

A.8 Retrieval

• Retrieval includes all publicly available datasets from BEIR (Thakur et al., 2021).These are ArguAna, ClimateFEVER, CQADupstack, DBPedia, FEVER, FiQA2018, HotpotQA, MSMARCO, NFCorpus, NQ, Quora, SCIDOCS, SciFact, Touche2020, and TRECCOVID.Source: §A.8 · p. 17

• MTEB aims to be a diverse benchmark.Source: B Limitations of MTEB · p. 17
• The authors acknowledge that MTEB has limitations.Source: B Limitations of MTEB · p. 17
• MTEB is intended to provide holistic performance reviews.Source: B Limitations of MTEB · p. 17

• A few hundred words is the text length of MTEB’s longest datasets, leaving very long documents unrepresented.Source: §1 · p. 17
• MTEB covers S2S, P2P, and S2P text-length settings.Source: §1 · p. 17
• Longer text sizes could be relevant for retrieval use cases.Source: §1 · p. 17

2. Task imbalance

• One dataset represents summarization; MTEB’s dataset-averaged scores favor better-represented tasks, notably retrieval, classification, and clustering.The authors hope to add datasets to underrepresented tasks, including summarization and pair classification.Source: §2 · p. 17

3. Multinguality

• MTEB includes multilingual classification, STS, and bitext mining, but English-only retrieval and clustering prevent comprehensive evaluation of SGPT-BLOOM-7B1-msmarco.Code datasets are also absent; proposed extensions include CodeSearchNet (Husain et al., 2019), TyDI QA (Clark et al., 2020), XOR QA (Asai et al., 2020), and MIRACL (Zhang et al., 2022).Source: §3 · p. 17

4. Additional modalities

• MTEB focuses solely on natural-language applications, leaving extensive benchmarking of text embeddings as inputs for other modalities to future work.Text embeddings can serve as downstream-model features, as in MTEB’s classification task; such applications can also involve image content.Source: §4 · p. 17

C Examples

• Tables 3–9 provide dataset examples across tasks, while retrieval examples are deferred to the BEIR paper (Thakur et al., 2021).Source: C Examples · p. 17

D Correlations

• Figure 6 presents correlation heatmaps for model performance and MTEB tasks.The supplied passage does not report specific correlation values or patterns.Source: D Correlations · p. 17

E Models

• Table 10 lists the publicly available model checkpoints used for MTEB evaluation.Source: E Models · p. 17

F Additional results

• Dataset-level results cover English evaluations and multilingual bitext mining, classification, and STS, with JSON outputs available on the Hugging Face Hub and results inspectable on the leaderboard.English results report each task’s main score. Multilingual bitext mining reports f1, classification reports accuracy, and STS reports Spearman correlations of cosine similarities.Source: F Additional results · p. 17, Table 11 · p. 21, Table 12 · p. 22
• Clustering and reranking performance correlate most strongly, while summarization and classification show weaker correlations with other tasks.The analysis uses Pearson correlations across model and task results.Source: Figure 6 · p. 20
• Size variants of the same model architecture show high correlations in their results.Figure 6 contrasts correlations among model variants with correlations across tasks.Source: Figure 6 · p. 20