Source-linked AI summary
Text Embeddings by Weakly-Supervised Contrastive Pre-training
Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, Furu Wei
TL;DR
General-purpose single-vector embeddings must transfer across tasks while reducing reliance on limited labeled or noisy synthetic supervision. E5 contrastively pre-trains on curated CCPairs and optionally fine-tunes on labeled data, achieving strong zero-shot and fine-tuned benchmark results, including performance competitive with much larger models.
Problem
Existing embedding approaches rely on task-specific labeled datasets or automatically constructed text pairs, limiting general-purpose text representation learning.
Method
E5 uses contrastive pre-training with in-batch negatives on heterogeneous CCPairs text pairs, followed by optional supervised fine-tuning.
Results
E5 provides strong performance across retrieval and other embedding tasks, including BEIR gains over BM25 without labeled data and MTEB performance competitive with models having 40× more parameters.
Takeaways & Limitations
E5 serves as an off-the-shelf general-purpose embedding model that can also be customized for downstream tasks through fine-tuning.
Takeaways & Limitations
BM25 still has advantages for simplicity, efficiency, and interpretability, and dense retrievers need further improvement for long-tail, long-document, and exact-match retrieval.
Abstract
from arXiv · showhide
This paper presents E5, a family of state-of-the-art text embeddings that transfer well to a wide range of tasks. The model is trained in a contrastive manner with weak supervision signals from our curated large-scale text pair dataset (called CCPairs). E5 can be readily used as a general-purpose embedding model for any tasks requiring a single-vector representation of texts such as retrieval, clustering, and classification, achieving strong performance in both zero-shot and fine-tuned settings. We conduct extensive evaluations on 56 datasets from the BEIR and MTEB benchmarks. For zero-shot settings, E5 is the first model that outperforms the strong BM25 baseline on the BEIR retrieval benchmark without using any labeled data. When fine-tuned, E5 obtains the best results on the MTEB benchmark, beating existing embedding models with 40x more parameters.
1 Introduction
E5 is introduced as a general-purpose single-vector text embedding model trained contrastively on curated weakly supervised text pairs. It targets retrieval, matching, and other downstream tasks in zero-shot and fine-tuned settings.
- E5 provides general-purpose single-vector text embeddings for tasks including retrieval and text matching.
- E5 supports both zero-shot use and further supervised fine-tuning for downstream applications.
- The model is trained contrastively using CCPairs, a curated web-scale dataset assembled from heterogeneous text-pair sources.
- E5 outperforms BM25 on BEIR retrieval without labeled data and remains competitive with models having 40× more parameters on MTEB.
2 Related Work
Prior work develops text embeddings through classical methods, pretrained language models, supervised fine-tuning, and self-supervised text-pair construction. E5 builds on this literature by showing strong embeddings from self-supervised pre-training alone and improved fine-tuned performance with less labeled data.
- Classical embedding methods include LSA, LDA, and weighted averages of word vectors.
- Pretrained language models are commonly fine-tuned to produce continuous sentence or text embeddings for semantic and retrieval tasks.
- Self-supervised approaches construct text pairs through tasks such as inverse cloze prediction, cropping, neighboring-text positives, or domain-matched pre-training.
- Embedding evaluation commonly uses downstream benchmarks, including BEIR for zero-shot retrieval and MTEB across 56 datasets, 8 tasks, and 112 languages.
- E5 shows that high-quality embeddings can be trained with self-supervised pre-training alone and can outperform existing models after fine-tuning on less labeled data.
3 CCPairs: A Large Collection of Text Pair Dataset
CCPairs is a large text-pair dataset curated from diverse semi-structured web sources and filtered to improve pair quality. Its design aims to provide heterogeneous training signals for general-purpose embeddings.
- CCPairs combines diverse web-derived text pairs that provide heterogeneous training signals for general-purpose embeddings.
- The dataset includes pairs from Reddit, Stackexchange, Wikipedia, scientific papers, Common Crawl, and news sources.
- CCPairs contains approximately 1.3 billion automatically mined text pairs after preliminary heuristic filtering.
- A consistency-based filter retains pairs ranked among the top k=2 against one million random passages.
4 Method
E5 uses contrastive pre-training to learn fixed-size embeddings from text pairs, then optionally fine-tunes them with labeled data. The resulting embeddings support retrieval, classification, similarity, and clustering workflows.
- 4.1 Contrastive Pre-training with Unlabeled Data: Contrastive pre-training distinguishes relevant text pairs from irrelevant or negative pairs using an InfoNCE loss.
- 4.1 Contrastive Pre-training with Unlabeled Data: The model scores query-passage pairs with temperature-scaled cosine similarity between average-pooled Transformer embeddings.
- 4.1 Contrastive Pre-training with Unlabeled Data: In-batch passages from other pairs serve as negatives, enabling stable training when batch sizes are sufficiently large.
- 4.2 Fine-tuning with Labeled Data: Supervised fine-tuning uses NLI, MS-MARCO, and NQ data to inject human knowledge and improve embedding quality.
- 4.2 Fine-tuning with Labeled Data: Hard negatives and cross-encoder knowledge distillation are used for MS-MARCO and NQ fine-tuning.
- 4.3 Applications to Text Embedding Tasks: The embeddings support zero-shot retrieval, few-shot classification, zero-shot classification, semantic similarity, and clustering.
5 Experiments
Experiments evaluate E5 across BEIR retrieval and MTEB embedding tasks, with unsupervised pre-training, supervised transfer, and analyses of training choices. E5 shows strong retrieval and broad embedding performance, while dataset curation, batch size, and domain coverage affect outcomes.
- Unsupervised BEIR: E5-PTbase outperforms BM25 by 1.2 points averaged over 15 BEIR datasets without labeled data.Scaling from E5-PTbase to E5-PTlarge further improves the average from 42.9 to 44.2.
- Supervised BEIR: E5base reaches 48.7 average nDCG@10 on BEIR after supervised fine-tuning, surpassing larger existing methods such as GTRlarge.Most datasets benefit, but FiQA, Scidocs, and Fever are exceptions, likely because the fine-tuning data lack sufficient domain diversity.
- MTEB: E5 models substantially outperform similarly sized models on MTEB and match much larger models, with E5large using 300M versus 4.8B parameters.
- MTEB: Contrastive pre-training on CCPairs contributes to the performance gap between E5base and BERT-FTbase, while supervised fine-tuning improves most MTEB categories except Clustering.The paper states that obtaining state-of-the-art embeddings purely self-supervised remains open.
- Analysis: Increasing batch size from 1K to 32K yields consistent gains across all six analyzed datasets.Hard negatives can enable smaller batches, but mining them for datasets larger than 100M is non-trivial.
- Analysis: Combining MS-MARCO, NQ, and NLI fine-tuning data produces the best overall MTEB scores, reflecting the value of dataset diversity.MS-MARCO plus NQ benefits retrieval, whereas NLI benefits STS and linear-probing classification.
- Analysis: Filtered data gives nearly a 6-point advantage when training on 1M pairs, while using four times more unfiltered data remains 1.6 points behind.
6 Conclusion
E5 combines weakly supervised contrastive pre-training with a simple negative-sampling design to produce strong general-purpose embeddings. It performs well across single-vector tasks and improves with downstream customization.
- E5 uses contrastive pre-training and compares different negative-sampling strategies as part of its embedding framework.
- E5 provides strong performance across retrieval, semantic textual similarity, and text-matching tasks requiring single-vector representations.
A Dataset Details
The dataset details describe the web and benchmark sources used to assemble and evaluate E5’s training data. They include Common Crawl, Reddit, S2ORC, and 15 publicly downloadable BEIR datasets.
- Common Crawl processing uses the 2022-33 snapshot and cc_net for language identification, deduplication, and language-model filtering.
- Reddit data spans 2018 through August 2022, while S2ORC receives a 0.3 sample weight to reduce scientific-domain overfitting.
- The BEIR evaluation uses 15 publicly downloadable datasets covering retrieval settings such as web search, question answering, and fact verification.
- The filtered data sources include an “Others” category containing SimpleWiki, GooAQ, WikiHow, and Yahoo Answers.
B Implementation Details
The implementation uses fixed model configurations, retrieval prefixes, and length constraints across pre-training and evaluation. It also distinguishes title-inclusive in-domain evaluation from title-free BEIR evaluation.
- Position embeddings are frozen and the maximum evaluation text length is set to 512 because some datasets contain long texts.
- Retrieval inputs use “query: ” and “passage: ” prefixes, while Quora questions receive the “query: ” prefix.
- MS-MARCO in-domain results include RocketQA document titles, whereas BEIR MS-MARCO lacks titles and is therefore expected to score lower.
- The paper reports model configurations and contrastive pre-training and fine-tuning hyperparameters in Tables 10 and 11.
- In-domain evaluation reports MRR@10 and Recall@1k for MS-MARCO, and Recall@20 and Recall@100 for NQ.
C Negative Results
Several attempted training modifications produced limited or unfavorable outcomes. The experiments also document practical costs associated with hard-negative mining and auxiliary objectives.
- Adding BM25 hard negatives improves BEIR results by ∼0.5 points with 15M data but becomes too time-consuming on datasets larger than 250M.
- RoBERTa initialization performs worse than BERT initialization on most BEIR benchmark datasets despite RoBERTa’s broader gains elsewhere.
- Adding masked-language-modeling loss to 25% of training pairs produces results on par with removing it while increasing training cost.
- The MTEB results are presented per dataset, with Retrieval omitted because those datasets are the same as BEIR.