Source-linked AI summary
Multilingual E5 Text Embeddings: A Technical Report
Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, Furu Wei
TL;DR
Existing embedding models are largely English-only, limiting their applicability in multilingual contexts. The report trains multilingual E5 models with contrastive pre-training and supervised fine-tuning, and evaluates them across English and multilingual tasks. The models show competitive multilingual performance, while the instruction-tuned variant surpasses strong English-only or task-specific comparators in reported evaluations.
Problem
Most existing embedding models are trained exclusively on English text, limiting their applicability in multilingual contexts.
Method
The report trains multilingual E5 models with weakly supervised contrastive pre-training on multilingual text pairs, followed by supervised fine-tuning on labeled datasets.
Results
The best mE5 model surpasses Cohere multilingual-v3 by 0.4 points and BGE-large-en-v1.5 by 0.2 points on MTEB, while mE5-large-instruct surpasses LaBSE on bitext mining.
Takeaways & Limitations
The publicly released models support information retrieval, semantic similarity, and clustering across a diverse range of languages.
Abstract
from arXiv · showhide
This technical report presents the training methodology and evaluation results of the open-source multilingual E5 text embedding models, released in mid-2023. Three embedding models of different sizes (small / base / large) are provided, offering a balance between the inference efficiency and embedding quality. The training procedure adheres to the English E5 model recipe, involving contrastive pre-training on 1 billion multilingual text pairs, followed by fine-tuning on a combination of labeled datasets. Additionally, we introduce a new instruction-tuned embedding model, whose performance is on par with state-of-the-art, English-only models of similar sizes. Information regarding the model release can be found at https://github.com/microsoft/unilm/tree/master/e5 .
1 Introduction
The report introduces multilingual E5 embedding models that extend English E5 through multilingual training, alongside an instruction-tuned variant. It evaluates their English and multilingual retrieval and bitext-mining capabilities.
- Multilingual E5 models extend English E5 with small, base, and large variants.
- The models follow two stages: weakly supervised contrastive pre-training, then supervised fine-tuning on high-quality labeled data.
- mE5-large-instruct uses synthetic data and task instructions to improve embedding quality by informing models about the task at hand.
- The evaluation covers English MTEB, multilingual MIRACL retrieval across 16 languages, and bitext mining in over 100 languages.
2 Training Methodology
Training uses multilingual text pairs for weakly supervised contrastive pre-training, followed by supervised fine-tuning with labeled data and additional training signals. The released instruction-tuned model uses a broader synthetic instruction mixture.
- Weakly-supervised Contrastive Pre-training: The first stage trains on a diverse multilingual text-pair mixture for approximately 30k steps with a 32k batch size, covering about 1 billion pairs.
- Weakly-supervised Contrastive Pre-training: Pre-training uses the standard InfoNCE contrastive loss with only in-batch negatives.
- Supervised Fine-tuning: The second stage fine-tunes the models on high-quality labeled datasets using in-batch negatives, mined hard negatives, and cross-encoder knowledge distillation.
- Supervised Fine-tuning: mE5-large-instruct adds 500k GPT-3.5/4-generated synthetic examples, spanning 150k unique instructions and 93 languages.
3 Experimental Results
The multilingual E5 models achieve strong performance across English evaluation, multilingual retrieval, and bitext mining. The instruction-tuned variant is particularly competitive against English-only and bitext-specific models.
- English Text Embedding Benchmark: The best mE5 model surpasses Coheremultilingual-v3 by 0.4 points and BGElarge-en-v1.5 by 0.2 points on English MTEB.
- Multilingual Retrieval: mE5 models significantly outperform mDPR on both nDCG@10 and recall in MIRACL retrieval averaged across 16 languages.
- Bitext Mining: mE5 models show competitive bitext-mining performance across both high-resource and low-resource languages.
- Bitext Mining: mE5-large-instruct surpasses LaBSE on bitext mining, with expanded language coverage attributed to its synthetic data.
4 Conclusion
The report introduces publicly available multilingual E5 embeddings trained with a multi-stage pipeline. These models support information retrieval, semantic similarity, and clustering across diverse languages.
- The publicly released multilingual E5 models are trained with a multi-stage pipeline and support retrieval, semantic similarity, and clustering across diverse languages.
A Implementation Details
The implementation constructs multilingual training data from diverse sources and combines supervised datasets with specialized evaluation resources. Training uses initialized multilingual encoder models and model-specific optimization settings.
- Contrastive Pre-training Text Pairs: Contrastive pre-training pairs are constructed from multilingual sources including Wikipedia, mC4, CCNews, NLLB, Reddit, S2ORC, Stackexchange, and xP3.
- Data Mixture for Supervised Fine-tuning: Supervised fine-tuning combines datasets such as ELI5, HotpotQA, FEVER, MIRACL, MSMARCO, NQ, NLLB, NLI, SQuAD, TriviaQA, Quora, MrTyDi, and DuReader.
- Training Hyperparameters: The mE5-large-instruct model uses the Wang et al. data mixture and the same hyperparameters as mE5-large.
- Training Hyperparameters: The mE5-small, mE5-base, and mE5-large models are initialized from multilingual MiniLM, XLM-R-base, and XLM-R-large, respectively.