PaperlayerView exact revision on arXiv

Source-linked AI summary

Multilingual E5 Text Embeddings: A Technical Report

Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, Furu Wei

arXiv:2402.05672v1cs.CLcs.IR

TL;DR

Existing embedding models are largely English-only, limiting their applicability in multilingual contexts. The report trains multilingual E5 models with contrastive pre-training and supervised fine-tuning, and evaluates them across English and multilingual tasks. The models show competitive multilingual performance, while the instruction-tuned variant surpasses strong English-only or task-specific comparators in reported evaluations.

  • Problem

    Most existing embedding models are trained exclusively on English text, limiting their applicability in multilingual contexts.

    Source: §1 · p. 1

  • Method

    The report trains multilingual E5 models with weakly supervised contrastive pre-training on multilingual text pairs, followed by supervised fine-tuning on labeled datasets.

    Source: §1 · p. 1, §2 · pp. 1–2

  • Results

    The best mE5 model surpasses Cohere multilingual-v3 by 0.4 points and BGE-large-en-v1.5 by 0.2 points on MTEB, while mE5-large-instruct surpasses LaBSE on bitext mining.

    Source: §3 · pp. 2–3

  • Takeaways & Limitations

    The publicly released models support information retrieval, semantic similarity, and clustering across a diverse range of languages.

    Source: §4 · p. 3

Abstract

from arXiv · show

This technical report presents the training methodology and evaluation results of the open-source multilingual E5 text embedding models, released in mid-2023. Three embedding models of different sizes (small / base / large) are provided, offering a balance between the inference efficiency and embedding quality. The training procedure adheres to the English E5 model recipe, involving contrastive pre-training on 1 billion multilingual text pairs, followed by fine-tuning on a combination of labeled datasets. Additionally, we introduce a new instruction-tuned embedding model, whose performance is on par with state-of-the-art, English-only models of similar sizes. Information regarding the model release can be found at https://github.com/microsoft/unilm/tree/master/e5 .

1 Introduction

The report introduces multilingual E5 embedding models that extend English E5 through multilingual training, alongside an instruction-tuned variant. It evaluates their English and multilingual retrieval and bitext-mining capabilities.

  • Multilingual E5 models extend English E5 with small, base, and large variants.Source: §1 · p. 1
  • The models follow two stages: weakly supervised contrastive pre-training, then supervised fine-tuning on high-quality labeled data.Source: §1 · p. 1
  • mE5-large-instruct uses synthetic data and task instructions to improve embedding quality by informing models about the task at hand.Source: §1 · p. 1
  • The evaluation covers English MTEB, multilingual MIRACL retrieval across 16 languages, and bitext mining in over 100 languages.Source: §1 · p. 1

2 Training Methodology

Training uses multilingual text pairs for weakly supervised contrastive pre-training, followed by supervised fine-tuning with labeled data and additional training signals. The released instruction-tuned model uses a broader synthetic instruction mixture.

Weakly-supervised Contrastive Pre-training

  • The first stage trains on a diverse multilingual text-pair mixture for approximately 30k steps with a 32k batch size, covering about 1 billion pairs.Source: §2 · p. 1
  • Pre-training uses the standard InfoNCE contrastive loss with only in-batch negatives.Source: §2 · p. 1

Supervised Fine-tuning

  • The second stage fine-tunes the models on high-quality labeled datasets using in-batch negatives, mined hard negatives, and cross-encoder knowledge distillation.Source: §2 · p. 2
  • mE5-large-instruct adds 500k GPT-3.5/4-generated synthetic examples, spanning 150k unique instructions and 93 languages.Source: §2 · p. 2

3 Experimental Results

The multilingual E5 models achieve strong performance across English evaluation, multilingual retrieval, and bitext mining. The instruction-tuned variant is particularly competitive against English-only and bitext-specific models.

English Text Embedding Benchmark

  • The best mE5 model surpasses Coheremultilingual-v3 by 0.4 points and BGElarge-en-v1.5 by 0.2 points on English MTEB.Source: §3 · p. 2

Multilingual Retrieval

  • mE5 models significantly outperform mDPR on both nDCG@10 and recall in MIRACL retrieval averaged across 16 languages.Source: §3 · p. 2, Table 4 · p. 2

Bitext Mining

  • mE5 models show competitive bitext-mining performance across both high-resource and low-resource languages.Source: §3 · pp. 2–3
  • mE5-large-instruct surpasses LaBSE on bitext mining, with expanded language coverage attributed to its synthetic data.Source: §3 · p. 3

4 Conclusion

The report introduces publicly available multilingual E5 embeddings trained with a multi-stage pipeline. These models support information retrieval, semantic similarity, and clustering across diverse languages.

  • The publicly released multilingual E5 models are trained with a multi-stage pipeline and support retrieval, semantic similarity, and clustering across diverse languages.Source: §4 · p. 3

A Implementation Details

The implementation constructs multilingual training data from diverse sources and combines supervised datasets with specialized evaluation resources. Training uses initialized multilingual encoder models and model-specific optimization settings.

Contrastive Pre-training Text Pairs

  • Contrastive pre-training pairs are constructed from multilingual sources including Wikipedia, mC4, CCNews, NLLB, Reddit, S2ORC, Stackexchange, and xP3.Source: A Implementation Details · p. 4

Data Mixture for Supervised Fine-tuning

  • Supervised fine-tuning combines datasets such as ELI5, HotpotQA, FEVER, MIRACL, MSMARCO, NQ, NLLB, NLI, SQuAD, TriviaQA, Quora, MrTyDi, and DuReader.Source: A Implementation Details · pp. 4–5

Training Hyperparameters

Related papers

Loading 2402.05672v1…