M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation

Jianlv Chen; Shitao Xiao; Peitian Zhang; Kun Luo; Defu Lian; Zheng Liu

Source-linked AI summary

M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation

Jianlv Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, Zheng Liu

arXiv:2402.03216v5Feb 5, 2024cs.CLcs.AIcs.LG

TL;DR

Text embeddings remain limited in language coverage, unified retrieval functionality, and long-document support. M3-Embedding addresses these gaps with multilingual, multi-functional, and multi-granular embeddings trained using self-knowledge distillation and efficient batching, achieving superior results across multilingual, cross-lingual, and long-document retrieval tasks. Its generalizability across diverse datasets and real-world scenarios, and its performance on extremely long documents beyond 8,192 tokens, remain to be investigated.

Problem
Existing embedding models have limited multilingual coverage, usually target one retrieval functionality, and often support only short inputs because long-document training is costly.
Method
M3-Embedding combines multilingual, dense, sparse, multi-vector, and long-input modeling with self-knowledge distillation, efficient batching, and curated training data.
Results
M3-Embedding achieves superior performance on multilingual, cross-lingual, and multilingual long-document retrieval tasks, including state-of-the-art results on MIRACL and MKQA.
Takeaways & Limitations
M3-Embedding provides one embedding model supporting more than 100 languages, three retrieval functionalities, and inputs ranging from short text to 8,192-token documents.
Takeaways & Limitations
Generalizability to diverse datasets and real-world scenarios, as well as performance on extremely long documents beyond 8,192 tokens, requires further investigation.

Abstract

from arXiv · show

In this paper, we introduce a new embedding model called M3-Embedding, which is distinguished for its versatility in \textit{Multi-Linguality}, \textit{Multi-Functionality}, and \textit{Multi-Granularity}. It provides a uniform support for the semantic retrieval of more than 100 working languages. It can simultaneously accomplish the three common retrieval functionalities: dense retrieval, multi-vector retrieval, and sparse retrieval. Besides, it is also capable of processing inputs of different granularities, spanning from short sentences to long documents of up to 8,192 tokens. The effective training of M3-Embedding presents a series of technical contributions. Notably, we propose a novel self-knowledge distillation approach, where the relevance scores from different retrieval functionalities can be integrated as the teacher signal to enhance the training quality. We also optimize the batching strategy, which enables a large batch size and high training throughput to improve the discriminativeness of embeddings. M3-Embedding exhibits a superior performance in our experiment, leading to new state-of-the-art results on multilingual, cross-lingual, and long-document retrieval benchmarks.

1 Introduction

M3-Embedding addresses limitations in language coverage, retrieval functionality, and input length by unifying multilingual, dense, sparse, multi-vector, and long-document retrieval capabilities. Its training combines self-knowledge distillation with efficient batching, and experiments report strong multilingual, cross-lingual, and long-document retrieval performance.

Motivation: Existing embedding models are limited by English-centric coverage, single retrieval functionality, and short-input support caused by costly long-document training.These limitations restrict versatility across languages, retrieval workflows, and document lengths.
M3-Embedding: M3-Embedding supports more than 100 languages, multilingual and cross-lingual retrieval, three retrieval functionalities, and inputs up to 8,192 tokens.The three functionalities are dense, sparse, and multi-vector retrieval.
Training: Self-knowledge distillation integrates relevance scores from dense, sparse, and multi-vector retrieval as a teacher signal for joint training.The [CLS] embedding serves dense retrieval, while other token embeddings support sparse and multi-vector retrieval.
Results: Experiments report superior retrieval quality across languages, retrieval functionalities, and input granularities, including state-of-the-art multilingual and cross-lingual benchmark performance.The model also maintains strong capability across input lengths within 8,192 tokens.

2 Related Work

Related work establishes the progress of text embeddings through pretrained encoders, contrastive learning, negative sampling, and knowledge distillation. It also identifies continuing gaps in unified retrieval functionality and balanced multilingual performance.

General Text Embeddings: Pretrained language models, contrastive learning, improved negative sampling, and knowledge distillation have driven recent progress in text embeddings.The related work cites versatile embedding methods including Contriever, LLM-Embedder, E5, BGE, SGPT, and Open Text Embedding.
Neural Retrieval: Neural retrieval uses embedding-based semantic relationships to retrieve answers relevant to an input query.Dense retrieval aggregates encoder outputs, while multi-vector retrieval uses fine-grained interactions and sparse retrieval uses term weights.
Neural Retrieval: Existing retrieval methods typically use different embedding models, and no existing method was known to unify dense, multi-vector, and sparse functionality.This motivates models that support multiple retrieval mechanisms within one embedding system.
Multilingual Embeddings: Most existing text embeddings were developed primarily for English, while multilingual systems still show gaps relative to English models and imbalance across languages.Related efforts include multilingual encoders, multilingual datasets, and community-developed multilingual embeddings.

3 M3-Embedding

M3-Embedding unifies dense, lexical, and multi-vector retrieval while supporting multilingual and cross-lingual inputs across different granularities. Its training combines multi-stage learning, self-knowledge distillation, diverse data, and efficient batching.

Hybrid Retrieval: M3-Embedding supports dense, lexical, and multi-vector retrieval within one embedding model, enabling hybrid retrieval and score-based reranking.Candidate results can be retrieved individually, then reranked using an integrated relevance score.
Hybrid Retrieval: Dense retrieval uses normalized [CLS] embeddings and inner-product similarity, while lexical retrieval estimates token importance and multi-vector retrieval uses full projected token embeddings.Lexical and multi-vector methods provide token-level or fine-grained representations beyond a single pooled embedding.
Hybrid Retrieval: The model combines retrieval-function scores through a weighted sum, with weights w1, w2, and w3 chosen according to the downstream scenario.The integrated score supports hybrid reranking across dense, lexical, and multi-vector retrieval.
Self-Knowledge Distillation: Self-knowledge distillation uses the integrated relevance score as a teacher while jointly optimizing the individual retrieval losses.This addresses potentially conflicting objectives among the retrieval functions by transferring soft relevance information across them.
Self-Knowledge Distillation: Training proceeds in stages: unsupervised pretraining first trains dense retrieval, followed by fine-tuning that establishes all three retrieval functionalities.The text encoder is initialized from an XLM-RoBERTa model adapted with RetroMAE.
Efficient Batching: Efficient batching groups examples by sequence length and samples similar-length instances together, reducing padding while supporting large batches for short and long inputs.Fixed sampling seeds balance cross-GPU loads, and the strategy is designed for the model’s need to learn across input granularities.

4 Experiment

The experiments evaluate M3-Embedding across multilingual, cross-lingual, long-document, and ablation settings. Results show that combining retrieval functionalities improves performance, while self-knowledge distillation and multi-stage training strengthen retrieval quality.

Evaluation Settings: M3-Embedding is evaluated on multilingual retrieval, cross-lingual retrieval, multilingual long-document retrieval, NarrativeQA, and technical ablations.MIRACL uses 18 languages and nDCG@10; MKQA uses 25 non-English query languages and Recall@100; MLDR and NarrativeQA evaluate long-document retrieval.
4.1 Multi-Lingual Retrieval: The All configuration combining dense, sparse, and multi-vector retrieval achieves the best multilingual retrieval performance.Multi-vector retrieval adds improvement through fine-grained embedding interactions, while Dense+Sparse further improves over either individual method.
4.2 Cross-Lingual Retrieval: M3-Embedding’s dense method outperforms other baselines in cross-lingual retrieval, while combining retrieval methods yields the best empirical performance.Performance remains relatively stable across languages, including low-resource languages, whereas sparse retrieval is weaker because query and passage languages share few terms.
4.3 Multilingual Long-Doc Retrieval: 65.0 average performance is achieved by combining retrieval methods on MLDR, while sparse retrieval improves by about 10 points over dense retrieval.Multi-vector retrieval brings 5.1+ points improvement over M3 (Dense), and the combined method performs best.
4.3 Multilingual Long-Doc Retrieval: As sequence length grows, M3-Embedding expands its advantage over baseline methods on NarrativeQA.The result reflects the model’s capability to handle long inputs.
4.4 Ablation Study: Self-knowledge distillation improves Dense, Sparse, and Multi-vec performance on MIRACL, with a more pronounced impact on sparse retrieval.Multi-stage training also helps: RetroMAE improves retrieval performance, and unsupervised pre-training further enhances embedding quality.

5 Conclusion

M3-Embedding advances text-embedding versatility by supporting multilingual retrieval, diverse input granularities, and unified retrieval functionalities. Its self-knowledge distillation, efficient batching, and curated data are empirically associated with superior multilingual, cross-lingual, and multilingual long-document retrieval performance.

5 Conclusion: M3-Embedding unifies multilingual retrieval, diverse input granularities, and dense, sparse, and multi-vector retrieval functionalities.The paper reports superior performance on multilingual, cross-lingual, and multilingual long-document retrieval tasks.
5 Conclusion: The paper contributes self-knowledge distillation, efficient batching, and high-quality data curation for training M3-Embedding.
5 Conclusion: M3-Embedding achieves superior performance across multilingual, cross-lingual, and multilingual long-document retrieval tasks.

Limitations

The paper identifies open questions about generalization, extremely long documents, and performance variation across the more than 100 supported languages.

Limitations: Generalizability to diverse datasets and real-world scenarios requires further investigation.
Limitations: Processing documents beyond 8,192 tokens may challenge computational resources and model efficiency, and remains insufficiently evaluated.
Limitations: Performance variation across the more than 100 supported languages is not thoroughly discussed and needs broader evaluation.

Ethics Consideration

M3-Embedding is presented as a versatile multilingual embedding model, but its performance may vary across languages because multilingual training data are unevenly distributed.

M3-Embedding is designed for multilinguality, multi-functionality, and multi-granularity.
The model uses multilingual training data covering different kinds of languages.
Uneven training-data distributions across languages may cause performance to vary between languages.

A.1 Collected Data

The collected-data section describes the distribution of unsupervised data, a segment-shuffling strategy for long texts, and GPT-3.5-generated questions for dataset construction.

Figure 4 illustrates the language and token-length distribution of the unsupervised data.
Long texts are divided into three segments whose order is randomly shuffled before recombination.This prevents the model from relying solely on summarizing statements at the beginning of texts.
GPT-3.5 is prompted to generate one specific question centered on each text’s core content without pronouns or additional material.The generated-dataset details are reported in Table 7.

B.1 Experimental Hyperparameters

M3-Embedding is pretrained and fine-tuned with long-context multilingual data, staged optimization, and a multiple-CLS method for long-text representation when direct long-text fine-tuning is constrained.

XLM-RoBERTa is extended to 8,192 positions and updated with RetroMAE using Pile, Wudao, and mC4 data.The sampled pretraining corpus contains 184 million text samples across 105 languages.
Pretraining uses query and passage maximum lengths of 512 and 8,192 tokens, respectively.The learning rate is 5 × 10^-5, with a 0.1 warmup ratio and 0.01 weight decay.
The second training stage runs for 25,000 steps with sequence-length-specific batch sizes on 96 A800 GPUs.
Fine-tuning samples seven negatives per query, warms up dense, sparse, and multi-vector embeddings, then applies unified self-knowledge distillation.These experiments use 24 A800 GPUs.
MCLS inserts a [CLS] token every 256 tokens so multiple CLS tokens jointly capture long-text semantics without long-text fine-tuning.The method targets settings constrained by limited long-text data or computation resources.

B.3 Split-batch Method

The split-batch strategy computes embeddings over smaller sub-batches while retaining the effective large-batch output, substantially increasing feasible batch sizes for long inputs.

B.3 Split-batch Method: Split-batch partitions a current batch into smaller sub-batches and concatenates their generated embeddings.The pseudocode enables gradient checkpointing and collects only the sub-batch embeddings.
B.3 Split-batch Method: Gradient checkpointing discards intermediate forward-pass activations during sub-batch processing.Without it, activations accumulate and consume GPU memory comparable to traditional batching.
B.3 Split-batch Method: At sequence length 8,192, split-batch increases batch size by more than 20 times.The batch-size gain becomes more pronounced as text length increases.
Evaluation: M3-Embedding outperforms all baselines on average on the MIRACL and MKQA benchmarks.
Evaluation: Ablation studies examine self-knowledge distillation and multi-stage training on the MIRACL development set.

C.2 Different Tokenizer for BM25

Tokenizer choice materially affects BM25 effectiveness and document representation, while M3 consistently outperforms BM25 with matched tokenization. M3’s sparse retrieval exceeds BM25 on MIRACL and MKQA but remains competitive rather than superior on MLDR.

Lucene9’s Analyzer improves BM25 effectiveness compared with directly using the XLM-RoBERTa tokenizer.The Analyzer typically performs tokenization, stemming, and stopword removal.
37% more unique terms per MLDR article are produced by Lucene’s Analyzer than by the XLM-RoBERTa tokenizer.The reported counts are 1,451 unique terms for Lucene and 1,056 for XLM-RoBERTa, increasing retrieval latency.
M3 outperforms BM25 models using the same tokenizer across all datasets.This indicates that M3 learns better weights than those calculated by BM25.
M3’s sparse retrieval outperforms BM25 on MIRACL and MKQA but does not surpass BM25 on MLDR.On long-document retrieval, M3’s sparse retrieval remains competitive, and tokenizer improvements are identified as future work.