Source-linked AI summary
mGTE: Generalized Long-Context Text Representation and Reranking Models for Multilingual Text Retrieval
Xin Zhang, Yanzhao Zhang, Dingkun Long, Wen Xie, Ziqi Dai, Jialong Tang, Huan Lin, Baosong Yang, Pengjun Xie, Fei Huang, Meishan Zhang, Wenjie Li, Min Zhang
TL;DR
Long and multilingual retrieval requires models that combine broad text coverage with practical computational cost. The paper builds an 8k-context multilingual encoder, then trains a hybrid TRM and cross-encoder reranker from it using contrastive learning. The encoder surpasses same-sized XLM-R, while the retrieval models match larger BGE-M3 models and perform better on long-context benchmarks.
Problem
Long and multilingual retrieval applications need upgraded models because conventional encoders cannot adequately process these texts, while multilingual LLMs may be computationally expensive for self-hosted search.
Method
The paper trains an 8,192-token multilingual encoder with RoPE and unpadding, then constructs a hybrid dense-and-sparse TRM and cross-encoder reranker through contrastive learning.
Results
The encoder outperforms same-sized XLM-R, while the TRM and reranker match larger BGE-M3 models and achieve better performance on long-context retrieval datasets.
Takeaways & Limitations
The resulting models are presented as efficient for industrial applications while retaining strong multilingual and long-context retrieval performance.
Takeaways & Limitations
Continuing training with RetroMAE after MLM-8192 did not improve performance in early English-model experiments, and contrastive pre-training did not improve reranker performance.
Abstract
from arXiv · showhide
We present systematic efforts in building long-context multilingual text representation model (TRM) and reranker from scratch for text retrieval. We first introduce a text encoder (base size) enhanced with RoPE and unpadding, pre-trained in a native 8192-token context (longer than 512 of previous multilingual encoders). Then we construct a hybrid TRM and a cross-encoder reranker by contrastive learning. Evaluations show that our text encoder outperforms the same-sized previous state-of-the-art XLM-R. Meanwhile, our TRM and reranker match the performance of large-sized state-of-the-art BGE-M3 models and achieve better results on long-context retrieval benchmarks. Further analysis demonstrate that our proposed models exhibit higher efficiency during both training and inference. We believe their efficiency and effectiveness could benefit various researches and industrial applications.
1 Introduction
The paper addresses the need for efficient retrieval models that handle long and multilingual texts by building a holistic pipeline from an 8k-context multilingual encoder to retrieval and reranking models. Its models target stronger performance than prior same-sized encoders while remaining suitable for practical retrieval settings.
- Motivation: Long and multilingual retrieval applications require upgraded models because conventional encoder-based models cannot adequately address these texts.Existing approaches extend multilingual encoders or use multilingual LLMs, the latter potentially being computationally expensive for self-hosted search services.
- Approach: The paper builds an 8,192-token multilingual encoder from scratch with RoPE and unpadding, using a two-stage MLM curriculum.The encoder is the first stage of the proposed training pipeline.
- Approach: Based on this encoder, the authors construct a hybrid TRM producing dense and sparse vectors and a cross-encoder reranker through contrastive learning.The TRM supports efficient first-stage retrieval, while the reranker refines candidate ordering.
- Results: The encoder outperforms the same-sized previous state-of-the-art XLM-R on XTREME-R and GLUE benchmarks.These evaluations assess the encoder on two natural language understanding benchmarks.
- Results: The TRM and reranker match state-of-the-art BGE-M3 on multilingual and long-context retrieval benchmarks and perform better on long-context settings with a smaller size.The reported benchmarks include MIRACL and MLDR.
2 Method
The method builds an 8k-context multilingual encoder from scratch, then derives a hybrid text retriever and cross-encoder reranker through contrastive learning.
- Text Encoder: The encoder starts from BERT, replaces absolute positional embeddings with RoPE, upgrades the FFN to GLU, and removes attention-score dropout.Token embeddings are padded to a multiple of 64 for throughput, while unpadding reduces computation from padding tokens.
- Text Encoder: The pre-training data combines multiple corpora, covers 75 languages, and uses multinomial language sampling with α = 0.5 to increase low-resource language exposure.The curriculum uses MLM-2048 followed by MLM-8192, with RoPE bases of 10,000 and 160,000 respectively.
- Text Representation Model: The TRM uses contrastive pre-training and fine-tuning with InfoNCE, using dense [CLS] representations, naturally occurring and multilingual text pairs, and hard-negative datasets.The pre-training uses 512-token queries and 1024-token documents, reverses the RoPE base to 20,000, and trains with in-batch negatives.
- Text Representation Model: TRM fine-tuning jointly learns Matryoshka embeddings and neural sparse representations, with weighted losses over embedding dimensions and token-level weights.The resulting model supports elastic dense embeddings and sparse retrieval representations.
- Text Reranking Model: The reranker is a cross-encoder that jointly reads [CLS] q [SEP] d and predicts relevance from the [CLS] output, fine-tuned with InfoNCE.It is initialized from the pre-trained 8k-context encoder and uses the TRM fine-tuning data and settings with adjusted hard negatives.
3 Evaluation
The evaluation covers multilingual, crosslingual, English, and long-context retrieval alongside NLU and embedding benchmarks. Across these settings, the encoder improves over same-sized XLM-R, while the TRM and reranker remain competitive with larger systems and show efficiency gains.
- Natural Language Understanding: The encoder outperforms the same-sized XLM-R on all XTREME-R and GLUE benchmarks.
- Natural Language Understanding: 3.22 and 2.42 points are the average-score gains of the 2048- and 8192-context encoders over XLM-R, respectively.
- Text Embedding: Our final TRM achieves the best scores on Chinese and French MTEB subsets and is competitive on English against BGE-M3 and mE5.
- Text Retrieval: On MIRACL, MKQA, and BEIR, the models are close to state-of-the-art large models, while on MLDR and LoCo they achieve better scores.
- First-Stage Retrieval: The TRM consistently outperforms mE5 and OpenAI APIs, exceeds BGE-M3 on MLDR, and remains close to BGE-M3 elsewhere.
- Reranking: The reranker outperforms the smaller bge-reranker-v2-m3 and greatly surpasses the same-sized jina-reranker-v2-multilingual.
- Efficiency: 14 times faster is the TRM's maximum speed advantage over BGE-M3, with encoding taking 52 seconds versus 744 seconds.
- Analysis: Reversed NTK scaling slightly lowers 1k-context performance but produces more stable 8k performance across training steps.
4 Related Work
Prior work established long-context retrieval with English encoders and extended multilingual models. This paper instead pre-trains native 8k multilingual models from scratch to target long-context performance and efficiency.
- English studies found that training long-context encoders from scratch can improve text retrieval.
- Earlier multilingual work extended XLM-RoBERTa-large to 8192 positions through continued training, whereas this paper pre-trains native 8k models from scratch.
5 Conclusion
The paper presents a native 8192-context multilingual retrieval pipeline spanning an encoder, hybrid TRM, and cross-encoder reranker. Its models outperform same-sized XLM-R, improve long-context retrieval, and are positioned as efficient retrieval systems.
- The work presents native 8192-context multilingual retrieval models built through a holistic practice.
- The encoder combines RoPE and unpadding with a two-stage MLM curriculum for 8k context and outperforms same-sized XLM-RoBERTa on NLU benchmarks.
- The hybrid TRM and cross-encoder reranker are constructed by contrastive learning, while the TRM uses reversed RoPE NTK scaling and produces Matryoshka and sparse representations.
- The models are close to larger systems on regular retrieval datasets and achieve better performance on long-context datasets.
- Pre-training Data: The encoder's MLM pre-training data are curated from multiple sources and cover 75 languages with 1,028B tokens.
A.2 Training Details
The text encoder is pretrained with a two-stage masked-language-modeling curriculum that extends the native context length to 8192 tokens. RoPE replaces absolute position embeddings to support context extension and bidirectional encoder modeling.
- Training Curriculum: A two-stage MLM curriculum trains first at maximum length 2048, then continues at maximum length 8192 after downsampling shorter texts.The first stage uses 250k steps, while the second uses 30k steps.
- Model Configuration: The base encoder has 12 layers with hidden size 768 and uses RoPE bases of 10,000 and 160,000 in the two stages.
- Training Cost: Pretraining took 10.75 days for MLM-2048 and 20.5 hours for MLM-8192 on 32 A100 80G GPUs.The authors state that this setting is not optimal and recommend further optimization.
- RoPE: RoPE replaces absolute position embeddings and supports training on shorter contexts followed by inference on longer contexts.It also provides asymmetric relative distance encoding for bidirectional encoder-only models.
B.1 Pre-Training Data
The paper combines multilingual corpora and large-scale weakly correlated text pairs for encoder and retrieval-model pretraining, then uses high-quality datasets for fine-tuning. The multilingual MLM corpus covers 75 languages and contains 1,028B tokens.
- MLM Pre-Training Data: The MLM pre-training data contain 1,028B tokens stored in 4.47 TiB of Arrow files.
- Contrastive Pre-Training Data: Contrastive pre-training uses English, Chinese, multilingual, and crosslingual instruction or translation pairs, totaling 2,938.8M filtered pairs.Duplicates and low-quality pairs are removed before training.
- Fine-Tuning Data: Fine-tuning draws on publicly available high-quality datasets spanning English and Chinese retrieval and language-understanding tasks.The listed English datasets include MS MARCO, Natural Questions, TriviaQA, HotpotQA, SQuAD, FEVER, and AllNLI.
B.4 Reranker Training Setup
The reranker uses the same fine-tuning data as the TRM but trains on a mixture of hard and random negatives. Contrastive pretraining uses large source-specific batches, with low-resource languages merged when necessary.
- Reranker Fine-Tuning: The reranker and TRM use identical fine-tuning datasets.
- Reranker Fine-Tuning: For each query, reranker training uses 10 negatives: 6 hard negatives and 4 randomly selected negatives.All parameters except batch size match those used for TRM training.
- Contrastive Pre-Training: Low-resource languages whose data cannot fill a complete batch are merged into a combined MIX group.This handling applies to cc-news multilingual pairs under the large-batch setup.
C NLU Evaluation
The text encoder is evaluated on multilingual XTREME-R and English GLUE, using task-specific fine-tuning and transfer setups. Detailed results are reported in Table 13, alongside English models and baselines.
- Benchmark Scope: The evaluation covers multilingual XTREME-R and English GLUE benchmarks, with setup details described for the following tasks.
- Cross-Lingual Transfer: Tatoeba evaluates zero-shot cross-lingual transfer after English fine-tuning and compares the encoder with mBERT-base-cased and XLM-RoBERTa-base.All models are fine-tuned with the same settings and hyperparameters.
- Evaluation Differences: The evaluation code deduplicates the retrieval corpus and implements XCOPA as a multiple-choice task, creating differences from the original XTREME-R evaluation.The authors note that XTREME-R has no final release.
- XTREME-R Tasks: XTREME-R evaluation includes UDPOS, WikiANN, XQuAD, MLQA, TyDiQA-GoldP, Mewsli-X, LAReQA, Tatoeba, XNLI, and XCOPA.The tasks use different fine-tuning or transfer configurations, including English-to-multilingual evaluation for several tasks.
- Results: Table 13 reports detailed results and includes scores for the paper’s English models and external baselines.
D Text Embedding Evaluation
The embedding evaluation spans English, Chinese, French, and Polish MTEB variants. Results show persistent gaps for multilingual models in English and Chinese, while the French TRM is competitive and Polish performance remains limited.
- MTEB-en: Multilingual models still show a noticeable gap compared with English models on English MTEB.The comparison includes the paper’s English embedding models and top-performing leaderboard systems.
- MTEB-zh: The gap between multilingual and Chinese models is quite noticeable on C-MTEB.The evaluation includes several LLM-based embedding models and APIs.
- MTEB-fr: Our dense TRM is comparable to the specialized French mistral-embed API on F-MTEB.Compared with the authors’ continued-pretraining model, fine-tuning provides no significant improvement.
- MTEB-pl: Our model does not outperform large-sized BGE and mE5 on Polish MTEB.The authors speculate that limited Polish contrastive-pretraining pairs caused insufficient training.
E Text Retrieval Evaluation
The retrieval evaluation covers multilingual, cross-lingual, English, and long-context benchmarks. Retrieval uses dense and sparse recall followed by reranking of the top 100 recalled documents.
- Evaluation setup: Retrieval proceeds through recall and reranking stages.Recall combines dense vectors and sparse representations; the ranking stage operates on the top 100 recall results.
- Evaluation setup: The final recall score weights dense retrieval with coefficient 1 and sparse retrieval with coefficients from 0.001 to 0.01.Documents retrieved by neither method receive a score of 0.
- Benchmarks: The evaluation includes MLDR, MKQA, MIRACL, BEIR, and LoCo benchmarks.These cover multilingual, cross-lingual, English, and English long-context retrieval settings.
- Metrics: The reported metrics are nDCG@10 for MLDR, MIRACL, BEIR, and LoCo, and Recall@20 for MKQA.The table captions specify the metric used for each benchmark.
- Benchmarks: MKQA reports cross-lingual retrieval across all 25 languages.The M3-Embedding reference includes a hybrid result combining dense, sparse, and multi-vector scores.