Source-linked AI summary
Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Models
Yanzhao Zhang, Mingxin Li, Dingkun Long, Xin Zhang, Huan Lin, Baosong Yang, Pengjun Xie, An Yang, Dayiheng Liu, Junyang Lin, Fei Huang, Jingren Zhou
TL;DR
Text embedding and reranking require models that capture relevance across varied languages, tasks, and retrieval settings. Qwen3 Embedding addresses this with Qwen3 backbones, synthetic multilingual data, staged training, and model merging. Across broad evaluations, the series achieves strong embedding and reranking results, while reranker training omits the first-stage weakly supervised phase.
Problem
Text embedding and reranking must support relevance estimation across multilingual, multi-task retrieval settings that motivate richer training data and model capabilities.
Method
The series uses Qwen3 backbones, foundation-model-synthesized multilingual data, multi-stage training, supervised fine-tuning, and model merging.
Results
Qwen3 Embedding achieves state-of-the-art performance across diverse benchmarks, including 70.58 on MTEB Multilingual and 80.68 on MTEB Code for Qwen3-8B-Embedding.
Takeaways & Limitations
The series provides embedding and reranking models in multiple sizes for multilingual retrieval, code retrieval, and other downstream tasks.
Takeaways & Limitations
Reranking models do not include a first-stage weakly supervised training phase.
Abstract
from arXiv · showhide
In this work, we introduce the Qwen3 Embedding series, a significant advancement over its predecessor, the GTE-Qwen series, in text embedding and reranking capabilities, built upon the Qwen3 foundation models. Leveraging the Qwen3 LLMs' robust capabilities in multilingual text understanding and generation, our innovative multi-stage training pipeline combines large-scale unsupervised pre-training with supervised fine-tuning on high-quality datasets. Effective model merging strategies further ensure the robustness and adaptability of the Qwen3 Embedding series. During the training process, the Qwen3 LLMs serve not only as backbone models but also play a crucial role in synthesizing high-quality, rich, and diverse training data across multiple domains and languages, thus enhancing the training pipeline. The Qwen3 Embedding series offers a spectrum of model sizes (0.6B, 4B, 8B) for both embedding and reranking tasks, addressing diverse deployment scenarios where users can optimize for either efficiency or effectiveness. Empirical evaluations demonstrate that the Qwen3 Embedding series achieves state-of-the-art results across diverse benchmarks. Notably, it excels on the multilingual evaluation benchmark MTEB for text embedding, as well as in various retrieval tasks, including code retrieval, cross-lingual retrieval and multilingual retrieval. To facilitate reproducibility and promote community-driven research and development, the Qwen3 Embedding models are publicly available under the Apache 2.0 license.
1 Introduction
Qwen3 Embedding builds embedding and reranking models on Qwen3 foundation models, using multilingual capabilities and a multi-stage training pipeline. Evaluations report strong performance across multilingual, code, and retrieval benchmarks.
- Qwen3 Embedding models use Qwen3 foundation models for multilingual text understanding, generation, and training-data synthesis.The pipeline uses Qwen3 models as both backbones and generators of multilingual, multi-task relevance data.
- The embedding pipeline combines large-scale unsupervised pre-training, supervised fine-tuning on high-quality data, and model merging.Rerankers instead use supervised fine-tuning followed by model merging.
- 70.58 on MTEB Multilingual and 80.68 on MTEB Code are achieved by Qwen3-8B-Embedding, surpassing Gemini-Embedding.
- Qwen3-Reranker-8B improves ranking results by 3.0 points over Qwen3-Reranker-0.6B across multiple tasks.The 0.6B reranker also exceeds previously top-performing models in numerous retrieval tasks.
- The report describes model architectures, training procedures, experiments, and future research directions.
2 Model Architecture
The models estimate task-aware query-document relevance using instruction-conditioned inputs. Embeddings use a final EOS hidden state, while rerankers classify each document as relevant or irrelevant.
- Embedding and reranking models evaluate query-document relevance using a similarity criterion defined by instruction I.Training examples contain instructions, queries, positive documents, and negative documents.
- The Qwen3 embedding and reranking models are available in 0.6B, 4B, and 8B parameter sizes.They are initialized from dense Qwen3 foundation models.
- Embedding Models: Embedding vectors are derived from the final-layer hidden state of an appended [EOS] token.Instructions and queries are concatenated, while documents remain unchanged before processing.
- Reranking Models: Rerankers assess one document at a time by framing instruction-conditioned similarity evaluation as binary yes-or-no classification.The LLM chat template requires the answer to be either “yes” or “no.”
- Table 1 defines MRL Support as custom final embedding dimensions and Instruction Aware as customizable task instructions.
- Reranking Models: Relevance scoring uses the likelihood of the next token being “yes” or “no.”The displayed expression combines the two token probabilities.
3 Models Training
The Qwen3 Embedding series uses task-specific objectives within a multi-stage training pipeline, combining synthetic weak supervision, supervised fine-tuning, and model merging.
- 3.1 Training Objective: The embedding objective uses an improved InfoNCE-based contrastive loss over positive and negative pairs, with cosine similarity, temperature scaling, and normalization.The loss also incorporates in-batch queries and documents, while a mask factor mitigates false negatives.
- 3.1 Training Objective: The reranking objective is supervised fine-tuning that assigns “yes” to positive documents and “no” to negatives.The loss encourages higher probabilities for correct labels.
- 3.2 Multi-stage Training: Embedding training proceeds from large-scale weakly supervised data to smaller high-quality supervised datasets, while reranking omits the first-stage weakly supervised phase.Embedding stages use the embedding objective, whereas reranking uses the supervised fine-tuning loss.
- 3.2 Multi-stage Training: Foundation models synthesize controllable pair data across tasks, languages, lengths, and difficulty, improving quality and diversity particularly in low-resource settings.The synthetic corpus spans retrieval, bitext mining, classification, and semantic textual similarity tasks.
- 3.2 Multi-stage Training: After supervised fine-tuning, spherical linear interpolation merges multiple checkpoints to improve robustness and generalization across data distributions.This model-merging step is applied after fine-tuning.
- 3.2 Multi-stage Training: Approximately 150 million synthetic weak-supervision pairs are generated, followed by selection of approximately 12 million high-quality supervised pairs.The filtering stage selects data for second-stage supervised training.
4 Evaluation
The Qwen3 Embedding series is evaluated across multilingual, code, retrieval, and reranking benchmarks, with strong performance across model sizes and training settings. Ablations indicate that weakly supervised pre-training and model merging are important components of the embedding training framework.
- Evaluation settings: The evaluation covers over 500 MMTEB tasks across more than 250 languages, including retrieval, classification, semantic textual similarity, instruction following, long-document retrieval, and code retrieval.
- Embedding: Qwen3-Embedding-4B/8B achieve the best performance on MMTEB, while Qwen3-Embedding-0.6B remains close to the strongest baseline despite its smaller size.The 0.6B model ranks just behind Gemini-Embedding and is competitive with gte-Qwen2-7B-instruct on additional benchmarks.
- Embedding: Qwen3-Embedding-4B/8B consistently outperform other models across MTEB English, CMTEB, and MTEB Code evaluations.
- Reranking: All three Qwen3-Reranker models improve over the embedding model and surpass baseline reranking methods, with Qwen3-Reranker-8B highest across most tasks.Evaluation uses top-100 candidates retrieved by Qwen3-Embedding-0.6B before reranking.
- Training analysis: Removing weakly supervised training causes a clear performance decline, indicating that large-scale synthetic-data training is crucial for superior embedding performance.
- Training analysis: Training without model merging performs considerably worse than the final Qwen3-Embedding-0.6B model, indicating that model merging is also critical.
5 Conclusion
The report presents Qwen3-based embedding and reranking models trained through a multi-stage pipeline with synthetic weak supervision, supervised fine-tuning, and model merging. The released 0.6B, 4B, and 8B models achieve state-of-the-art results across diverse benchmarks and are open sourced for community use.
- Qwen3-Embedding models support multilingual retrieval, code retrieval, and complex instruction-following tasks through a multi-stage training pipeline.The pipeline combines weakly supervised synthetic-data pre-training, supervised fine-tuning, and model merging.
- Qwen3 LLMs synthesize diverse training data across multiple languages and tasks to enhance the embedding and reranking models.
- The Qwen3-Embedding and Qwen3-Reranker models are released in 0.6B, 4B, and 8B sizes for community use and further development.
- Comprehensive evaluations show state-of-the-art performance across MTEB, CMTEB, MMTEB, and several retrieval benchmarks.
A.1 Synthetic Data
The paper constructs diverse synthetic training data for similarity tasks using a staged document-to-query generation process. LLM-selected configurations guide query generation across languages and difficulty levels.
- Synthetic data types: Four synthetic data types—retrieval, bitext mining, semantic textual similarity, and classification—support adaptation to varied similarity tasks during pre-training.
- Retrieval data generation: A document-to-query pipeline first selects a character, question type, and difficulty before generating a query from the passage.
- Language diversity: Multilingual and cross-lingual diversity is produced by generating data with Qwen3 32B and sourcing documents from the Qwen3 base model’s multilingual pre-training corpus.
- Query generation: The query-generation stage specifies the desired query length and language while using the earlier configuration to guide generation.
- Data statistics: The training-data statistics are reported for each training stage in Table 6.
A.2 Detail Results
The detailed results section reports benchmark evaluations on English, Chinese, and code retrieval settings. The supplied passages identify the corresponding result tables but do not state their numerical outcomes.
- English benchmark: Table 7 reports results on the English MTEB benchmark, MTEB(eng, v2).
- Chinese benchmark: Table 8 reports results on the Chinese C-MTEB benchmark, identified as MTEB(cmn, v1).
- Code benchmark: Table 9 reports code-retrieval performance on MTEB(Code, v1) using nDCG@10 scores.