Qwen3-VL-Embedding and Qwen3-VL-Reranker: A Unified Framework for State-of-the-Art Multimodal Retrieval and Ranking

Mingxin Li; Yanzhao Zhang; Dingkun Long; Keqin Chen; Sibo Song; Shuai Bai; Zhibo Yang; Pengjun Xie; An Yang; Dayiheng Liu; Jingren Zhou; Junyang Lin

Source-linked AI summary

Qwen3-VL-Embedding and Qwen3-VL-Reranker: A Unified Framework for State-of-the-Art Multimodal Retrieval and Ranking

Mingxin Li, Yanzhao Zhang, Dingkun Long, Keqin Chen, Sibo Song, Shuai Bai, Zhibo Yang, Pengjun Xie, An Yang, Dayiheng Liu, Jingren Zhou, Junyang Lin

arXiv:2601.04720v2Jan 8, 2026cs.CL

TL;DR

The paper addresses the need for retrieval across heterogeneous multimodal content and introduces Qwen3-VL-based embedding and reranking models. A multi-stage training pipeline combines contrastive learning, supervised task training, and reranker distillation, while deployment features target efficiency. Across broad benchmarks, the series achieves strong multimodal retrieval performance while retaining competitive pure-text capability.

Problem
Multimodal search requires systems that match semantic concepts across diverse images, documents, infographics, screenshots, videos, and text.
Method
The series combines a bi-encoder embedding model and cross-encoder reranker trained through contrastive learning, multi-task supervision, and reranker-to-embedding distillation.
Results
Qwen3-VL-Embedding achieves state-of-the-art performance across multiple downstream tasks, including 77.8 on MMEB-V2 for the 8B model.
Takeaways & Limitations
Matryoshka representation learning and quantization-aware training provide practical deployment flexibility while preserving retrieval performance.

Abstract

from arXiv · show

In this report, we introduce the Qwen3-VL-Embedding and Qwen3-VL-Reranker model series, the latest extensions of the Qwen family built on the Qwen3-VL foundation model. Together, they provide an end-to-end pipeline for high-precision multimodal search by mapping diverse modalities, including text, images, document images, and video, into a unified representation space. The Qwen3-VL-Embedding model employs a multi-stage training paradigm, progressing from large-scale contrastive pre-training to reranking model distillation, to generate semantically rich high-dimensional vectors. It supports Matryoshka Representation Learning, enabling flexible embedding dimensions, and handles inputs up to 32k tokens. Complementing this, Qwen3-VL-Reranker performs fine-grained relevance estimation for query-document pairs using a cross-encoder architecture with cross-attention mechanisms. Both model series inherit the multilingual capabilities of Qwen3-VL, supporting more than 30 languages, and are released in $\textbf{2B}$ and $\textbf{8B}$ parameter sizes to accommodate diverse deployment requirements. Empirical evaluations demonstrate that the Qwen3-VL-Embedding series achieves state-of-the-art results across diverse multimodal embedding evaluation benchmarks. Specifically, Qwen3-VL-Embedding-8B attains an overall score of $\textbf{77.8}$ on MMEB-V2, ranking first among all models (as of January 8, 2025). This report presents the architecture, training methodology, and practical capabilities of the series, demonstrating their effectiveness on various multimodal retrieval tasks, including image-text retrieval, visual question answering, and video-text matching.

1 Introduction

The report introduces Qwen3-VL-Embedding and Qwen3-VL-Reranker as multimodal retrieval models built on Qwen3-VL. The series combines specialized embedding and reranking architectures, flexible deployment features, and strong benchmark performance.

Multimodal search must match semantic concepts across images, documents, infographics, screenshots, videos, and text.
Qwen3-VL-Embedding uses contrastive pre-training followed by ranking-model distillation to produce semantically rich multimodal embeddings.It supports Matryoshka Representation Learning and quantization-aware training for flexible dimensions and efficient deployment.
77.8 on MMEB-V2 is achieved by Qwen3-VL-Embedding-8B, which the report identifies as surpassing all models on the leaderboard.The evaluation passage dates this result to January 2026.
The report covers architecture, training procedures, experiments, and future research directions for both model families.

2 Model

Both model families use the Qwen3-VL backbone but specialize it for retrieval through distinct encoding strategies. Embeddings support dense representation of multimodal inputs, whereas reranking directly judges query-document pairs.

Both models are built on Qwen3-VL with causal attention and are trained in 2B and 8B parameter sizes.They retain the backbone’s multimodal perception, world knowledge, and instruction-following capabilities while gaining relevance estimation.
The embedding model produces task-aware dense vectors for inputs consisting of text, images, videos, or multimodal combinations.Its input format follows the Qwen3-VL context structure and appends a PAD token.
The embedding input template places the instruction and instance in Qwen3-VL system and user messages before the assistant-side end-of-text token.
The reranker uses pointwise ranking to evaluate relevance between two multimodal instances according to an instruction-defined relevance criterion.Inputs may contain text, images, videos, or combinations of these modalities.

3 Data

The data pipeline combines multimodal dataset curation, synthesis, annotation, and relevance-based mining. Its design targets broad coverage across modalities, tasks, domains, and retrieval difficulty levels.

Data synthesis addresses imbalance and scarcity in existing datasets by constructing a balanced corpus across modalities, tasks, and domains.
Each sub-dataset contains an instruction, multimodal queries, a multimodal corpus, and relevance labels.Queries and documents may contain text, images, videos, or multimodal combinations.
Image and video annotations cover classification, question answering, retrieval, video retrieval, and moment retrieval paradigms.Moment retrieval specifically targets fine-grained temporal grounding with relevant and irrelevant segments.
Before task-specific annotation, the system generates descriptive captions for each image or video to improve context, quality, and consistency.
Hard-negative mining first recalls top-K candidates with embeddings, then filters positives and selects negatives using score thresholds and a safety margin.The filtering process is designed to reduce noisy labels and false negatives.

4 Training Strategy

Training proceeds through contrastive pre-training, multi-task learning and reranker training, then embedding distillation and model merging. This sequence balances broad relevance learning with retrieval precision and performance across task types.

The training strategy uses three stages to address the imbalance between abundant weak supervision and scarce high-quality data.
Stage 1: Contrastive Pre-training: Stage 1 contrastively pre-trains the embedding model on large-scale multimodal, multi-task synthetic data.This produces the initial Qwen3-VL-Embedding s0 model.
Stage 2: Multi-Task Contrastive Learning and Supervised Fine-Tuning: Stage 2 combines curated, proprietary, public, and synthetic data for multi-task embedding training and trains a reranker on retrieval-specific tasks.The reranker covers image retrieval, video retrieval, moment retrieval, and visual document retrieval.
Stage 3: Distillation and Model Merging: Stage 3 distills fine-grained relevance scores from the reranker into the embedding model using a balanced retrieval subset.
Stage 3: Distillation and Model Merging: Model merging combines s2 and s1 after s2’s retrieval gains introduce slight degradation on classification and question-answering tasks.The resulting s3 model is reported to provide balanced performance across evaluated tasks.

5 Training Objective

The models use task-specific objectives across multimodal data, then add efficiency-oriented representation training and reranker-guided distillation. The reranker learns pointwise relevance through binary yes/no classification.

Training objectives: Training uses distinct objectives for different data categories, including retrieval, classification, semantic textual similarity, and distillation.The embedding model extends its loss to diverse data types and applies tailored objectives across training stages.
Retrieval loss: Retrieval training uses InfoNCE with cosine similarity, positive pairs, multiple in-batch negatives, and masking to mitigate false negatives.Stage 2 removes query–query and document–document terms from the normalization set, which empirically improves performance on high-quality multimodal retrieval data.
Similarity loss: Semantic textual similarity training uses CoSent loss to preserve the ordering induced by real-valued ground-truth similarity scores.The paired embeddings are optimized so their cosine similarities reflect the supervision ordering.
Distillation: Reranker-guided distillation aligns the embedding model’s cosine-similarity distribution with offline relevance logits from a strong reranker over one positive and k negative documents.The alignment uses a distribution-matching cross-entropy objective over k+1 candidate documents.
Efficiency objectives: Matryoshka training optimizes full embeddings and truncated prefixes, while quantization-aware training jointly uses full-precision and quantized representations.These objectives target competitive intermediate dimensions and robustness under int8 or binary storage.
Reranking loss: The reranker frames pairwise relevance as binary classification, predicting a special yes token for relevant pairs and no for irrelevant pairs.At inference, relevance is computed from the sigmoid of the difference between the yes and no logits.

6 Evaluation

Evaluation covers multimodal, visual-document, and text retrieval benchmarks, with comparisons against open-source, proprietary, text-only, and ColPali-style baselines. The models show strong multimodal and visual-document performance while remaining competitive on pure-text tasks.

MMEB-v2: MMEB-v2 evaluates image, video, and visual-document tasks across nine categories and 78 datasets under a 16,384-token context limit.The evaluation compares the models with prominent open-source and proprietary baselines.
Visual-document benchmarks: The embedding model achieves performance comparable to higher-cost ColPali-style models on JinaVDR and ViDoRe-v3 visual-document retrieval benchmarks.These additional evaluations target visual-document retrieval beyond MMEB-v2.
Visual-document benchmarks: The reranker substantially outperforms ColPali models of similar parameter size on visual-document retrieval.The comparison is reported for the JinaVDR and ViDoRe-v3 evaluations and illustrated in Table 3.
Text benchmarks: 67.9 mean task score on MMTEB is achieved by Qwen3-VL-Embedding-8B, which performs on par with similarly sized text-only embedding models.The series is slightly below text-only Qwen3 embedding models of similar sizes but remains competitive on pure-text tasks.
Reranking evaluation: Reranking is evaluated across image, video, moment-retrieval, visual-document, and text tasks, using Qwen3-VL-Embedding-2B to retrieve the top 100 candidates first.The refinement stage is compared across MMEB-v2, MMTEB, JinaVDR, and ViDoRe v3.

7 Analysis

This analysis examines deployment trade-offs in embedding dimensions and quantization, visual and temporal resource allocation, and performance changes across training stages.

Efficacy of Matryoshka Representation Learning and Embedding Quantization: The analysis evaluates dimensionality and quantization on MSMARCO text retrieval and VL3-Syn text-to-image retrieval.
Efficacy of Matryoshka Representation Learning and Embedding Quantization: 1.4% lower retrieval performance from 1024 to 512 dimensions enables 50% storage reduction and doubles retrieval speed in text retrieval.The same analysis covers text-to-image retrieval and reports that int8 preserves performance with negligible degradation, while binary quantization significantly impairs effectiveness.
Impact of Spatial and Temporal Granularity: Performance improves with greater visual and temporal resource consumption, but diminishing returns and slight regression emerge at the highest levels.Image resolution is measured by visual tokens; video uses frame count for temporal granularity and aggregate tokens across frames for spatial resolution.
Performance Across Training Stages: Distilling from a reranking model substantially boosts the embedding model on retrieval-oriented tasks.The training pipeline produces four embedding models at the 2B size, with performance reported on MMEB-V2.
Performance Across Training Stages: The final model merging stage reconciles slight declines in other task categories and yields robust overall performance across benchmarks.

8 Conclusion

The report presents Qwen3-VL-Embedding and Qwen3-VL-Reranker as a multimodal retrieval model series combining multi-stage training, multimodal data, and Qwen3-VL capabilities. It reports broad benchmark performance alongside deployment efficiencies from representation learning and quantization-aware training.

Qwen3-VL-Embedding and Qwen3-VL-Reranker are presented as a state-of-the-art model series for multimodal retrieval.
The series combines a multi-stage training pipeline with high-quality multimodal data and Qwen3-VL’s multimodal knowledge and general understanding capabilities.
Matryoshka representation learning and quantization-aware training reduce downstream computational costs while preserving performance.

A Dataset Examples

This section provides dataset-format and relevance examples, including a text query paired with positive and negative documents and an image-caption retrieval example.

The dataset examples include Docmatix and MS-COCO formats.
The relevance example marks d_01 as positive for q_01 and d_02 and d_03 as negatives.
The image-caption example pairs a baseball-batting image with related but distinct baseball descriptions.

B Examples of Data Synthesis Prompts

The data-synthesis prompts generate structured multimodal training examples by describing inputs, selecting task types, and producing correct and deceptive alternatives in JSON.

Image Question Answering: Image question answering prompts require visual descriptions, task selection, grounded questions, correct answers, and hard-negative answers.
Image Question Answering: The synthesis process uses task orientations such as factoid identification, visual reasoning, OCR-based extraction, and domain-specific inquiry.
Image Question Answering: The image prompt constrains task types, language fields, and output to a JSON object.
Video Classification: Video classification prompts require visual analysis followed by selection of one task type and structured example population.

C Model Applications and Examples

Qwen3-VL-Embedding is illustrated across real-world multimodal retrieval scenarios spanning text, images, videos, and visual documents. The examples pair queries with candidate content and report similarity scores for selected matches.

Application coverage: The application showcases cover text, image, video, and visual-document retrieval tasks through query–document examples.Tables 8–11 organize similarity scores by text, image, video, and visual-document tasks.
Text tasks: Text retrieval examples include Super Bowl questions, Walgreens salary queries, and bartender wage queries matched with relevant passages.The examples include both factual question answering and topical document retrieval.
Image tasks: Image retrieval examples use visual question answering prompts and image-description matching, including questions about people and a moped scene.The listed tasks include VQAv2 and MS COCO instructions.
Video tasks: Video retrieval examples include questions about a girl’s painted nails and matching a baseball-player description to a video.The examples are associated with NExTQA and MST-VTT tasks.