Qwen3-VL-Embedding and Qwen3-VL-Reranker: A Unified Framework for State-of-the-Art Multimodal Retrieval and Ranking

1 Introduction

• Multimodal search addresses growing demand to retrieve relevant content across query and document modalities, from e-commerce discovery to scientific literature exploration and social media navigation.The introduction frames this as moving beyond traditional text-only search.Source: §1 · p. 2
• VLM-based retrieval builds on shared image-text representations and attention mechanisms, supporting fine-grained cross-modal understanding, complex multimodal documents, and multilingual generalization.The introduction cites CLIP (Radford et al., 2021) for aligned representations and names Qwen-VL (Wang et al., 2024b; Bai et al., 2025), GPT-4o (Hurst et al., 2024), E5-V (Jiang et al., 2024), GME, BGE-VL (Zhou et al., 2025), and VLM2Vec (Meng et al., 2025; Jiang et al., 2025) as prior work.Source: §1 · p. 2
• The series combines multistage, quantization-aware, MRL-capable embeddings with cross-encoder reranking, supporting 32,768-token inputs, more than 30 languages, and 2B and 8B sizes.Embedding training progresses from large-scale multimodal contrastive pre-training to ranking-model distillation; the reranker uses deep cross-attention for relevance scoring.Source: §1 · p. 3
• 77.8 on MMEB-V2 placed Qwen3-VL-Embedding-8B ahead of all leaderboard models, including open-source models and closed-source APIs, in the January 2026 evaluation.The passage identifies the benchmark as Meng et al. (2025).Source: §1 · p. 3
• 67.9 was the Embedding-8B mean task score on MTEB Multilingual; Reranker-8B improved ranking results by 4.1 points over 2B across multiple tasks.Reranker-2B exceeded previously top-performing models in numerous retrieval tasks.Source: §1 · p. 3

Model Architecture

• The embedding model uses a bi-encoder and cosine similarity, while the reranker uses a cross-encoder for finer-grained query–document relevance estimates.Source: §2 · p. 4
• Both models use the causal-attention Qwen3-VL backbone, are trained on multimodal, multi-task relevance data, and come in 2B and 8B sizes.They retain the backbone’s world knowledge, multimodal perception, and instruction-following capabilities while gaining relevance-estimation ability.Source: §2 · p. 4

Embedding Method

• The embedding model encodes text, images, videos, or combinations using a system-message instruction and represents the instance with the final PAD token’s hidden state.The default instruction is “Represent the user’s input.”Source: §2 · p. 4

Reranking Method

• The reranker evaluates multimodal instance pairs pointwise according to an instruction and estimates relevance from the next-token probability of “yes” or “no.”The instruction and pair are supplied as user messages; the instances may be text, images, videos, or combinations.Source: §2 · p. 4

3 Data

• Data synthesis addresses imbalances and scarcity in public and in-house data by constructing a balanced corpus spanning modalities, tasks, and domains.The category distribution is illustrated in Figure 3.Source: §3 · p. 5

3.1 Dataset Format

• Each sub-dataset pairs task instructions with multimodal queries, multimodal documents, and relevance labels.Queries and documents may each contain text, images, videos, or multimodal combinations.Source: §3.1 · p. 5

3.2 Data Synthesis

• A filtered, category-rebalanced image and video seed pool supports Qwen3-VL-32B multimodal annotation, preceded by descriptive caption generation.Filtering removes low-resolution or irregular assets, static or corrupted video segments, and samples with low-confidence labels or poor visual-text correspondence.Source: §3.2 · p. 6
• Image annotations cover classification, question answering, and retrieval, using category labels, grounded answers, and search queries with semantically confusing negatives or distractors.Tasks include object and scene recognition, visual reasoning, OCR, domain-specific questions, and retrieval at varied semantic depths.Source: §3.2 · p. 6
• Video annotations cover classification, question answering, retrieval, and fine-grained moment retrieval with localized segments and temporally separated negative contrasts.The tasks include activity and event categorization, temporal grounding, thematic discovery, and tutorial localization.Source: §3.2 · p. 6

3.3 Positive Refinement and Hard Negative Mining

• A two-stage recall and relevance-filtering pipeline refines positives and selects hard negatives while guarding against false negatives.Queries without a sufficiently similar positive are discarded; non-positive candidates qualify as hard negatives only below the refined positives’ average score plus a safety margin.Source: §3.3 · p. 7

4 Training Strategy

• The three-stage pipeline moves from noisy, weakly supervised data to high-quality task-specific tuning, iteratively improving both data quality and model performance.The authors describe the stages as addressing the imbalance between abundant weakly supervised data and scarce high-quality samples.Source: §4 · p. 7, §4.1 · p. 7

4.1 Multi-stage Training

• Stage 1 uses contrastive learning on large-scale multimodal, multi-task synthetic data mined with an existing open-source embedding model to produce Qwen3-VL-Embedding: s0.The training objective is defined in Equation 1.Source: §4.1 · p. 8
• Stage 2 mines data with s0 and trains the embedding model using task-tailored multi-task contrastive objectives on curated public, proprietary, and synthetic data.This produces Qwen3-VL-Embedding: s1.Source: §4.1 · p. 8
• In Stage 2, the reranker trains on mined retrieval data spanning image, video, moment, and visual-document retrieval tasks.The passage reports superior performance across these retrieval-centric tasks.Source: §4.1 · p. 8
• Stage 3 distills reranker-generated relevance scores on a balanced retrieval subset into the embedding model, then merges s2 with s1 to form the final, balanced s3 model.The merge addresses s2’s slight degradation on classification and QA tasks after retrieval-centric gains; the method follows Li et al. (2024).Source: §4.1 · p. 8

4.2 Implementation

• Training uses LoRA initialized from Qwen3-VL-Instruct, dynamic image resolution capped at 1,280 tokens, and video sampled at 1 FPS up to 64 frames and 4,500 total tokens.The stated LoRA benefits include lower memory use, larger effective batches, better generalization, and more efficient model-merging hyperparameter search.Source: §4.2 · p. 8

5.1 Loss Functions for the Embedding Model

• For T2T, T2I, and IT2IT retrieval, Stage 1 uses cosine-similarity InfoNCE with positive and in-batch negative pairs, masking potential false negatives (Oord et al., 2018).Source: §5.1 · p. 9
• Removing query-query and document-document terms in Stage 2 improves performance on high-quality multimodal retrieval data.Source: §5.1 · p. 9
• For text and image classification, the instance is a query and its class label is positive; only explicitly incorrect labels for that query serve as negatives.Other labels in the batch are ignored to avoid false negatives.Source: §5.1 · p. 9
• For symmetric STS data with real-valued supervision, CoSent loss preserves the ground-truth ordering of paired-embedding cosine similarities.Source: §5.1 · p. 9
• In final-stage distillation, the embedding model matches its score distribution to offline reranker logits for one positive and k negative documents per query.The objective is cross-entropy over the k+1 candidate documents, with embedding scores computed online using cosine similarity.Source: §5.1 · p. 9

5.1.1 Additional Techniques for Efficient Inference

• MRL trains full embeddings and truncated prefixes for flexible dimensions, while QAT trains on full-precision and quantized embeddings to support low-bit deployment (Kusupati et al., 2022; Esser et al., 2020).QAT uses LSQ with a learnable quantization scale and a Straight-Through Estimator for rounding gradients; dense MRL dimensions generalize to intermediate dimensions not explicitly trained.Source: §5.1.1 · p. 10

5.2 Loss Function for the Reranking Model

• The reranker trains query-document relevance as yes/no binary classification and computes inference scores with the sigmoid of the yes-minus-no logit difference (Dai et al., 2025).Source: §5.2 · p. 10

6.1 Multimodal Benchmarks

• 77.8 average score on MMEB-v2 (Meng et al., 2025), 6.7% above the previous best open-source model, is the reported SOTA for Qwen3-VL-Embedding-8B across image, video, and visual-document domains.MMEB-v2 covers nine task categories and 78 datasets; evaluation used a 16,384-token context limit, with image tokens capped at 1,800 and video at 15,000 tokens and 64 frames.Source: §6.1 · p. 10, Table 2 · p. 11

6.2 Visual Document Benchmarks

• On JinaVDR (Günther et al., 2025) and ViDoRe v3, embeddings are comparable to higher-compute ColPali-style models, while the reranker substantially outperforms similarly sized ColPali models.Source: §6.2 · p. 10, Table 3 · p. 11

6.3 Text Benchmarks

• 67.9 mean task score on MMTEB (Enevoldsen et al., 2025b) places Qwen3-VL-Embedding-8B on par with similarly sized text-only embedding models, despite slightly lower performance than similarly sized text-only Qwen3 models.Source: §6.3 · p. 11

6.4 Evaluation for Reranking Model

• All three Qwen3-VL-Reranker models outperform the base embedding model and baseline rerankers, with the 8B variant performing best on most tasks.Reranking refines the top 100 candidates retrieved using Qwen3-VL-Embedding-2B across multimodal and text retrieval evaluations.Source: §6.4 · p. 11, Table 5 · p. 12

7.3 Performance Across Training Stages

• Distillation from a reranking model substantially boosts retrieval-oriented tasks, while final model merging reconciles slight declines elsewhere for robust, superior overall benchmark performance.Table 6 reports the performance of four 2B training-stage models on MMEB-V2; the supplied text gives no specific scores.Source: Table 6 · p. 13, §7.3 · pp. 13–14

7.1 Efficacy of Matryoshka Representation Learning and Embedding Quantization

• At 512 dimensions versus 1024, text-retrieval performance drops only 1.4% while storage halves and speed doubles; int8 is near-lossless, unlike binary quantization.Qwen3-VL-Embedding integrates MRL and QAT to address storage and latency needs; evaluation used Qwen3-VL-Embedding-2B and MRR@10 on MSMARCO text retrieval and VL3-Syn text-to-image retrieval.Source: §7.1 · pp. 12–13, Figure 6 · p. 12

7.2 Impact of Spatial and Temporal Granularity

• Across image, video, and visual-document tasks, performance generally improves with greater visual-resource use, but gains diminish and slightly regress at the highest levels.The analysis varies image spatial resolution, video frame count, and aggregate video token budget; the authors suggest excessively long contexts may explain the regression.Source: §7.2 · p. 13, Figure 7 · p. 13

8 Conclusion

• The Qwen3-VL-Embedding and Qwen3-VL-Reranker series achieve unprecedented performance across multimodal retrieval benchmarks while maintaining strong pure-text capabilities.The models integrate a multi-stage training pipeline and high-quality multimodal data with the knowledge and general understanding of Qwen3-VL Foundation models.Source: §8 · p. 14
• Matryoshka representation learning and quantization-aware training reduce downstream computational costs while preserving performance, supporting practical deployment of Qwen3-VL-Embedding.Source: §8 · p. 14
• Future directions include supporting additional modalities, improving training efficiency and compositional reasoning, and establishing more comprehensive evaluation protocols.Source: §8 · p. 14

• Table 7 identifies Docmatix† and MS-COCO (Lin et al., 2014) as the dataset formats illustrated.Source: Table 7 · p. 18
• The Docmatix example asks what research project the Danish Cancer Society announced on 01/02/21, with d_01 positive and d_02 and d_03 negative.Source: A Dataset Examples · p. 18
• For MS-COCO (Lin et al., 2014), an image-caption query marks “A man swinging a baseball bat on a baseball field” positive and two other baseball captions negative.The query instruction asks for a caption describing an everyday image.Source: A Dataset Examples · p. 18, Table 7 · p. 18

Image Question Answering Prompt

• IQA synthesis begins with a visual description of scene details and reasoning opportunities, then selects one task type: factoid identification, visual reasoning, OCR extraction, or domain-specific inquiry.The description is limited to 500 words.Source: B Examples of Data Synthesis Prompts · p. 19
• The IQA example contains a description, task type, visually grounded question, positive answer, and plausible hard-negative answer, returned as a single JSON object.The question must be answerable from the visual or embedded text, and the task type must come from the specified list.Source: B Examples of Data Synthesis Prompts · p. 19

Video Classification Prompt

• Video synthesis starts with an overview of content, actions, setting, and event type, then selects one task: activity recognition, scene parsing, event categorization, or sentiment/intent analysis.The visual analysis is limited to 300 words.Source: B Examples of Data Synthesis Prompts · p. 19
• The video example outputs a description, task type, correct label, and plausible misleading label in a single JSON object, with the specified task type and language constraints.The description, label, and misleading label must be in the target language; no extra text or explanations are allowed.Source: B Examples of Data Synthesis Prompts · pp. 19–20

C Model Applications and Examples

• The showcases illustrate the model’s practical utility and integration into downstream tasks involving diverse queries and complex visual data.Source: C Model Applications and Examples · p. 20
• Table 8 reports Qwen3-VL-Embedding similarity scores for text tasks.Source: Table 8 · p. 20
• Table 9 reports Qwen3-VL-Embedding similarity scores for image tasks.Source: Table 9 · p. 21
• The video example uses the query “baseball player hits ball,” and Table 10 reports similarity scores for video tasks.Source: Table 10 · p. 22, C Model Applications and Examples · p. 22
• Table 11 reports Qwen3-VL-Embedding similarity scores for visual-document tasks.Source: Table 11 · p. 23