PaperlayerView exact revision on arXiv

Source-linked AI summary

Embed-RL: Reinforcement Learning for Reasoning-Driven Multimodal Embeddings

Haonan Jiang, Yuji Wang, Yongjie Zhu, Xin Lu, Wenyu Qin, Meng Wang, Pengfei Wan, Yansong Tang

arXiv:2602.13823v3cs.CV

TL;DR

Existing generative multimodal embedding methods produce textual CoTs that can be retrieval-misaligned, while joint generative and embedding optimization can create gradient conflicts. Embed-RL decouples these processes with Embedder-Guided Reinforcement Learning and multimodal evidential Traceability CoT, and outperforms state-of-the-art counterparts on MMEB-V2 and UVRB under computational constraints. The paper reports that targeted reasoning optimization substantially enhances multimodal embeddings.

  • Problem

    Existing generative UME methods use text-only, retrieval-irrelevant CoTs, while joint generative and embedding objectives can produce conflicting gradients.

  • Method

    Embed-RL uses decoupled Embedder-Guided Reinforcement Learning with retrieval-oriented dual rewards to optimize multimodal evidential Traceability CoT.

  • Results

    Embed-RL outperforms state-of-the-art counterparts on MMEB-V2 and UVRB under computational constraints.

  • Takeaways & Limitations

    Targeted reasoning optimization substantially enhances multimodal embeddings and supports efficient reasoning-driven UME development.

  • Takeaways & Limitations

    The multi-component reward weights are empirically fixed, which may yield suboptimal performance for specific multimodal scenarios.

Abstract

from arXiv · show

Leveraging Multimodal Large Language Models (MLLMs) has become pivotal for advancing Universal Multimodal Embeddings (UME) in addressing diverse cross-modal tasks. Recent studies demonstrate that incorporating generative Chain-of-Thought (CoT) reasoning can substantially enhance task-specific representations compared to discriminative methods. However, the generated reasoning CoTs of existing generative embedding methods are limited to the textual analysis of queries and are irrelevant to the retrieval of the targets. To address these limitations, we propose a reasoning-driven UME framework that integrates Embedder-Guided Reinforcement Learning (EG-RL) to optimize the Reasoner to produce evidential Traceability CoT (T-CoT). Our key contributions are threefold: (1) We design an EG-RL framework where the Embedder provides explicit supervision to the Reasoner, ensuring the generated CoT traces are aligned with embedding tasks. (2) We introduce T-CoT, which extracts critical multimodal cues to focus on retrieval-relevant elements and provides multimodal inputs for the Embedder. (3) With limited computational resources, our framework outperforms the pioneering embedding model on both MMEB-V2 and UVRB benchmarks. The integration of multimodal evidence in structured reasoning, paired with retrieval-oriented alignment, effectively strengthens cross-modal semantic consistency and boosts the fine-grained matching capability of the model as well as the generalization across complex scenarios. Our work demonstrates that targeted reasoning optimization can significantly improve multimodal embedding quality, providing a practical and efficient solution for reasoning-driven UME development.

1 Introduction

Existing multimodal embedding methods increasingly use MLLMs and generative reasoning, but current approaches suffer from objective conflicts, retrieval-misaligned textual CoTs, and insufficient multimodal evidence. Embed-RL addresses these limitations with Embedder-Guided Reinforcement Learning and evidential Traceability CoT.

  • Motivation: MLLM-powered embedding methods seek to improve cross-modal understanding beyond traditional dual-encoder architectures.Traditional methods include CLIP, BLIP, and SigLIP, while MLLMs offer multimodal understanding and instruction-following capabilities.
  • Limitations of Existing Methods: Discriminative MLLM embedding models extract features from final hidden states, underusing generative reasoning capabilities.Recent work therefore explores generative reasoning for universal multimodal embedding tasks.
  • Limitations of Existing Methods: Jointly optimizing contrastive loss and next-token prediction creates conflicting gradients that can reduce performance.UME-R1 is identified as an example of this unified optimization approach.
  • Limitations of Existing Methods: Decoupled Reasoner-Embedder methods avoid objective conflict but may generate CoTs misaligned with embedding tasks, introducing noise or hallucinations.TTE generates offline CoTs without training the Reasoner together with the Embedder.
  • Proposed Framework: Embed-RL uses Embedder-Guided Reinforcement Learning to optimize Reasoner-generated CoTs with retrieval-oriented rewards.The trained Embedder provides supervision through process and outcome rewards aligned with query-target retrieval.
  • Proposed Framework: Evidential Traceability CoT integrates visual localization, video keyframes, and text keywords to emphasize retrieval-relevant multimodal information.The design aims to reduce the impact of redundant multimodal and textual data on embedding alignment.

2 Related Work

Universal multimodal embedding research advances through MLLM-based retrieval models, instruction-aware benchmarks, and reinforcement-learning methods for multimodal reasoning. Embed-RL builds on these directions by structuring retrieval-relevant visual and textual evidence into multimodal reasoning chains.

  • Universal Multimodal Embedding: Dual-encoder models such as CLIP and ALIGN learn multimodal representations through contrastive learning but struggle with interleaved image-text inputs and complex text.MLLM-based embedding models address these limitations by leveraging stronger multimodal comprehension.
  • Universal Multimodal Embedding: Prior embedding research includes contrastive MLLM adaptation, zero-shot reranking, and two-stage retrieval training with joint reranking.VLM2Vec, MM-Embed, and LamRA represent these complementary directions.
  • Instruction-Aware Representations: MMEB and MMEB-V2 provide instruction-aware evaluation across 78 multimodal tasks.UME-R1 introduces reasoning but encounters optimization conflicts and redundant CoT trajectories, while TTE is computationally expensive and retrieval-misaligned.
  • Multimodal Reasoning with Reinforcement Learning: Recent multimodal reasoning work applies RL to visual grounding, response generation, visual attention, and image-based thinking.Examples include GRIT, Ground-R1, BRPO, DeepEyes, and TreeVGR.
  • Multimodal Reasoning with Reinforcement Learning: T-CoT extends grounding-oriented reasoning by extracting image bounding boxes, video keyframes, and text keywords into structured multimodal chains.This design focuses the model on core retrieval regions and improves embedding quality.

3 Methodology

The methodology constructs retrieval-aligned multimodal reasoning data, then uses a frozen Embedder to guide Reasoner optimization through structured rewards and GRPO. T-CoT combines modality-specific evidence with retrieval-oriented reasoning so generated embeddings better distinguish relevant targets.

  • 3.1 Preliminaries: The task maps multimodal queries to candidate targets and trains embeddings to maximize positive-query similarity over negatives using InfoNCE.The embeddings are extracted from the final-token hidden states, and the temperature τ controls the contrastive similarity distribution.
  • 3.2 Data Construction: The data pipeline samples image, video, and visual-document tasks, annotates query-positive pairs with T-CoT, filters contradictions, and splits data for contrastive and reinforcement learning.Each T-CoT extracts text keywords, image bounding boxes, and video key frames, then refines reasoning before summarizing retrieval-relevant information.
  • 3.3 Embedder-Guided Reinforcement Learning: EG-RL keeps the Reasoner and contrastively trained Embedder decoupled, using the frozen Embedder to provide stable supervision for retrieval-aligned T-CoT generation.This separation optimizes reasoning without disrupting the Embedder’s learned discriminative capabilities, while the <emb> hidden state becomes the final embedding evaluated by the Embedder.
  • 3.3 Embedder-Guided Reinforcement Learning: The reward combines format compliance, outcome-level retrieval effectiveness, and process-level alignment between query and target T-CoTs.Outcome guidance evaluates positive ranking accuracy and the similarity margin against hard negatives, while process reward identifies whether shuffled candidates select an aligned positive T-CoT.
  • 3.3 Embedder-Guided Reinforcement Learning: GRPO samples groups of candidate T-CoTs and updates the Reasoner using clipped importance ratios, group-normalized advantages, and a KL penalty against a reference policy.For each query-target pair, the method samples G = 8 candidate sequences; the coefficients α, β, and γ balance the reward components.

4 Experiments

Experiments evaluate Embed-RL across multimodal benchmarks and ablations, showing strong performance across modalities and evidence that EG-RL and multimodal T-CoT components improve retrieval alignment and discrimination.

  • Baselines and Datasets: The evaluation uses MMEB-V2’s 78 tasks across image, video, and visual-document modalities, plus UVRB’s 16 video-retrieval datasets.MMEB-V2 reports Hit@1 for image and video tasks and NDCG for visual-document tasks, while UVRB reports mAP.
  • Main Results: Embed-RL-4B achieves 68.1 on MMEB-V2, exceeding UME-R1-7B by 3.6 points, while Embed-RL-2B reaches 66.8.Embed-RL-4B also records 91.4 for image grounding; both models outperform baselines on video overall scores.
  • Main Results: On UVRB, Embed-RL-4B leads coarse-grained retrieval at 60.7 and fine-grained retrieval at 55.6, while Embed-RL-2B leads long-context retrieval at 86.9.The reported results outperform existing baselines across the three domain dimensions.
  • Main Results: T-CoT crops bounding boxes and video keyframes to provide multimodal reasoning inputs that locate retrieval-relevant evidence.The visualization covers text, image, and video retrieval scenarios.
  • Ablation Study: Removing RL lowers MMEB-V2 performance from 66.8 to 65.3, while removing weighted negative sampling lowers it to 66.5.The process reward contributes 0.8 points, with its removal reducing video performance from 52.1 to 51.3.
  • Ablation Study: Removing reasoning reduces performance from 66.8 to 65.5, and removing multimodal cues reduces it to 65.8.Reasoning removal lowers image grounding from 69.2 to 67.9 and video moment retrieval from 52.1 to 50.5.
  • Ablation Study: EG-RL enlarges the similarity gap between top-ranked candidates across three modalities, indicating improved discrimination among similar candidates.The analysis defines the gap using similarity to the most similar candidate versus the second-most similar candidate.
  • Ablation Study: After reinforcement learning, image T-CoT uses more bounding boxes while video T-CoT focuses on fewer keyframes, alongside consistently improved retrieval metrics.The changes are especially pronounced for complex multi-object localization and multi-person relationship reasoning samples.

5 Conclusion

Embed-RL addresses text-only, retrieval-misaligned CoT and conflicts between generative and embedding objectives through EG-RL and multimodal T-CoT. Experiments report stronger cross-modal retrieval and out-of-domain generalization within computational constraints.

  • Embed-RL targets text-only CoT, poor retrieval relevance, and gradient conflicts between generative and embedding objectives.
  • EG-RL decouples reinforcement learning to align Reasoner-generated CoT trajectories with embedding objectives.
  • Multimodal evidential T-CoT and a retrieval-oriented dual-reward mechanism support precise reasoning-embedding alignment.
  • Embed-RL outperforms state-of-the-art counterparts on MMEB-V2 and UVRB within computational constraints.
  • The framework improves cross-modal retrieval and out-of-domain generalization, while targeted reasoning optimization enhances multimodal embeddings.

Supplementary Material

The supplementary material expands the paper with additional insights, detailed derivations, and qualitative results.

  • The supplementary material provides further insights, detailed derivations, and additional qualitative results.

B Training Details

Training uses Qwen3-VL models with DeepSpeed and dataset-consistent batching, then applies GRPO to optimize the Reasoner with contrastive and retrieval-oriented rewards. Visual processing uses fixed pixel, frame, and cropping constraints.

  • Model and training setup: Qwen3-VL-2B-Instruct and Qwen3-VL-4B-Instruct are trained with DeepSpeed Zero2 for 2 epochs using batch sizes 512 and 256.Sub-batch sizes are 256 and 128, respectively, with samples grouped by dataset.
  • Reinforcement learning: Qwen3-VL-8B-Instruct Reasoner training uses GRPO with in-batch negative contrastive rewards for one epoch.The configuration uses group size G = 8, ε = 0.2, β = 0.01, batch size 256, and learning rate 3e-6.
  • Reward design: The outcome reward jointly evaluates positive-sample ranking accuracy and the similarity margin against hard negatives.The margin uses softmax-weighted negative similarities scaled by temperature τ.
  • Reward design: Acc8 checks whether all 8 T-CoT rollouts rank in the top-8 by cosine similarity to the query embedding.This measures embedding-alignment consistency for GRPO reward optimization.
  • Reward design: The temperature parameter τ scales softmax weights for hard-negative sampling, with τ = 0.5 used in all experiments.The setting emphasizes hard negatives while avoiding overfitting to noisy negatives.
  • Reward design: The process reward uses listwise VLM discrimination to select the candidate T-CoT that best matches the query-target relationship.Selection correctness supplies the process reward signal.
  • Visual input processing: Visual processing fixes image and video pixel limits, samples 8 frames, and re-extracts cropped keyframes for multimodal reasoning.Bounding boxes are mapped from 0–1000 relative coordinates to original image coordinates before cropping.

C.1 Data Sources and Initial Sampling Strategy

The training corpus combines video-language, visual-document, and image-task data with modality-balanced sampling and CoT relevance filtering. Three classification datasets are excluded because false negatives and noise impair contrastive representation learning.

  • Data sources: The dataset combines video-language instruction, visual-document retrieval, and image-based vision-task data.
  • Initial sampling strategy: Stratified sampling balances modalities, capping image, document, and video datasets at 50,000, 100,000, and 300,000 samples.Datasets smaller than each maximum are fully sampled.
  • Data limitation: HatefulMemes, N24News, and VOC2007 are excluded from first-stage contrastive learning because limited categories create false negatives and noisy large sub-batches.The exclusion leaves the model without these samples during training, explaining weak image-classification performance.
  • CoT-guided filtering: CoT annotations are generated for queries and positive samples, then filtered to remove irrelevant or conflicting annotations.Only samples labeled "No" by the relevance-and-conflict judgment prompt are retained.

C.3 RL Dataset Sampling

The RL dataset is constructed from high-quality CoT-filtered data with equidistant sampling, while task-specific prompts structure multimodal reasoning and evidence extraction. The resulting evaluation spans diverse retrieval scenarios, including coarse- and fine-grained video retrieval.

  • C.3 RL Dataset Sampling: Equidistant sampling constructs the RL training set with a uniform distribution across challenging sub-datasets.The sampling is applied to the high-quality CoT-filtered dataset.
  • C.3 RL Dataset Sampling: Hierarchical prompts constrain reasoning logic, output formatting, and evidence anchoring across text, image, and video modalities.The system uses two scenario-specialized modules and two-round fixed-format reasoning.
  • C.3 RL Dataset Sampling: Text-to-image prompts extract visual concepts and output fixed-format keyword lists for retrieval.The prompts adapt reasoning to task types such as caption-, news-, dialogue-, and question-based retrieval.
  • C.3 RL Dataset Sampling: Image prompts ground reasoning in visual evidence and require coordinates for key elements relevant to the task.They support retrieval, classification, question answering, and object identification or segmentation.
  • C.3 RL Dataset Sampling: Video prompts identify relevant key frames and adapt reasoning to captioning, question answering, and retrieval tasks.Text-to-video prompts additionally extract temporal visual concepts into standardized keyword outputs.
  • C.3 RL Dataset Sampling: UVRB evaluation covers coarse-grained, fine-grained, and long-context retrieval across textual, composed, and visual scenarios.The benchmark contains 16 datasets targeting distinct retrieval capabilities.

G.1 Video Retrieval Performance on UVRB Datasets

UVRB evaluates video retrieval across 16 datasets using task-appropriate recall and precision metrics. Embed-RL-2B and Embed-RL-4B outperform all baselines in average performance, with Embed-RL-4B reaching an AVG of 0.602.

  • G.1 Video Retrieval Performance on UVRB Datasets: UVRB comprises 16 datasets spanning coarse-grained, fine-grained, and long-context retrieval scenarios.The benchmark includes textual, composed, and visual retrieval paradigms.
  • G.1 Video Retrieval Performance on UVRB Datasets: R@1 is the primary metric, while R@10 is additionally reported for ambiguous queries and P@1 for multiple-positive datasets.These metrics measure top-1 retrieval accuracy, top-k retrieval, and precision at one, respectively.
  • G.1 Video Retrieval Performance on UVRB Datasets: 0.602 is the highest AVG achieved by Embed-RL-4B, which outperforms all baselines on UVRB.Embed-RL-2B also exceeds the baselines in average score.
  • G.1 Video Retrieval Performance on UVRB Datasets: Embed-RL models secure first- or second-place results on coarse-grained, fine-grained, and long-context datasets.Reported examples include DiDeMo, CRB-G, CRB-S, and VDC-O/VDC-D.

G.2 Capability Characterization of the UVRB Evaluation Metrics

UVRB characterizes model capability across task, domain, and sub-domain dimensions using unweighted arithmetic means. Its AVG aggregates six capability columns rather than averaging raw dataset scores.

  • G.2 Capability Characterization of the UVRB Evaluation Metrics: UVRB uses unweighted arithmetic means for metric calculations to compare heterogeneous datasets fairly.Results are rounded to three decimal places.
  • G.2 Capability Characterization of the UVRB Evaluation Metrics: UVRB partitions 16 datasets into non-overlapping task, domain, and sub-domain categories.This partition supports capability measurement across distinct retrieval scenarios.
  • G.2 Capability Characterization of the UVRB Evaluation Metrics: The task dimension distinguishes textual, composed, and visual retrieval according to query format.It reflects cross-modal alignment ability for different query types.
  • G.2 Capability Characterization of the UVRB Evaluation Metrics: The domain dimension measures performance on coarse-grained, fine-grained, and long-context retrieval.It assesses generalization across semantic granularity and context length.
  • G.2 Capability Characterization of the UVRB Evaluation Metrics: The AVG score is the arithmetic mean of the TXT, CMP, VIS, CG, FG, and LC columns.This aggregates core retrieval paradigms and domains rather than raw dataset averages.

H.1 Training Metrics

Training analysis tracks RL-phase entropy, response length, reward, contrastive loss, and gradient norm for 2B and 4B models. Entropy plateaus, response length rises, and reward fluctuates upward during RL.

  • H.1 Training Metrics: Entropy declines gradually and then plateaus during the reinforcement learning phase.This behavior is reported alongside increasing response length and fluctuating reward.
  • H.1 Training Metrics: Response length increases steadily as reinforcement learning proceeds.The metric is tracked together with entropy and reward.
  • H.1 Training Metrics: Reward shows a fluctuating upward trend attributed to in-batch reward sample sampling.The passage attributes the fluctuations to differences between individual samples exceeding the inherent reward growth.
  • H.1 Training Metrics: Contrastive training loss and gradient norm are tracked for both 2B- and 4B-scale Embed-RL models.The analysis uses these metrics to characterize contrastive-learning training behavior.
  • H.1 Training Metrics: T-CoT requires one offline generation per multimodal retrieval target, after which its embedding can be stored in the retrieval database.This design gives it negligible latency overhead compared with traditional generative embedding methods.

J Limitations

The framework’s main limitations concern reward tuning, classification coverage, and the absence of hard-negative or curriculum strategies. These constraints may affect performance in specific scenarios, especially image-classification subtasks.

  • Reward optimization: Empirically chosen reward weights lack adaptive optimization for diverse multimodal tasks, potentially causing suboptimal performance in specific scenarios.The limitation concerns the multi-component reward function used by the framework.
  • Task coverage: Excluding partial classification tasks from the constructed dataset results in relatively weak performance on image-classification subtasks.The authors recommend an additional classification loss to avoid false negatives alongside large-batch contrastive loss.
  • Training strategy: The framework does not use hard-negative mining or curriculum learning, which the authors expect could improve discriminative capability and training stability.These strategies remain prospective enhancements rather than evaluated components of the reported system.
Loading 2602.13823v3…