MMEmb-R1: Reasoning-Enhanced Multimodal Embedding with Pair-Aware Selection and Adaptive Control

Yuchi Wang; Haiyang Yu; Weikang Bian; Jiefeng Long; Xiao Liang; Chao Feng; Hongsheng Li

Source-linked AI summary

MMEmb-R1: Reasoning-Enhanced Multimodal Embedding with Pair-Aware Selection and Adaptive Control

Yuchi Wang, Haiyang Yu, Weikang Bian, Jiefeng Long, Xiao Liang, Chao Feng, Hongsheng Li

arXiv:2604.06156v1Apr 7, 2026cs.CVcs.AIcs.CL

TL;DR

Multimodal embedding has not fully utilized generative reasoning, while directly adding chain-of-thought can misalign instance-level reasoning with pairwise supervision and cause overthinking. MMEmb-R1 models reasoning as a latent variable, selects paths using pair-aware counterfactual evaluation, and learns when to invoke reasoning through reinforcement learning. It reaches 71.2 on MMEB-V2 with 4B parameters and reduces inference latency relative to existing reasoning-enhanced methods.

Problem
Generative reasoning remains underutilized in multimodal embedding, and direct chain-of-thought integration faces structural misalignment with pairwise contrastive supervision and unnecessary reasoning for simple inputs.
Method
MMEmb-R1 models reasoning as a latent variable, uses pair-aware counterfactual selection for query-target alignment, and adaptively invokes reasoning through reinforcement learning.
Results
71.2 overall with Qwen3-VL-4B on MMEB-V2 establishes state-of-the-art performance while using fewer parameters and reducing inference latency by 2.5× versus UME-R1.
Takeaways & Limitations
Selective reasoning integrates generative capabilities into multimodal embeddings while avoiding the overhead of indiscriminate reasoning.
Takeaways & Limitations
The pipeline cannot jointly optimize offline reasoning generation, pair-aware selection, and two-stage training, and its binary reasoning policy may limit fine-grained control.

Abstract

from arXiv · show

MLLMs have been successfully applied to multimodal embedding tasks, yet their generative reasoning capabilities remain underutilized. Directly incorporating chain-of-thought reasoning into embedding learning introduces two fundamental challenges. First, structural misalignment between instance-level reasoning and pairwise contrastive supervision may lead to shortcut behavior, where the model merely learns the superficial format of reasoning. Second, reasoning is not universally beneficial for embedding tasks. Enforcing reasoning for all inputs may introduce unnecessary computation and latency, and can even obscure salient semantic signals for simple cases. To address these issues, we propose MMEmb-R1, an adaptive reasoning-based multimodal embedding framework. We formulate reasoning as a latent variable and introduce pair-aware reasoning selection that employs counterfactual intervention to identify reasoning paths beneficial for query-target alignment. Furthermore, we adopt reinforcement learning to selectively invoke reasoning only when necessary. Experiments on the MMEB-V2 benchmark demonstrate that our model achieves a score of 71.2 with only 4B parameters, establishing a new state-of-the-art while significantly reducing reasoning overhead and inference latency.

1 Introduction

MMEmb-R1 addresses the underuse of generative reasoning in multimodal embedding by aligning reasoning with pairwise contrastive supervision and invoking it adaptively. The framework targets shortcut behavior and overthinking while achieving strong MMEB-V2 results.

Existing multimodal embedding models use MLLMs mainly as static feature extractors, leaving their generative reasoning capabilities underutilized.Generative paradigms have improved abstraction, reasoning, and structured understanding in LLMs and MLLMs.
Reasoning creates structural misalignment because instance-level reasoning is trained alongside pairwise contrastive supervision, potentially inducing shortcut behavior.The paper identifies this mismatch as a central challenge for reasoning-enhanced embedding.
MMEmb-R1 treats reasoning paths as latent variables and uses pair-aware selection to identify reasoning beneficial for query-target alignment.The approach generates diverse candidates and uses counterfactual comparison of matching confidence with and without rationales to estimate marginal contribution.
Adaptive reasoning addresses overthinking by invoking reasoning selectively rather than enforcing it for every input.Figure 2 contrasts reasoning benefits for complex queries with irrelevant noise introduced for the simple target “cat.”
71.2 overall with a Qwen3-VL-4B backbone surpasses RzenEmbed-v1 (68.9) and Embed-RL (66.8) on MMEB-V2 while using fewer parameters.The 2B configuration attains 68.3 overall, and the adaptive mechanism reduces inference latency by 2.5× compared to UME-R1 with improved retrieval accuracy.

2 Related Works

Related work establishes multimodal embedding through contrastive alignment across heterogeneous modalities and increasingly explores generative reasoning and reinforcement learning. However, prior reasoning-driven embedding methods overlook the mismatch between instance-level reasoning and pair-level contrastive supervision.

Multimodal embedding research projects heterogeneous inputs into a shared semantic space, building on CLIP’s dual-tower contrastive image-text alignment.Subsequent work extends the paradigm to additional modalities.
Prior reasoning-oriented embedding approaches demonstrate reasoning’s potential but largely overlook structural misalignment between instance-level reasoning and pair-level contrastive supervision.This gap motivates MMEmb-R1’s pair-aware design.
Recent work uses chain-of-thought prompting and reinforcement learning, including GRPO-inspired methods, to enhance reasoning capabilities in LLMs and MLLMs.These developments provide the reasoning and optimization context for the paper’s approach.

3 Methodology

MMEmb-R1 combines pair-aware latent-reasoning selection, joint reasoning and contrastive embedding training, and reinforcement-learning-based control over whether reasoning is invoked. Its framework uses counterfactual query–target alignment signals to select useful rationales and continuous utility estimates to balance retrieval benefit against computational cost.

Framework Overview: MMEmb-R1 comprises reasoning-pool construction, joint reasoning and contrastive embedding training, and utility-aware reinforcement learning for adaptive reasoning control.The three stages are described as diverse candidate generation with counterfactual selection, joint training, and adaptive policy learning.
Architecture Overview: The model supports direct embeddings z_d = E(x) and reasoning-enhanced embeddings z_r = E(x ⊕r), where reasoning is generated before embedding extraction.Direct mode encodes the raw input, whereas reasoning mode conditions the embedding on a generated reasoning path.
Latent Reasoning and Selection: Reasoning is modeled as a latent variable sampled from a prior, with diverse multi-worker generation and pair-aware posterior selection used to approximate useful reasoning paths.The framework avoids deterministic marginalization by simulating the prior, scoring candidates against query–target alignment, and learning whether to sample reasoning.
Latent Reasoning and Selection: Counterfactual reasoning gain Δr = c_r − c_0 measures a rationale’s marginal improvement in matching confidence, and candidates with Δr > ε receive normalized selection weights.Confidence is compared with and without each rationale, so higher-gain paths contribute more strongly to subsequent training.
Joint Training: Joint training combines reasoning-enhanced contrastive learning, direct-path contrastive learning, and next-token prediction over curated chain-of-thought trajectories.The direct path preserves embedding quality without reasoning overhead, while the reasoning path and CoT objective cultivate reasoning-aware representations and generation.
Adaptive Reasoning: The adaptive policy selects DIRECT or REASON using continuous utility derived from direct-versus-reasoning similarity, while GRPO rewards retrieval improvement and penalizes reasoning cost.δ_i > 0 indicates improved retrieval from reasoning; the reward also accounts for rationale length, format, embedding quality, and reverse-direction symmetry.

4 Experiements

Experiments on MMEB-V2 show that MMEmb-R1 achieves strong performance across model sizes while adaptive reasoning improves the accuracy–latency trade-off. Analyses and ablations indicate that pair-aware selection and selective reasoning are important to the framework’s results.

Main Results: MMEmb-R1 achieves state-of-the-art performance in both small- and medium-size settings on MMEB-V2.
Main Results: 68.3 overall with Qwen3-VL-2B surpasses Embed-RL by +1.5 points and RzenEmbed-v1 by +3.9 points.
Main Results: 71.2 with Qwen3-VL-4B outperforms the best medium-size baseline RzenEmbed-v1-7B with nearly half the parameters.
Adaptive Reasoning: 185s inference latency for MMEmb-R1 Adaptive is a 1.8× speedup over always-reasoning and a 2.5× speedup over UME-R1, while delivering the highest accuracy.
Adaptive Reasoning: Accuracy peaks at a 74.3% reasoning invocation ratio, then declines to 61.9 under near-universal reasoning, indicating overthinking.
Ablation Studies: Removing reasoning causes a 5.8-point drop, while always-reasoning scores 1.4 points below the full model.

5 Conclusion

MMEmb-R1 integrates generative reasoning into multimodal embedding learning through latent-variable reasoning and pair-aware selection. Its adaptive reinforcement-learning stage selectively invokes reasoning, achieving state-of-the-art performance with reduced inference latency.

MMEmb-R1 integrates generative reasoning into multimodal embedding learning by treating reasoning paths as latent variables and selecting them pair-awarely.
Utility-aware reinforcement learning trains MMEmb-R1 to invoke reasoning selectively, addressing overhead from indiscriminate reasoning.
Experiments on MMEB-V2 demonstrate state-of-the-art performance while substantially reducing inference latency compared with existing reasoning-enhanced methods.

Limitations

The framework has limitations in optimization, policy granularity, and inference cost that motivate future investigation.

The pipeline’s offline generation, pair-aware selection, and two-stage training prevent joint optimization of its components.
Binary reasoning-invocation decisions may be suboptimal compared with policies controlling reasoning depth or granularity.
Reasoning-enhanced embeddings incur additional inference cost, and fundamentally reducing latency remains an open challenge.

A.1 Qualitative Analysis

Qualitative cases show that MMEmb-R1 adapts reasoning to query complexity and selects rationales that best align queries with targets. It skips reasoning for unambiguous visual inputs, invokes it for temporal inference, and weights semantically bridging rationales most highly.

Adaptive reasoning: MMEmb-R1 skips reasoning for an unambiguous penguin query, avoiding spurious alternatives that cause UME-R1 to retrieve the wrong target.For a cooking video requiring temporal inference, it instead invokes reasoning and identifies seasoning as the logical step after stir-frying.
Adaptive reasoning: MMEmb-R1 invokes reasoning for a temporal cooking query and decomposes the sequence to retrieve the correct next action.The non-reasoning VLM2Vec-V2 captures only the coarse cooking concept and retrieves a temporally preceding action.
Pair-aware selection: The Proprietary worker receives the highest weight, w = 0.55, because its rationale directly mirrors the target semantics.It reframes 24% as about one in four respondents and associates 9% with the most-or-all-of-the-time frequency band.
Pair-aware selection: The pair-aware evaluator favors rationales that bridge the query–target semantic gap rather than those that are merely more elaborate.The qualitative analyses are presented as main reasoning traces produced by the model.

A.2 Scaling Behavior Across Backbones

MMEmb-R1 scales monotonically within backbone families while newer architectures deliver higher performance at smaller parameter counts. Counterfactual gains vary across workers, motivating filtering and relative weighting of reasoning candidates.

Backbone scaling: Performance improves monotonically with model size across Qwen2-VL, Qwen2.5-VL, and Qwen3-VL backbone families.The framework is evaluated across six backbones spanning three families and multiple parameter scales.
Backbone scaling: Qwen3-VL-2B surpasses Qwen2-VL-7B, while Qwen3-VL-4B outperforms Qwen2.5-VL-7B at roughly half the size.The passage attributes these gains to both stronger representations and larger capacity.
Counterfactual gains: The Proprietary worker has the highest median counterfactual gain, whereas the Thinking worker has the widest, bimodal distribution with negative values.Thinking-worker reasoning is described as occasionally exceptional but frequently noisy.
Counterfactual gains: ϵ = −0.1 retains candidates with only small performance drops, while the Thinking worker produces the largest number of filtered samples.Accepted samples are additionally differentiated by pair-aware alignment scores.
Reasoning utility: Roughly 60% of instances have positive reasoning utility, while 40% have negative utility after pair-aware selection.The utility compares normalized similarity from reasoning-enhanced and direct embeddings.

A.5 Detailed Results on MMEB-V2 and MMEB-V1

MMEmb-R1 is evaluated across MMEB-V2 and MMEB-V1, with the framework extending beyond the newly added video and document modalities. Its reasoning prior combines heterogeneous workers with distinct generation styles.

Benchmark results: 74.8 overall on MMEB-V1 is achieved by MMEmb-R1 with Qwen3-VL-4B, outperforming all listed baselines including Embed-RL-4B and UME-R1-7B.This result supports evaluation beyond the video and document modalities introduced in MMEB-V2.
Reasoning prior: MMEmb-R1 approximates the reasoning latent-space distribution by combining heterogeneous MLLM workers with complementary reasoning styles and knowledge coverage.The worker pool is designed to provide a richer prior than any single model.
Worker diversity: The Instruct worker produces consistently formatted retrieval-oriented rationales, while the Thinking worker generates longer, more exploratory chains with higher variance.The Proprietary worker generates richer rationales and has the highest median counterfactual gain.

B.2 Pair-Aware Evaluator Implementation

The pair-aware evaluator compares query–target relevance with and without generated rationales. It converts the resulting confidence difference into counterfactual gains used to assess whether each rationale improves retrieval effectiveness.

Evaluator inputs: The evaluator receives a query item, target item, and their associated images or videos to judge semantic relevance for retrieval.Its baseline pass evaluates relevance using the raw query and target without reasoning.
Counterfactual comparison: The with-rationale pass adds generated query and target rationales to test whether they improve retrieval effectiveness.The evaluator considers whether the rationales capture essential semantic information for matching the two items.
Counterfactual gain: ∆r = diffwith − diffbaseline measures the rationale’s marginal contribution to evaluator confidence beyond the raw inputs.A positive ∆r indicates that the rationale improves confidence in the query–target match.

B.3 Details of Joint Reasoning and Embedding Training

Joint training combines direct and reasoning-enhanced embedding paths, while adaptive control uses reinforcement learning to select reasoning and reward retrieval-discriminative outputs. The training setup also incorporates symmetric embedding rewards and benchmark evaluation tables.

<d_emb> marks direct embedding extraction, while <r_emb> marks extraction after optional reasoning tokens.
Direct-embedding samples are included during joint training to reduce reliance on reasoning generation and support subsequent policy learning.Samples are selected when pair-aware reasoning weights are very low, indicating generated rationales provide little benefit.
The embedding reward evaluates whether positive targets outrank negative targets and the magnitude of their similarity gap, averaged across query-to-target directions.This reward is designed to favor reasoning trajectories that produce more discriminative embeddings.
GRPO updates the adaptive reasoning policy using group-normalized advantages, a clipped objective, and KL regularization without a separate critic network.The related-work description identifies lower memory use, stable gradients from group normalization, and compatibility with standard language-model training infrastructure.
Results are organized in tables for the full MMEB v2 benchmark and the 36-task MMEB-V1 image-embedding benchmark, including in-distribution and out-of-distribution results.Table 7 reports detailed baseline and MMEmb-R1 results; Table 8 distinguishes IND and OOD datasets.