PaperlayerView exact revision on arXiv

Source-linked AI summary

MMEB-V3: Measuring the Performance Gaps of Omni-Modality Embedding Models

Haohang Huang, Xuan Lu, Mingyi Su, Xuan Zhang, Ziyan Jiang, Ping Nie, Kai Zou, Tomas Pfister, Wenhu Chen, Wei Zhang, Xiaoyu Shen, Rui Meng

arXiv:2604.23321v2cs.IR

TL;DR

Existing benchmarks and methods provide limited coverage for systematically evaluating full-modality embeddings and instruction-conditioned retrieval. MMEB-V3 adds full-modality and agent-centric evaluation, with OmniSET enabling controlled analysis across semantically equivalent instances. Experiments find that models often fail to retrieve the instructed modality, with asymmetric, query-modality-biased behavior.

  • Problem

    Existing benchmarks provide limited full-modality coverage and systematic evaluation of instruction-following in complex retrieval scenarios.

    Source: §1 · pp. 1–2

  • Method

    MMEB-V3 evaluates embeddings across text, image, video, audio, and agent tasks, while OmniSET groups semantically equivalent instances across modalities for controlled analysis.

    Source: §1 · p. 2, §3.1 · p. 3

  • Results

    Models often fail to retrieve the instructed target modality, while cross-modal retrieval is asymmetric and dominated by query-modality bias.

    Source: §1 · p. 2, §6 · p. 10

  • Takeaways & Limitations

    Current multimodal embeddings do not reliably enforce instruction-specified modality constraints or exhibit consistent modality-aware retrieval behavior.

    Source: §1 · p. 2

    Benchmarks may inherit biases from their underlying public datasets; the paper does not explicitly address fairness.

    Source: Ethics Statement · p. 10

Abstract

from arXiv · show

Multimodal embedding models aim to map heterogeneous inputs, such as text, images, videos, and audio, into a shared semantic space. However, existing methods and benchmarks remain largely limited to partial modality coverage, making it difficult to systematically evaluate full-modality representation learning. In this work, we take a step toward the full-modality setting. We introduce MMEB-V3, a comprehensive benchmark that evaluates embeddings across text, image, video, audio, as well as agent-centric scenarios. To enable more fine-grained diagnosis, we further construct OmniSET (Omni-modality Semantic Equivalence Tuples), where semantically equivalent instances are represented across modalities, allowing us to disentangle semantic similarity from modality effects. Through experiments on MMEB-V3, we conduct a systematic analysis of full-modality embeddings and identify three key findings: (1) models often fail to retrieve the intended target modality; (2) cross-modal retrieval is highly asymmetric and dominated by query-modality bias; and (3) instruction-induced shifts are either insufficient or misaligned with the target modality, and therefore do not reliably improve retrieval. These results indicate that current multimodal embeddings are not yet capable of reliably enforcing modality constraints specified by instructions, and consequently fail to exhibit consistent modality-aware retrieval behavior. We hope MMEB-V3 provides a useful benchmark for understanding and diagnosing these limitations, and for guiding future research on full-modality embeddings.

1 Introduction

Multimodal embeddings support applications across heterogeneous inputs, but existing benchmarks offer limited coverage of full-modality retrieval and instruction-following. MMEB-V3 expands evaluation and uses OmniSET to diagnose modality-aware behavior, finding that models often fail to follow instructed modality constraints.

1 Introduction

  • Existing benchmarks provide limited support for evaluating full-modality coverage and instruction-following in complex retrieval scenarios.Earlier evaluations largely emphasize cross-modal alignment, such as text-to-image retrieval.Source: §1 · p. 2
  • In agent-centric settings, modality-mismatched retrieval can disrupt downstream actions such as tool invocation, GUI interaction, or memory access.The passage warns that such errors can lead to failures in the decision-making loop.Source: §1 · p. 2
  • MMEB-V3 evaluates instruction-conditioned embeddings across audio, complex text, and agent tasks, while OmniSET enables controlled analysis of semantic content versus modality effects.OmniSET groups semantically equivalent content across modalities into unified tuples.Source: §1 · p. 2
  • Experiments find that models often miss the instructed target modality, with retrieval dominated by the original query modality rather than the instructed target.The paper describes modality constraints as not reliably enforced as part of the instruction.Source: §1 · p. 2

2 Related Work

Prior multimodal benchmarks expanded from image–text alignment toward documents and video, while embedding models increasingly unify subsets or broader sets of modalities. Instruction-driven full-modality evaluation, especially for complex and agent tasks, remains underexplored.

2 Related Work

  • Early benchmarks established image–text retrieval and matching, while later work added document-level visual retrieval and temporal video retrieval.The passage describes early benchmarks as largely limited to static visual inputs and brief text descriptions.Source: §2 · p. 2
  • Existing benchmark coverage remains fragmented: MTEB lacks multimodal support, while MMEB-V2 and M-BEIR omit audio.Emerging tool, GUI, and memory retrieval tasks also remain outside comprehensive benchmarks.Source: §2 · pp. 2–3
  • MMEB-V3 extends benchmark coverage to audio, complex text retrieval, and agent tasks for more realistic instruction-conditioned evaluation.Complex settings such as multi-condition reasoning and tool, memory, and GUI retrieval are described as underexplored.Source: §2 · p. 3
  • Embedding models progressed from dual encoders toward systems unifying modality subsets, with newer work broadening coverage to text, image, audio, and video.The passage cites Omni-Embed-Nemotron as supporting those four modalities and WAVE as focusing on audio–video embeddings.Source: §2 · p. 3

3 MMEB-V3: A Unified Evaluation Framework for Omni-Modality Embeddings

MMEB-V3 expands evaluation across modalities and task types, and adds OmniSET for controlled analysis of semantic content and modality effects. It also characterizes instructions by their output-data constraints and the operations tasks require.

3.1 Modality Coverage and Task Diversity

  • MMEB-V3 extends MMEB-V2 with 111 new tasks, for 190 total across text, image, video, and audio, including cross-modal directions and varied task types.New coverage includes complex text retrieval and agent tasks alongside audio tasks.Source: §3.1 · p. 3, Figure 1 · p. 4
  • OmniSET groups semantically equivalent instances across modalities for controlled comparisons of semantic content and modality effects, outside leaderboard evaluation.Its tuples cover text, image, video, and audio instances.Source: §3.1 · pp. 3–4
  • Audio tasks span classification, cross-modal retrieval, and temporal grounding, while text tasks include instruction following, reasoning, long-context understanding, and multi-condition matching.The text suite contains 53 tasks, and agent tasks cover tool, GUI trajectory, and memory retrieval across 47 tasks.Source: §3.1 · pp. 3–4

3.2 Instruction Diversity across Data Types and Tasks

  • MMEB-V3 describes instruction diversity through expected output-data types, including modalities and structured data, and the operations associated with tasks.Those operations include classification, retrieval, grounding, and question answering.Source: §3.2 · pp. 4–5
  • The benchmark varies modality combinations across task types and instruction patterns across data requirements and task contexts.Figure 2 summarizes these distributions across 190 tasks.Source: Figure 2 · p. 5

4 Experiments

Across task categories, no evaluated model is optimal everywhere, and stronger performance in some areas often comes with weaker performance elsewhere. The experiments compare omni-modal and vision-language embedding baselines using task-appropriate metrics.

4.1 Experimental Settings

  • Evaluation uses Hit@1 for audio, image, video, and agent tasks, and NDCG@5 for text and VisDoc tasks.The metrics are selected to match task characteristics.Source: §4.1 · p. 5
  • The experiments compare omni-modal embedding models with vision-language embedding models.Table 1 describes MMEB-V3 as comprising 190 tasks, including 111 new tasks across audio, agent, and complex text retrieval categories.Source: §4.1 · p. 5, Table 1 · p. 6

4.2 Main Results

  • No single model achieves optimal performance across all task categories, and models strong on some tasks often perform more weakly on others.Fully multimodal models show relatively stable cross-modal performance but do not consistently outperform specialized models in specific categories.Source: §4.2 · pp. 5–6
  • Audio retrieval and complex text and agent tasks involving reasoning, multi-condition constraints, and long-context understanding are challenging across models.The authors suggest this may indicate limitations in capturing fine-grained semantic structure and implicit constraints.Source: §4.2 · p. 6
  • Observed trade-offs are associated with modality differences, task complexity, and instruction requirements.The paper points to unified and instruction-aware embeddings as an area for development.Source: §4.2 · p. 6

5 Analysis

The analysis uses OmniSET to examine three embedding models under modality-constrained retrieval. It finds frequent target-modality failures, asymmetric and query-modality-biased retrieval, and instruction-induced shifts that do not reliably align with targets.

5.1 Explicit Modality Instructions Often Fail in Cross-Modal Retrieval

  • Explicit modality instructions often fail to retrieve the intended target modality, with Hit@1 near zero in many cross-modal directions.The passage notes that some stronger directions may partly reflect OmniSET’s construction, since videos are generated from images and audio from text.Source: §5.1 · p. 7, Table 4 · p. 8
  • Nemotron’s I→V Hit@1 is 100.0 versus 5.0 for V→I, while retrieved modalities can track the query: Nemotron returns text for 84.9% of T2I results and WAVE returns video 99.9% across directions.The reverse-direction gap illustrates asymmetry; WAVE’s pattern shows that dominant retrieved modality need not follow the instructed target.Source: §5.1 · pp. 7–8

5.2 Model Sensitivity to Modality-Constrained Instructions

  • Mean instruction-induced shifts vary by model: about 0.4 cosine distance for Nemotron, 0.08 for WAVE, and 0.15 for Qwen3-VL-Embedding-8B.The authors suggest training emphasis may help explain WAVE’s weaker response, but note Nemotron’s instruction-related training details are not fully specified.Source: §5.2 · pp. 8–9

5.3 Instruction-Induced Shifts Do Not Consistently Move Queries Toward the Target

  • For Nemotron, only a few directions improve target distance, all by less than 0.09; T→V improves by +0.041 while V→T degrades by −0.158.The t-SNE views show noticeable shifts that are not consistently toward the target; image queries with text-target instructions may move closer to the video cluster.Source: §5.3 · p. 9, Figure 4 · p. 9

6 Conclusion

The paper presents MMEB-V3 and OmniSET for evaluating full-modality embeddings and diagnosing modality-constrained retrieval. Its analysis finds unreliable target-modality retrieval, directional bias, and instruction signals that do not reliably guide queries to desired modalities.

6 Conclusion

  • MMEB-V3 evaluates embeddings across text, image, video, audio, and agent-centric scenarios.The paper frames this as advancing evaluation toward the full-modality setting.Source: §6 · p. 10
  • OmniSET uses semantically equivalent instances across modalities to isolate modality effects under consistent semantics and evaluate instruction-constrained cross-modal behavior.Source: §6 · p. 10
  • The analysis finds that models often miss intended target modalities, retrieval is asymmetric and directionally biased, and instruction-conditioned signals do not reliably guide retrieval to desired modalities.The paper concludes that current multimodal embeddings remain limited in reliable modality-aware retrieval under instruction constraints.Source: §6 · p. 10

Ethics Statement

The ethics statement describes evaluation using public datasets, notes that inherited corpus biases are not explicitly addressed, and identifies potential dual-use risk.

Ethics Statement

  • The systems are developed and evaluated using publicly available datasets, with no private or personally identifiable data used.Source: Ethics Statement · p. 10
  • The benchmarks may inherit corpus biases that are not explicitly addressed for fairness, and those biases may affect downstream applications.Source: Ethics Statement · p. 10
  • The methods are intended for beneficial information access but may carry dual-use risks if misapplied; the authors encourage responsible use, careful curation, and transparency.Source: Ethics Statement · p. 10

A.1 Details of Baseline Models

The baselines span unified embedding models with differing modality coverage and retrieval settings, from audio-visual models to systems supporting text, images, video, and audio.

A.1 Details of Baseline Models

  • Omni-Embed-Nemotron is designed for retrieval over text, images, audio, and video, using Qwen2.5-Omni-3B’s Thinker component and a bi-encoder contrastive objective.It targets multimodal retrieval and RAG scenarios.Source: §A.1 · p. 13
  • WAVE learns shared audio-visual representations through joint multimodal training and hierarchical feature fusion for retrieval and multimodal understanding.The passage reports strong performance on audio-visual retrieval and multimodal question answering.Source: §A.1 · p. 13
  • Qwen3-VL-Embedding supports text, images, screenshots, videos, and mixed-modal inputs within a unified framework for cross-modal retrieval and understanding.It is built on the Qwen3-VL foundation model and uses dense embeddings.Source: §A.1 · p. 13
  • VLM2Vec-V2.0 supports images, videos, and visual documents in a shared embedding space, using unified training and contrastive embedding learning.The passage describes retrieval across diverse multimodal settings.Source: §A.1 · p. 13
  • VLM2Vec converts an instruction-tuned vision-language model into a unified multimodal embedding model for retrieval using embedding-oriented representations and contrastive learning.The passage reports strong performance across diverse multimodal retrieval tasks.Source: §A.1 · p. 14
  • GME supports text, images, and image-text pairs, mapping single- and combined-modal inputs into universal vector representations for Any2Any retrieval.Examples include text-to-image, image-to-image, and multimodal search.Source: §A.1 · p. 14

A.2 Details of Benchmark Construction

MMEB-V3 consolidates 190 tasks across text, image, video, audio, and agent-centric settings, while OmniSET enables controlled cross-modal retrieval evaluation.

A.2 Details of Benchmark Construction

  • MMEB-V3 consolidates 190 heterogeneous tasks into a ranking-based framework spanning text, image, video, and audio, with added complex text reasoning, agentic capabilities, and multi-condition retrieval.The image domain includes MCMR for fine-grained cross-modal matching under multiple constraints.Source: §A.2 · p. 14
  • Audio evaluation covers classification, cross-modal retrieval, and temporal grounding, with large audio datasets subsampled to up to 1,000 queries and candidate pools capped at 10,000.Classification tests acoustic-event and sound-category recognition; retrieval includes text-audio, audio-video, and audio-image alignment; grounding localizes events in continuous audio.Source: §A.2 · p. 14
  • Text evaluation adds instruction-following retrieval, reasoning retrieval, and long-context retrieval to address scenarios beyond standard text retrieval.The tasks test complex instructions and constraints, logical inference and multi-hop reasoning, and retrieval from long documents and extended contexts.Source: §A.2 · p. 14
  • The benchmark also includes multi-condition retrieval and a compact, diverse general text-retrieval subset.MultiConIR measures simultaneous satisfaction of multiple retrieval constraints; nanoBEIR is a curated subset of BEIR.Source: §A.2 · pp. 14–15
  • Agent-centric evaluation includes tool selection, GUI interaction, and agent-memory retrieval across representative memory types and multimodal interaction trajectories.The selected memory tasks cover episodic, dialogue, semantic, and procedural memory; Tool-REX contains over 43,000 tools.Source: §A.2 · p. 15
  • OmniSET represents the same semantic meaning as aligned text, image, video, and audio instances, requiring retrieval to match both meaning and the instruction-specified target modality.It contains 100 queries with human-verified hard negatives, expanded into 12 directional tasks for ordered input-target modality pairs.Source: §A.2 · p. 15

A.2.1 Construction of OmniSET

OmniSET supplies semantically aligned four-modality tuples for controlled retrieval comparisons, while the appendix documents possible synthetic-data biases and inconsistent instruction-induced shifts toward target modalities.

A.2.1 Construction of OmniSET

  • OmniSET extends MSCOCO image–text instances into text, image, video, and audio tuples, generating videos from images and speech from captions.The tuples are intended to hold semantics constant while varying modality.Source: §A.2.1 · p. 15
  • Hard negatives are selected from object-overlap candidates and manually screened; all 12 directed tasks share a mixed-modality pool, with source instances as distractors and same-modality tasks excluded.The source-instance distractor and exclusion are used to prevent trivial identity matching.Source: §A.2.1 · pp. 16–17

A.2.2 Impact of Synthetic Data and Potential Modality Bias

  • Because videos derive from images and audio from captions, I→V and A→T may inherit generation artifacts and data-driven modality preferences.These dependencies may partly explain unusually strong retrieval in those directions, rather than purely learned cross-modal alignment.Source: §A.2.2 · p. 17
  • Shared candidate pools and source-instance distractors mitigate construction confounds, while retrieval failures and misaligned shifts also occur in directions not directly affected by synthetic generation.The authors suggest human-annotated multimodal data or targeted ablations for further study.Source: §A.2.2 · p. 17

A.3 Detailed Scores

  • Per-dataset scores are reported for image, video, VisDoc, audio, text, and agent tasks.Source: §A.3 · p. 17

A.4 Visualization and Analysis

  • Across WAVE and Qwen3-VL-Embedding-8B, target distance usually stayed unchanged or increased after instruction; WAVE changed little while Qwen's gains were sparse.The comparison is shown for instruction-induced changes in query–target cosine distance.Source: §A.4 · p. 18, Figure 10 · p. 20
  • t-SNE shows Nemotron's modality-separated clusters, WAVE's audio–video entanglement, and Qwen3-VL-Embedding-8B's substantial modality overlap.The visualizations illustrate differences in how modality information is encoded across the three models.Source: §A.4 · p. 18, Figure 11 · p. 21
  • For video and audio queries, instruction augmentation perturbs representations but does not consistently move them toward the intended target modality.Shifts can approach neighboring modality clusters or directions unrelated to the target.Source: §A.4 · p. 19, Figure 12 · p. 22

Related papers

Loading 2604.23321v2…