Source-linked AI summary
VLM2Vec-V2: Advancing Multimodal Embedding for Videos, Images, and Visual Documents
Rui Meng, Ziyan Jiang, Ye Liu, Mingyi Su, Xinyi Yang, Yuepeng Fu, Can Qin, Zeyuan Chen, Ran Xu, Caiming Xiong, Yingbo Zhou, Wenhu Chen, Semih Yavuz
TL;DR
Existing multimodal embeddings and benchmarks focus mainly on natural images, with limited support for videos and visual documents. The paper introduces MMEB-V2 and trains VLM2Vec-V2 as a unified embedding model across these modalities, achieving stronger performance than prior baselines across the expanded evaluation.
Problem
Existing multimodal embedding models and datasets predominantly focus on natural images, leaving videos and visual documents insufficiently covered.
Method
The paper introduces MMEB-V2 and trains VLM2Vec-V2 with instruction-following and contrastive learning across diverse modalities and task categories.
Results
VLM2Vec-V2 outperforms prior baselines across 78 datasets and achieves the highest overall average score of 58.0 across image, video, and visual document tasks.
Takeaways & Limitations
MMEB-V2 provides a broader evaluation framework, while VLM2Vec-V2 demonstrates strong and balanced multimodal embedding performance across diverse tasks.
Abstract
from arXiv · showhide
Multimodal embedding models have been crucial in enabling various downstream tasks such as semantic similarity, information retrieval, and clustering over different modalities. However, existing multimodal embeddings like VLM2Vec, E5-V, GME are predominantly focused on natural images, with limited support for other visual forms such as videos and visual documents. This restricts their applicability in real-world scenarios, including AI agents, multi-modal search and recommendation, and retrieval-augmented generation (RAG). To close this gap, we propose VLM2Vec-V2, a unified framework for learning embeddings across diverse visual forms. First, we introduce MMEB-V2, a comprehensive benchmark that extends MMEB with five new task types: visual document retrieval, video retrieval, temporal grounding, video classification and video question answering - spanning text, image, video, and visual document inputs. Next, we train VLM2Vec-V2, a general-purpose embedding model that supports text, image, video, and visual document inputs. Extensive experiments show that VLM2Vec-V2 achieves strong performance not only on the newly introduced video and document retrieval tasks, but also improves over prior baselines on the original image benchmarks. Through extensive evaluation, our study offers insights into the generalizability of various multimodal embedding models and highlights effective strategies for unified embedding learning, laying the groundwork for more scalable and adaptable representation learning in both research and real-world settings.
1 Introduction
Existing multimodal embedding benchmarks and models primarily target natural images, leaving videos and visual documents underrepresented. The paper introduces MMEB-V2 and VLM2Vec-V2 to evaluate and embed diverse visual modalities across tasks.
- Existing benchmarks focus predominantly on natural images and photographs, with limited coverage of documents, websites, slides, and videos.
- MMEB-V2 extends the original benchmark with five tasks covering video retrieval, moment retrieval, video classification, video question answering, and visual document retrieval.
- The benchmark and model are used to study cross-modal generalization, training strategies, and challenges in representing temporal video and structured document information.
- VLM2Vec-V2 is a unified embedding model supporting diverse input formats and producing general-purpose embeddings for downstream tasks.
- VLM2Vec-V2 outperforms prior baselines across 78 datasets and supports the paper’s investigation of unified multimodal embedding learning.
2 MMEB-V2: Expanding Embedding Benchmarks Beyond Image-Text
MMEB-V2 expands multimodal embedding evaluation beyond image-text tasks to cover videos and visual documents. Its tasks use multimodal queries and candidate selection across static, temporal, and structured visual data.
- MMEB-V2 evaluates combinations of text, images, videos, and visual documents, adding four video understanding tasks and one visual document task.
- Each task presents a query and candidate responses, asking the model to select the correct target for retrieval, classification, or understanding.
- Video inputs are represented as frames sampled at uniform intervals to provide consistent temporal coverage.
- The benchmark contains 9 meta-tasks and 78 tasks in total, with newly introduced video and visual-document meta-tasks distinguished from original MMEB tasks.
- The benchmark includes video retrieval, moment retrieval, and video classification among its newly introduced task types.
3 Unified Embedding Model for Video, Image, and Visual Document
The model learns a shared embedding space for heterogeneous visual modalities through standardized multimodal formatting, instruction conditioning, contrastive learning, and data-source balancing.
- VLM2Vec-V2 aims to align diverse modalities in a shared embedding space while using natural-language instructions to guide task-specific behavior.
- The model uses a shared vision-language backbone capable of encoding interleaved text, images, and videos, including long-form videos and multi-page documents.
- Contrastive training obtains query and target embeddings from a pretrained vision-language model and applies InfoNCE over in-batch and hard negatives.
- Query-target pairs are standardized across modalities, and task instructions are applied to queries to improve contextualization and cross-task generalization.
- The sampling pipeline mixes datasets using sampling weights and interleaved sub-batches to balance sources and increase contrastive-learning difficulty.
4 Experiments
Experiments evaluate VLM2Vec-V2 across image, video, and visual-document tasks using diverse baselines and controlled training ablations. The model achieves the strongest overall benchmark result, while modality-specific trends reveal effects of data mixing, LoRA capacity, and training duration.
- Experimental Setup: The experiments combine video-language, visual-document retrieval, and image-text instruction data to train unified multimodal embeddings.The training uses video-caption and video-QA data, ViDoRe and VisRAG document data, and MMEB image-text tasks.
- Main Results: VLM2Vec-V2 achieves the highest overall average score, 58.0, across 78 image, video, and visual-document datasets.It outperforms strong baselines including GME, LamRA, and VLM2Vec, including models using the same Qwen2-VL backbone.
- Main Results: VLM2Vec-V2 performs strongly on image tasks, competitively on video tasks, and better than all VLM2Vec variants on visual-document retrieval.Its visual-document performance still trails ColPali, which is specifically optimized for visual-document tasks.
- Ablation Study on Data Sampling Strategies: Increasing interleaved sub-batch size consistently improves VisDoc and Video performance, whereas Image performance peaks at size 64 before declining.An interleaved sub-batch size of 64 divides a batch of 1,024 into data from 16 distinct sources.
- Ablation Study on Model Settings: LoRA rank 16 yields the best overall performance across image, video, and visual-document tasks, while rank 32 adds no further gains.This result identifies moderate parameter-efficient tuning capacity as the strongest tested setting.
- Ablation Study on Model Settings: All three modalities improve with more training steps, with no clear saturation by 5K steps, especially for VisDoc and Video.Long-horizon training and convergence behavior remain for future investigation.
5 Related Works
Related work has advanced multimodal embeddings through image-text benchmarks, video understanding, visual-document retrieval, and unified modality retrieval. However, existing approaches do not unify image, video, and visual-document retrieval within one framework.
- Multimodal Benchmarks and Embeddings: Early multimodal benchmarks primarily used static image-text pairs, while newer M-BEIR and MMEB introduced multitask embedding evaluations.These benchmarks supported progress in image captioning, retrieval, classification, and related tasks.
- Video Representation Learning: Existing video benchmarks target retrieval and captioning but lack unified evaluation frameworks for multimodal embeddings.MMEB-V2 addresses this gap by integrating instruction-following tasks across videos and structured documents.
- Video Representation Learning: Image-text vision-language models often struggle with video temporal dynamics, motivating architectures and methods specialized for video understanding.Some image-text-trained models nevertheless generalize to text-video retrieval in zero-shot settings.
- Visual Document Representation Learning: Visual-document retrieval methods combine visual and textual features because traditional text models miss documents’ visual and structural information.ColPali and VisRAG exemplify vision-language approaches for document retrieval and image-based RAG.
- Unified Modality Retrieval: Unified modality retrieval methods support multiple data types or dynamically route queries, but they do not jointly unify image, video, and visual-document retrieval.VLM2Vec-V2 is presented as addressing this specific unification gap.
6 Conclusion
The paper introduces MMEB-V2 as a broad benchmark and VLM2Vec-V2 as a contrastively trained multimodal embedding baseline. Experiments demonstrate their effectiveness across diverse modalities and tasks.
- Conclusion: MMEB-V2 evaluates multimodal embedding models across text, image, video, and visual-document modalities.VLM2Vec-V2 is trained through contrastive learning across diverse tasks and modality combinations.
- Conclusion: The experiments demonstrate VLM2Vec-V2’s effectiveness and MMEB-V2’s diagnostic value for multimodal embedding evaluation.
A.1 Author Contributions
The project involved shared leadership across research, benchmark construction, infrastructure, modeling, evaluation, and maintenance. Specific contributors led video and visual-document data work alongside model training and scoring.
- Project and Research Leadership: Semih Yavuz, Wenhu Chen, Yingbo Zhou, Caiming Xiong, Ran Xu, and Zeyuan Chen provided overall project leadership and research guidance.
- Project and Research Leadership: Rui and Ziyan co-drove technical direction, model development, and MMEB-v2 creation, with Semih managing project progress.
- Codebase and Infrastructure: Rui led the codebase and training infrastructure, while Ziyan refactored evaluation and Xinyi contributed to visual-document evaluation.
- Benchmark and Data Curation: MMEB-v2 benchmark construction was distributed across video tasks, visual-document curation, dataset contributions, parsing, and evaluation logic.Named contributors handled video retrieval, classification, moment retrieval, video QA, visual-document datasets, and corresponding parsers.
- Modeling and Experiments: Ye conducted most model-training experiments, Can ran overall evaluations, and multiple authors jointly collected baseline results.
- Maintenance: The team committed to long-term maintenance of the leaderboard and code package, with Mingyi responsible for leaderboard maintenance.
A.2 Details of Baseline Models
The baselines unify multimodal retrieval or embedding through different model architectures and training strategies. They span instruction-following ranking, document multi-vector matching, two-stage multimodal retrieval training, and contrastive learning over multimodal pairs.
- VLM2Vec: VLM2Vec reformulates multimodal tasks as instruction-following ranking problems and learns fixed-dimensional, cross-modal embeddings with contrastive learning.Its task-specific instructions are intended to align representations across modalities.
- ColPali: ColPali generates multi-vector embeddings from document page images and applies late-interaction matching for visual document retrieval.The model uses a vision-language model as its embedding backbone.
- LamRA: LamRA unifies diverse retrieval tasks without task-specific fine-tuning through language-only pretraining followed by multimodal instruction tuning.Its two-stage training procedure is designed to improve retrieval effectiveness.
- GME: GME is fine-tuned from Qwen2-VL and supports retrieval across single-modal, cross-modal, and fused-modal settings.It uses contrastive learning on text, image, and image-text pairs.
A.3.1 Video Retrieval
The video-retrieval evaluation draws from diverse datasets covering caption retrieval, temporal segments, actions, and video question answering. It also standardizes several evaluation choices, including candidate ranking and selected benchmark subsets.
- Video Retrieval: MSR-VTT, MSVD, and VATEX provide caption-based video retrieval cases from open-domain, YouTube, and Kinetics-derived videos.The benchmark uses 1,000 MSR-VTT pairs, 670 MSVD test cases, and 4,468 accessible VATEX videos.
- Moment Retrieval: DiDeMo, YouCook2, Charades-STA, and MomentSeeker evaluate retrieval involving temporally localized or long-video moments.Their constructions use paragraph-to-video retrieval, temporal annotations, filtered candidate clips, or sampled negative clips.
- Moment Retrieval: QVHighlights evaluates query-based moment retrieval by ranking candidate clips with Recall@1 rather than the IoU-thresholded Recall@0.5 and Recall@0.7 used in related evaluations.The benchmark text describes IoU-thresholded evaluation as less suited to embedding-based approaches.
- Video Classification: Kinetics-700-2020, Something Something v2, HMDB51, Breakfast, and UCF101 supply video-classification cases with candidate action labels.The datasets cover broad human actions, object interactions, movies and web videos, breakfast activities, and open-domain actions.
- Video QA: MVBench, Video-MME, NExT-QA, and EgoSchema provide multiple-choice video question answering cases spanning temporal, causal, long-form, and egocentric understanding.The study uses selected subsets or configurations, including 500 publicly annotated EgoSchema questions and the multiple-choice portion of NExT-QA.