PaperlayerView exact revision on arXiv

Source-linked AI summary

VLM2Vec: Training Vision-Language Models for Massive Multimodal Embedding Tasks

Ziyan Jiang, Rui Meng, Xinyi Yang, Semih Yavuz, Yingbo Zhou, Wenhu Chen

arXiv:2410.05160v3cs.CVcs.AIcs.CL

TL;DR

Universal multimodal embeddings lack broad benchmarks and methods despite their importance for cross-task applications. The paper introduces MMEB and trains VLM2VEC, an instruction-following contrastive embedding framework built on vision-language models. VLM2VEC improves substantially over existing multimodal embedding baselines across in-distribution and out-of-distribution MMEB datasets.

  • Problem

    Multimodal embedding research lacks broad benchmarks and methodologies, while existing models have limited cross-modal integration and zero-shot generalization.

  • Method

    The paper introduces MMEB and trains VLM2VEC by applying task instructions and contrastive query-target learning to pretrained vision-language models.

  • Results

    VLM2VEC improves average precision@1 from 44.7 to 62.9 across all 36 MMEB datasets and from 41.7 to 57.1 on 16 out-of-distribution datasets against the best non-fine-tuned baseline.

  • Takeaways & Limitations

    Vision-language models can serve as strong general multimodal embedding models across diverse tasks and unseen datasets.

Abstract

from arXiv · show

Embedding models have been crucial in enabling various downstream tasks such as semantic similarity, information retrieval, and clustering. Recently, there has been a surge of interest in developing universal text embedding models that can generalize across tasks (e.g., MTEB). However, progress in learning universal multimodal embedding models has been relatively slow despite its importance and practicality. In this work, we aim to explore the potential for building universal embeddings capable of handling a wide range of downstream tasks. Our contributions are twofold: (1) MMEB (Massive Multimodal Embedding Benchmark), which covers 4 meta-tasks (i.e. classification, visual question answering, multimodal retrieval, and visual grounding) and 36 datasets, including 20 training and 16 evaluation datasets covering both in-distribution and out-of-distribution tasks, and (2) VLM2Vec (Vision-Language Model -> Vector), a contrastive training framework that converts any state-of-the-art vision-language model into an embedding model via training on MMEB. Unlike previous models such as CLIP and BLIP, which encodes text or images independently without any task instruction, VLM2Vec can process any combination of images and text to generate a fixed-dimensional vector based on task instructions. We build a series of VLM2Vec models on SoTA VLMs like Phi-3.5-V, LLaVA-1.6 and evaluate them on MMEB's evaluation split. Our results show that VLM2Vec achieves an absolute average improvement of 10% to 20% over existing multimodal embedding models on both in-distribution and out-of-distribution datasets in MMEB. We show that VLMs are secretly strong embedding models.

1 INTRODUCTION

The paper addresses missing benchmarks and methods for universal multimodal embeddings by introducing MMEB and VLM2VEC. Together, they support instruction-following embeddings across modality combinations and show substantial gains across multimodal tasks.

  • Multimodal embedding research lacks both broad benchmarks and established methodologies.
  • Existing models often isolate image and text processing or use shallow fusion, limiting cross-modal relationships, reasoning, and zero-shot generalization.
  • MMEB contains 36 datasets spanning classification, visual question answering, retrieval, and visual grounding, with 20 for training and 16 for evaluation.
  • VLM2VEC adapts pretrained vision-language models through deep visual-textual integration and supports arbitrary combinations of images and text.
  • 18.2 points separate VLM2VEC from the best non-fine-tuned baseline across all 36 MMEB datasets, increasing performance from 44.7 to 62.9.
  • 15.4 points separate VLM2VEC from the best non-fine-tuned baseline on 16 out-of-distribution datasets in zero-shot evaluation, increasing performance from 41.7 to 57.1.

2 MMEB: A BENCHMARK FOR MULTIMODAL EMBEDDINGS

MMEB is a broad multimodal embedding benchmark that unifies diverse domains, modality combinations, and instructions under ranking-based evaluation. It organizes 36 datasets into four meta-tasks with separate training and out-of-distribution evaluation splits.

  • MMEB comprises 36 datasets across classification, visual question answering, retrieval, and visual grounding, divided into 20 training and 16 out-of-distribution evaluation datasets.
  • All tasks are reformulated as ranking problems in which an instructed model selects the correct target from candidates for a multimodal query.
  • The benchmark covers common, news, Wikipedia, web, and fashion domains with text, image, and image-text queries and targets.
  • Classification: Classification uses an instructed image query, optionally with related text, and treats class labels as candidate targets.
  • Visual Question Answering: Visual question answering combines an instruction, image, and textual question, then ranks one answer against 999 distractors.
  • Visual Grounding: Visual grounding ranks cropped image regions, including one ground-truth box and 999 distractors that may be hard negatives or unrelated objects.

3 VL M2VE C: TRANSFORMING LVMS TO EMBEDDERS

VLM2VEC converts pretrained vision-language models into instruction-following embedding models using contrastive query-target training. Its design supports heterogeneous multimodal inputs and uses large effective batches to improve learning.

  • VLM2VEC converts state-of-the-art vision-language models into embedding models through contrastive training.
  • The framework accepts queries and targets that are images, text, or image-text pairs.
  • Task-specific instructions are prepended to queries so the model can represent the requested embedding task and improve generalizability.
  • Query and target embeddings are extracted from the last-token representation of a pretrained VLM, then optimized with InfoNCE over in-batch and hard negatives.
  • Larger batches increase in-batch random negatives, but multimodal inputs create GPU-memory bottlenecks during training.
  • GradCache decouples contrastive-loss backpropagation from encoder computation, enabling gradient accumulation across memory-fitting sub-batches.

4 EXPERIMENTS

The experiments evaluate VLM2VEC across MMEB, compare baseline and fine-tuned systems, and ablate fine-tuning, training parameters, meta-task specialization, and instructions. VLM2VEC improves broadly across task categories and transfers to unseen tasks.

  • 4.2 MAIN RESULT: 62.9% average Precision@1 is achieved across all 36 MMEB datasets, while 57.1% is achieved on 16 out-of-distribution tasks in zero-shot evaluation.The best variant uses LLaVA-1.6, LoRA, and 1344 × 1344 input images.
  • 4.2 MAIN RESULT: 18.2 points separate VLM2VEC from the best non-fine-tuned baseline across all 36 MMEB datasets, increasing performance from 44.7 to 62.9.Against the best fine-tuned baseline, the improvement is 15.7 points, from 47.2 to 62.9.
  • 4.2 MAIN RESULT: At least 50% performance is maintained across all four MMEB meta-task categories, unlike the baseline models.The comparison includes baselines with and without MMEB fine-tuning and VLM2VEC models using LLaVA-1.6 or Phi-3.5 backbones.
  • 4.3.1 FULL FINE-TUNING VS. LORA: LoRA outperforms full fine-tuning when its rank is appropriately configured under consistent training and data setups.Table 3 compares fully fine-tuned VLM2VEC with LoRA variants at different ranks using Phi-3.5-V.
  • 4.3.2 TRAINING PARAMETERS: Performance gradually improves with larger batch sizes, more sub-image crops, and more training steps.Large batches are especially important because random negatives compensate for the lack of hard negatives, with GradCache supporting the setup.
  • 4.3.3 META-TASK GENERALIZATION: Retrieval-specialized VLM2VEC generalizes better to unseen meta-tasks than classification- or VQA-specialized models.Retrieval training combines more diverse text and visual modalities on both query and target sides.
  • 4.3.4 IMPACT OF INSTRUCTIONS: Task-specific instructions improve VLM2VEC performance by 49.4% but reduce CLIP performance by 29.4%.The comparison evaluates both models with and without task-specific instructions.

5 RELATED WORK

Related work progresses from universal text and image embeddings toward multimodal systems, but multimodal evaluation and representation integration remain comparatively limited. Existing benchmarks and models use varied task coverage and fusion strategies.

  • Text Embeddings: Universal text embedding research expanded from task-specific models toward broad task coverage using large paired datasets and natural-language prompts.Examples include GTR, E5, TART, and InstructOR.
  • Multimodal Embeddings: Early universal multimodal models such as CLIP, BLIP, Align, SigLIP, SimVLM, and CoCa generally encode images and text separately into a shared space.These approaches learn representations from large-scale, weakly supervised image-text pairs.
  • Multimodal Embeddings: Many multimodal embedding systems combine visual and language information through shallow fusion, including feature addition in UniIR and shallow self-attention in MagicLens.E5-V instead trains on text pairs, whereas VLM2VEC uses multimodal pairs with varied image-text combinations.
  • Retrieval Benchmarks: Retrieval benchmarks broadened from MS MARCO and Natural Questions to BEIR and MTEB, which cover more diverse domains and tasks.The cited progression concerns text retrieval evaluation.
  • Retrieval Benchmarks: MBEIR evaluates multimodal retrieval across 8 tasks and 16 datasets using varied query modalities and instructions.It provides a prior multimodal retrieval benchmark for cross-modal evaluation.

6 CONCLUSION

The paper presents MMEB and VLM2Vec as a large-scale multimodal embedding framework, using diverse tasks and vision-language backbones to improve generalization through instruction following.

  • MMEB includes 36 datasets across classification, visual question answering, retrieval, and visual grounding.
  • VLM2Vec uses vision-language models as backbones to deeply fuse visual and textual spaces.
  • Instruction following enhances generalization to unseen tasks.

A DETAILS OF MMEB

This appendix section directs readers to MMEB’s dataset specifications and its procedure for determining candidate-set sizes.

  • MMEB’s 36 datasets are detailed in Section A.1.
  • Section A.1 provides the specifics of the datasets used in MMEB.
  • Section A.2 explains how MMEB determines the number of candidates.

A.1.1 CLASSIFICATION

MMEB’s classification portion contains 10 datasets spanning object recognition, geolocation, scene recognition, multimodal news and meme classification, and distribution-shifted imagery.

  • 10 datasets are included for classification tasks.
  • ImageNet-A and ImageNet-R assess classification under difficult or stylistically varied image distributions.ImageNet-A contains harder examples from ImageNet classes, while ImageNet-R includes artistic and other renditions.
  • The classification datasets also cover realistic objects, multimodal news and hate-speech memes, and broad scene-recognition categories.VOC2007 covers 20 object classes; N24News combines article text and images across 24 categories; HatefulMemes targets multimodal hate speech; Place365 and SUN397 cover scenes.
  • ObjectNet tests recognition on 50K images with unusual poses, cluttered scenes, and controls for rotation, background, and viewpoint.
  • Country-211 evaluates geolocation using photos from 211 countries with GPS coordinates.

A.1.2 VISUAL QUESTION ANSWERING (VQA)

MMEB’s VQA portion contains 10 datasets covering visual reasoning over text, scenes, external knowledge, documents, infographics, charts, science, and natural images.

  • 10 datasets are included for visual question answering tasks.
  • OK-VQA and A-OKVQA require external resources, commonsense, or broad world knowledge to answer questions.
  • DocVQA, InfographicsVQA, and ChartQA evaluate reasoning over documents, layouts, graphical elements, data visualizations, and charts.
  • ScienceQA includes diverse science questions with answers linked to lectures and explanations, while VizWiz uses answerable questions from blind users’ images.
  • Visual7W-telling links textual descriptions with image regions through object-level grounding and includes telling and pointing questions.
  • TextVQA and GQA test reasoning over text in images, scene graphs, object attributes, and relationships.

A.1.3 RETRIEVAL

The retrieval portion repurposes diverse multimodal datasets into retrieval settings spanning image, text, dialogue, questions, and human similarity judgments. These tasks use different query-target constructions, including composed retrieval, cross-modal search, and Wikipedia-based answering.

  • Retrieval datasets: Retrieval includes 12 datasets covering cross-modal search, composed image retrieval, caption retrieval, multimodal question answering, and image similarity.The datasets include EDIS, CIRR, FashionIQ, VisualNews, MSCOCO, WebQA, NIGHTS, OVEN, and VisDial, among others.
  • Retrieval datasets: VisDial is reformulated to retrieve an image from a 10-round dialogue whose answerer had access to that image.The dialogue is produced by a questioner viewing only the image description and an answerer viewing the image.
  • Retrieval datasets: CIRR and FashionIQ perform composed image retrieval using a reference image, a modification sentence, and a target image.Both datasets describe changes between reference and target images, with FashionIQ focusing on fashion products.
  • Retrieval datasets: VisualNews and MSCOCO each support both image-to-text and text-to-image retrieval setups.The first retrieves captions from images, while the second retrieves images from captions.
  • Retrieval datasets: WebQA, OVEN, and EDIS require retrieval over multimodal candidates or news content to connect questions, entities, events, images, and text.WebQA retrieves Wikipedia page images and descriptions, OVEN uses related Wikipedia image-text references, and EDIS pairs news images with headlines.
  • Retrieval datasets: NIGHTS converts human judgments over perturbed image pairs into retrieval, using the reference image as query and the judged-more-similar version as target.The original triplets contain a reference, two perturbations, and a human similarity choice.
Loading 2410.05160v3…