PaperlayerView exact revision on arXiv

Source-linked AI summary

MM-Embed: Universal Multimodal Retrieval with Multimodal LLMs

Sheng-Chieh Lin, Chankyu Lee, Mohammad Shoeybi, Jimmy Lin, Bryan Catanzaro, Wei Ping

arXiv:2411.02571v2cs.CLcs.AIcs.CVcs.IRcs.LG

TL;DR

Existing retrievers typically target fixed tasks and single modalities, limiting broader multimodal search. The paper develops MLLM-based universal multimodal retrievers, modality-aware training, continuous fine-tuning, and zero-shot reranking, with MM-Embed achieving state-of-the-art multimodal retrieval and stronger text retrieval than NV-Embed-v1.

  • Problem

    Existing retrieval models focus on fixed tasks and single-modality queries and results, motivating retrieval across diverse tasks and multimodal inputs.

  • Method

    The paper fine-tunes MLLM bi-encoder retrievers with modality-aware hard negatives and continuous text-retrieval fine-tuning, and prompts MLLMs as zero-shot rerankers.

  • Results

    MM-Embed achieves state-of-the-art performance on M-BEIR while surpassing NV-Embed-v1 on MTEB retrieval tasks; zero-shot reranking further improves difficult multimodal retrieval.

  • Takeaways & Limitations

    MLLMs can support universal multimodal retrieval, while reranking remains useful for especially complex interleaved text-image queries.

  • Takeaways & Limitations

    The retrieval formulation considers text and image only, leaving audio and video for future work.

Abstract

from arXiv · show

State-of-the-art retrieval models typically address a straightforward search scenario, in which retrieval tasks are fixed (e.g., finding a passage to answer a specific question) and only a single modality is supported for both queries and retrieved results. This paper introduces techniques for advancing information retrieval with multimodal large language models (MLLMs), enabling a broader search scenario, termed universal multimodal retrieval, where multiple modalities and diverse retrieval tasks are accommodated. To this end, we first study fine-tuning an MLLM as a bi-encoder retriever on 10 datasets with 16 retrieval tasks. Our empirical results show that the fine-tuned MLLM retriever is capable of understanding challenging queries, composed of both text and image, but it underperforms compared to a smaller CLIP retriever in cross-modal retrieval tasks due to the modality bias exhibited by MLLMs. To address the issue, we propose modality-aware hard negative mining to mitigate the modality bias exhibited by MLLM retrievers. Second, we propose continuously fine-tuning the universal multimodal retriever to enhance its text retrieval capability while preserving multimodal retrieval capability. As a result, our model, MM-Embed, achieves state-of-the-art performance on the multimodal retrieval benchmark M-BEIR, which spans multiple domains and tasks, while also surpassing the state-of-the-art text retrieval model, NV-Embed-v1, on the MTEB retrieval benchmark. We also explore prompting the off-the-shelf MLLMs as zero-shot rerankers to refine the ranking of the candidates from the multimodal retriever. We find that, through prompt-and-reranking, MLLMs can further improve multimodal retrieval when the user queries (e.g., text-image composed queries) are more complex and challenging to understand. These findings also pave the way for advancing universal multimodal retrieval in the future.

1 INTRODUCTION

The paper broadens retrieval from narrow, single-modality tasks to universal multimodal retrieval with diverse instructions, queries, and documents. It develops MLLM retrievers and rerankers, addressing modality bias while achieving strong multimodal and text retrieval performance.

  • Universal multimodal retrieval accommodates diverse user-instructed tasks with text, image, or interleaved text-image queries and documents.
  • MLLM retrievers outperform CLIP retrievers on challenging interleaved text-image queries but underperform on cross-modal retrieval because of modality bias.
  • Modality-aware hard negative mining and continuous fine-tuning produce MM-Embed, a universal retriever with state-of-the-art multimodal performance and competitive text retrieval.
  • Zero-shot MLLM rerankers further improve accuracy on especially challenging interleaved text-image queries.On CIRCO, reranking raises mAP@5 by over 7 points against the existing state-of-the-art composed-image retriever and the universal retrievers.

2 RELATED WORK

Prior work expanded retrieval from single-modal settings toward instruction-aware and multimodal tasks, but important gaps remained in modality coverage and text retrieval. This paper extends multimodal retrieval with MLLM retrievers and zero-shot reranking.

  • Instruction-aware dense retrieval uses task-specific instructions to learn diverse retrieval tasks beyond text retrieval.M-BEIR established instruction-aware fine-tuning as important for universal multimodal retrieval.
  • Vision-language retrieval research progressed from single-modal to cross-modal and complex multimodal tasks, often assuming homogeneous query and document modalities.
  • M-BEIR introduced heterogeneous queries and candidate pools with diverse retrieval tasks, but its study was limited to CLIP-based retrievers and omitted important text-to-text tasks.
  • This work extends prior text-reranking research by prompting MLLMs as zero-shot rerankers for multimodal retrieval tasks.

3 UNIVERSAL MULTIMODAL RETRIEVAL

Universal multimodal retrieval retrieves ranked candidates for an instructed search intent from collections containing text, images, and interleaved text-image documents. The paper restricts its modality scope to text and images.

  • Given a multimodal query and task instruction, the retriever ranks candidates from a pool of multimodal documents according to a specified ranking metric.
  • Queries and candidates may each be text, image, or interleaved text-image, while the same query can express different search intents through task-specific instructions.For example, an image query can request either an image caption or a similar image.
  • The paper considers text and image modalities only; audio and video are left for future work.

4 METHOD

The method fine-tunes MLLM bi-encoder retrievers with task instructions, modality-aware hard negatives, and continued mixed-task training, then uses prompted MLLMs to rerank retrieved candidates.

  • 4 METHOD: Instruction-guided MLLM fine-tuning projects multimodal queries and documents into a shared space for k-nearest neighbor retrieval.The query representation includes the task instruction to capture retrieval-task intent.
  • 4.1 FINE-TUNING MULTIMODAL LLMS FOR UNIVERSAL MULTIMODAL RETRIEVAL: Modality-aware negative mining addresses MLLM modality bias by selecting incorrect-modality and same-modality candidates with unsatisfactory information.Incorrect-modality negatives rank above the labeled positive, while same-modality negatives rank below position k′; the experiment sets k′ = 45.
  • 4.1.1 MODALITY-AWARE HARD NEGATIVE MINING: The contrastive-learning candidate pool uses in-batch samples as random negatives before mining hard negatives from the retriever’s top-50 candidates.Hard-negative training uses twice as many candidate documents as random-negative mining under the same batch size, so batch sizes are adjusted for fair comparison.
  • 4.1.2 CONTINUOUS FINE-TUNING: Continuous fine-tuning combines universal multimodal and text-to-text retrieval data while retaining previously mined incorrect-modality negatives.The procedure rem ineates same-modality hard negatives with M hard and retains first-type negatives when M hard finds none.
  • 4.2 ZERO-SHOT MULTIMODAL LLM RERANKER: A zero-shot LLaVa-Next reranker evaluates the top 10 retrieved candidates through True/False prompts and uses the True-token probability as the relevance score.The preliminary study reports improvements mainly for interleaved text-image queries, including composed image retrieval and visual question answering.

5 EXPERIMENTS

Experiments evaluate universal multimodal retrievers across M-BEIR and MTEB, comparing CLIP- and MLLM-based backbones, hard-negative training, continuous fine-tuning, and zero-shot reranking. Results show complementary strengths: MLLMs handle multimodal queries well, modality-aware hard negatives improve single-modal retrieval, and continuous fine-tuning preserves multimodal capability while strengthening text retrieval.

  • Datasets and models: M-BEIR contains 10 datasets with 16 multimodal retrieval tasks, while MTEB evaluation covers 15 diverse text retrieval datasets.M-BEIR training uses 1.1M queries and evaluation uses 190K test queries; MTEB reports average nDCG@10 across its datasets.
  • Datasets and models: LLaVa-Next MLLM retrievers use a Mistral 7B language model with a CLIP image encoder and vision-language projector, alongside CLIP-Large baselines.The experiments include LLaVa-Next variants and CLIPSF, a CLIP-based model that separately encodes image and text inputs.
  • Main results: MLLM retrievers outperform CLIPSF on multimodal-query tasks but lag on single-modal and especially cross-modal retrieval, whereas NVEmb leads text-to-text retrieval on WebQA.The results also show domain-dependent behavior for BLIPFF, which is stronger on fashion datasets than on VisualNews.
  • Main results: Hard-negative fine-tuning significantly improves MLLM retrieval accuracy, particularly for single-modal queries, while modality-aware mining mitigates bias toward retrieving relevant text instead of images.The authors attribute the gains partly to accelerating contrastive learning in MLLM retrievers and identify modality bias in top-1 errors.
  • Ablations and reranking: Continuous fine-tuning improves text retrieval while retaining strong multimodal capability, and zero-shot reranking helps visual question answering but not FashionIQ or CIRR.Joint fine-tuning lowers multimodal accuracy by over 2 points, whereas staged fine-tuning incurs a 0.8-point multimodal drop; reranking benefits OVEN and InfoSeek.
  • Ablations and reranking: Zero-shot MLLM rerankers improve difficult interleaved text-image retrieval but degrade performance on most single-modal tasks, including after scaling from 7B to 34B.The authors hypothesize that cross-encoders better handle challenging multimodal queries, while lacking task- and domain-specific knowledge limits zero-shot performance.

6 CONCLUSION AND FUTURE WORK

The paper develops MLLM-based techniques for universal multimodal retrieval and reports strong benchmark performance, while identifying efficiency and distillation as future directions.

  • Modality-aware hard negative mining improves MLLM-based retriever accuracy by 5 points on M-BEIR.
  • MM-Embed achieves state-of-the-art universal multimodal retrieval accuracy while ranking in the top five on MTEB retrieval tasks.
  • Zero-shot MLLM rerankers further boost retrieval accuracy on challenging multimodal-query tasks.
  • Future work includes distilling MM-Embed into smaller multimodal retrievers to improve retrieval efficiency.
  • Another proposed direction is distilling an MLLM reranker into a retriever for multimodal-query tasks.

A.1 IMPLEMENTATION DETAILS

The implementation uses Tevatron and distinct training configurations for MLLM-based and CLIP-based retrievers, with length limits set for multimodal and continuous fine-tuning.

  • Training and inference use Tevatron, while CLIP-based retrievers follow Wei et al.’s settings.
  • MLLM-based retrievers are fine-tuned with DeepSpeed Zero 2 and gradient checkpointing.
  • During M-BEIR fine-tuning, query and document maximum length is set to 128.
  • Continuous fine-tuning uses both M-BEIR and text-to-text retrieval training data.

A.2 RETRIEVAL EFFICIENCY COMPARISONS

The efficiency comparison evaluates storage and latency for retrievers on the 5.6M-document M-BEIR corpus using query encoding and vector-search measurements.

  • Storage is measured on the 5.6M documents in the M-BEIR dataset.
  • Retrieval latency includes query encoding and vector search.
  • Query latency is measured by sampling 100 queries from each of the 16 M-BEIR test-query pools with batch size 1.

A.3 BASELINE REPRODUCING

The appendix documents baseline reproduction, dataset statistics, modality distributions, benchmark comparisons, and prompting configurations for retrieval and reranking experiments.

  • Baseline reproduction: The reproduced M rand(CLIPSF) implementation is intended to be equivalent to Wei et al.’s fine-tuned CLIPSF checkpoint.
  • Benchmark statistics and comparisons: M-BEIR includes statistics on datasets, document modalities, and comparisons with existing multimodal retrieval models.
  • Benchmark statistics and comparisons: Local M-BEIR evaluation restricts retrieval to each dataset’s sub-million-candidate, single-modality corpus.
  • Modality analysis: Most MLLM retrievers show low Task 1 modality accuracy because of modality bias, while modality-aware hard negatives resolve this issue for MM-Embed.
  • Evaluation configurations: The appendix reports detailed MTEB retrieval results and prompts for NVEmb, LLaVa-P, and reranking experiments.
  • Negative samples: Modality-aware negative samples can have incorrect modalities despite similar semantic meaning, whereas poor information-need matches are less accurate than correct answers.
  • Reranking analysis: For composed image retrieval, retrieved and reranked top-1 candidates sometimes differ from the sole labeled positive while appearing correct.
Loading 2411.02571v2…