PaperlayerView exact revision on arXiv

Source-linked AI summary

GME: Improving Universal Multimodal Retrieval by Multimodal LLMs

Xin Zhang, Yanzhao Zhang, Wen Xie, Mingxin Li, Ziqi Dai, Dingkun Long, Pengjun Xie, Meishan Zhang, Wenjie Li, Min Zhang

arXiv:2412.16855v2cs.CLcs.IR

TL;DR

UMR seeks a unified retriever for text, images, and fused-modal content, but prior MLLM approaches often relied on limited or imbalanced training data. The paper introduces GME with balanced multimodal training and a fused-modal synthesis pipeline, then evaluates it on UMRB, where it achieves state-of-the-art performance among existing UMR methods.

  • Problem

    Prior MLLM-based UMR approaches used limited modality training, while existing multimodal data were imbalanced and fused-modal content was under-represented.

  • Method

    GME uses an MLLM backbone with balanced single-modal, cross-modal, and fused-modal training, supported by a pipeline that synthesizes fused-modal pairs and by the UMRB benchmark.

  • Results

    GME achieves state-of-the-art performance among existing UMR methods on UMRB, with the larger model further improving over the smaller GME model and prior UMR systems.

  • Takeaways & Limitations

    Unified MLLM retrieval can cover single-modal, cross-modal, fused-modal, and visual-document tasks when trained and evaluated with broader multimodal data.

  • Takeaways & Limitations

    The paper states that its limitations are outlined in the limitations section, without specifying them in the supplied passage.

Abstract

from arXiv · show

Universal Multimodal Retrieval (UMR) aims to enable search across various modalities using a unified model, where queries and candidates can consist of pure text, images, or a combination of both. Previous work has attempted to adopt multimodal large language models (MLLMs) to realize UMR using only text data. However, our preliminary experiments demonstrate that more diverse multimodal training data can further unlock the potential of MLLMs. Despite its effectiveness, the existing multimodal training data is highly imbalanced in terms of modality, which motivates us to develop a training data synthesis pipeline and construct a large-scale, high-quality fused-modal training dataset. Based on the synthetic training data, we develop the General Multimodal Embedder (GME), an MLLM-based dense retriever designed for UMR. Furthermore, we construct a comprehensive UMR Benchmark (UMRB) to evaluate the effectiveness of our approach. Experimental results show that our method achieves state-of-the-art performance among existing UMR methods. Last, we provide in-depth analyses of model scaling and training strategies, and perform ablation studies on both the model and synthetic data.

1. Introduction

Universal Multimodal Retrieval unifies search across text, images, visual documents, and fused-modal content. The paper introduces GME, balanced multimodal training, fused-modal data synthesis, and UMRB to address modality imbalance and evaluate unified retrieval.

  • Motivation: Universal Multimodal Retrieval lets queries and candidates use arbitrary modalities, motivating a unified dense retriever instead of separate modality-specific systems.Inputs can be text-only, image-only, or fused; embeddings are projected into a shared space for efficient top-k ranking.
  • Motivation: Text-only training can produce multimodal embeddings, but more diverse multimodal compositions may better unlock MLLM performance and generalization.Existing UMR models also largely neglect visual documents such as document screenshots.
  • Approach: GME is an instruction-based MLLM embedding framework supporting text, images, visual documents, and fused-modal contents.Its training categorizes tasks as single-modal, cross-modal, and fused-modal, with experiments indicating that a balanced mixture performs best.
  • Approach: 1.1M fused-modal pairs are generated through an efficient synthesis pipeline to address the under-representation of fused-modal training data.The pipeline is designed to enhance training and model capabilities.
  • Evaluation: UMRB combines text, multimodal, visual-document, and newly processed fused-modal retrieval tasks to evaluate unified retrieval comprehensively.The authors report state-of-the-art performance on UMRB and analyze scaling, training strategies, and synthetic-data ablations.

2. Related Work

Prior multimodal retrieval work used separate or cross-modal architectures and increasingly adapted MLLMs with limited training modalities. GME extends unified MLLM retrieval to visual documents while retaining text-to-text capabilities.

  • Multimodal Large Language Models: MLLMs extend language models with visual information processing through a language model, vision encoder, and projector.Examples include GPT-4V, LLaVA, Qwen-VL, InternVL, and MiniCPM-V.
  • Multimodal Retrieval: Earlier multimodal retrieval research focused on single-modal or cross-modal tasks, often using pretrained CLIP- or BLIP-based embedding approaches.Unified retrieval models emerged to support complex multimodal retrieval and retrieval-augmented generation scenarios.
  • Multimodal Retrieval: E5-V and VLM2VEC showed that MLLMs fine-tuned on single-text or vision-centric relevance data can transfer to multimodal retrieval.Their training settings remain narrower than the unified retrieval scope addressed here.
  • Research Gap: GME is presented as the first MLLM-based universal retriever combining visual retrieval with strong text-to-text retrieval capabilities.It also extends unified retrieval beyond natural images to text-rich visual documents.
  • Embedding Models: Pretrained language models underpin strong text embedding models such as Contriever, E5, GTE, and BGE.This line of work provides the text-retrieval foundation for related embedding approaches.

3. Universal Multimodal Retrieval

UMR tasks are organized by the modalities of queries and candidates into single-modal, cross-modal, and fused-modal retrieval. UMRB evaluates these settings through a broad collection of datasets, including visual-document and fused-modal tasks.

  • Task Taxonomy: Single-modal retrieval uses the same modality for queries and candidates, including text-to-text and image-to-image retrieval.
  • Task Taxonomy: Cross-modal retrieval uses different query and candidate modalities, including text-to-image and text-to-visual-document retrieval.The visual-document setting covers rich-text images such as screenshots converted from scholarly PDFs.
  • Task Taxonomy: Fused-modal retrieval combines text and images in queries, candidates, or both, as in EVQA.These tasks represent more complicated mixed-modality retrieval settings.
  • UMRB: UMRB contains 47 evaluation datasets spanning text-to-text, vision-centric, fused-modal, and text-to-visual-document retrieval.Its sources include BEIR, M-BEIR, additional fused-modal datasets, and ViDoRe.
  • UMRB: UMRB-Partial samples 39% of the full benchmark while retaining evaluation richness for faster experimental validation.

4. Method

GME adapts an MLLM into a unified retriever for text, images, visual documents, and image-text content. Its method combines contrastive learning, diverse training-data mixtures, and a synthesis pipeline for scarce fused-modal data.

  • Model Architecture: GME uses an MLLM that accepts text, images, and image-text pairs, representing each input with the final hidden state of its last token.Task-specific fine-tuning is used because pretrained MLLM objectives are not optimized for representation learning.
  • Contrastive Learning: Contrastive training minimizes cosine distance for relevant query-candidate pairs while maximizing it for irrelevant candidates, with task-specific instructions guiding retrieval.The setup supports text, images, and image-text pairs on both query and candidate sides.
  • Training Data Composition: A balanced mixture of single-modal, cross-modal, and fused-modal data improves retrieval across diverse settings, whereas single-type data excels mainly on corresponding tasks.The mixed-data model combines T→T, I→I, T→VD, T→I, and IT→IT data types.
  • Training Data Composition: Fused-modal data is scarce: common datasets collectively contain fewer than one million training instances, compared with over ten million single-modal and cross-modal instances.This imbalance motivates the synthetic-data pipeline.
  • Fused-Modal Data Synthesis: The synthesis pipeline generates natural queries, filters them by passage retrieval relevance, then adds entities, retrieved images, and captions to create fused-modal pairs.The pipeline discarded 1.2% of generated queries during its relevance filter.
  • Fused-Modal Data Synthesis: The pipeline produced 1,135,000 fused-modal entries and retained 1,102,000 after filtering, with 2.9% data loss and 600 A100 GPU hours of computation.The retained entries include T→IT and IT→IT types.

5. Experiments

Experiments evaluate GME across universal multimodal retrieval settings using multiple model sizes, baselines, embedding visualizations, and ablations. GME outperforms prior UMR systems, handles fused-modal and visual-document retrieval, and produces semantically organized embeddings.

  • Main Results: GME-Qwen2-VL-2B already outperforms VISTA, while GME-Qwen2-VL-7B further improves average performance across UMRB retrieval categories.Table 3 averages scores over single-modal, cross-modal, and fused-modal subtasks and reports an overall micro-average.
  • Main Results: GME outperforms VISTA and One-Peace, while E5-V performs well on text-dominated tasks but falls short elsewhere.The authors attribute the broader performance pattern to the importance of multimodal training data for UMR.
  • Main Results: GME is designed for fused-modal content and achieves significantly better performance than the compared UMR baselines on that setting.The cited comparison contrasts GME with VISTA, E5-V, and One-Peace.
  • Main Results: GME is competitive with or exceeds DSE, a roughly 4B-parameter model specialized for text-to-visual-document retrieval.This comparison evaluates whether visual-document retrieval can be integrated into a unified retriever.
  • Embedding Analysis: In t-SNE visualizations, GME embeddings are intermingled and semantically organized, whereas CLIP embeddings are distinctly separated by modality.Points from the same semantic group are closely clustered for GME.
  • Synthetic Data Ablation: Google-retrieved images perform marginally better than FLUX-generated images, with the difference described as minor and acceptable for large-scale generation.The comparison is part of the synthetic fused-modal data ablation.
  • Modeling Ablation: LoRA with rank 8 yields the best performance among the tested fine-tuning choices, while mean pooling and bidirectional attention negatively affect performance.These findings come from the modeling ablation on Qwen2-VL-2B.

6. Conclusion

The work targets universal multimodal retrieval by categorizing UMR tasks, proposing GME, and introducing UMRB for broader evaluation. GME achieves state-of-the-art performance while supporting diverse multimodal retrieval settings.

  • GME is an MLLM-based UMR model trained with contrastive learning on diverse multimodal data, including visual retrieval.The approach supports text, images, visual documents, and fused-modal contents.
  • UMRB integrates multiple data sources to balance existing UMR tasks with text and visual document retrieval.The benchmark is intended to provide a more thorough assessment of UMR model performance.
  • GME achieves state-of-the-art performance on UMRB and is accompanied by analyses validating the methods.The analyses cover model behavior and method effectiveness.

7. UMRB Details

UMRB combines text, image, visual-document, and fused-modal retrieval tasks across diverse datasets. Its full benchmark contains 47 subtasks, approximately 200,000 evaluation instances, and 40 million candidates.

  • Task coverage: UMRB includes text-to-text, image-to-image, text-to-image, image-to-text, text-to-visual-document, and fused-modal retrieval tasks.Examples include Wikipedia paragraphs, image pairs, news headline-image pairs, and document screenshots.
  • Cross-modal tasks: VisualNews supports both T→I and I→T retrieval using news headlines and associated images.The evaluation uses 19,995 T→I samples and 20,000 I→T samples.
  • Fused-modal tasks: Fused-modal tasks retrieve image-text candidates or use image-text queries, including WebQA, EVQA, EDIS, OVEN, INFOSEEK, ReMuQ, OKVQA, and LLaVA.These tasks combine modalities in queries, candidates, or both.
  • Composed image retrieval: FashionIQ and CIRR evaluate composed image retrieval with a reference image and a modification sentence.FashionIQ uses 6,003 evaluation samples, while CIRR uses 4,170.
  • Benchmark scale: 47 subtasks span approximately 200,000 evaluation instances and 40 million candidates, making full evaluation computationally expensive.A full GME-7B evaluation required approximately 400 A100*80G GPU hours, motivating UMRB-Partial.

8. Results Details

GME outperforms prior UMR systems across the benchmark suite, with gains from larger models and mixed training data. It also performs strongly on multimodal and visual-document benchmarks, while pure text retrieval remains a trade-off.

  • UMRB results: GME-Qwen2-VL-2B outperforms VISTA on average UMRB scores, and GME-Qwen2-VL-7B improves further.GME reaches state-of-the-art performance on every dataset except Nights.
  • Training data: Mixed training data produces the best performance across the evaluated UMRB-partial training-data conditions.The comparison covers T→T, I→I, T→VD, T→I, IT→IT, and Mix data types.
  • BEIR results: GME is the only multimodal embedder reported to match pure text embedders on BEIR text retrieval tasks.Other multimodal embedders do not match pure text embedders, including E5-V, which is fine-tuned exclusively on text.
  • BEIR results: 60.25 versus 55.63: gte-Qwen2-7B-instruct outperforms GME-Qwen2-VL-7B on pure text retrieval at the same model scale.The comparison identifies an explicit pure-text performance compromise for multimodal embedding.
  • ViDoRe results: GME-Qwen2-VL-2B surpasses ColPali on ViDoRe, while GME-Qwen2-VL-7B improves further on visual document retrieval.ColPali was trained exclusively on ViDoRe for this task.

9. Experiment Details

The experiments use Qwen2-VL-based GME models with contrastive learning and task-specific instructions. Results are reported with task-dependent ranking metrics across UMRB-partial and UMRB.

  • Implementation: GME-2B and GME-7B are initialized from Qwen2-VL and trained with BF16 precision and LoRA rank 8.Training uses a decoupled AdamW optimizer with learning rate and weight decay of 1e-4.
  • Training objective: The contrastive learner uses the [EOS] hidden state as the embedding, with temperature 0.03 and eight hard negatives per query.Each query has one positive candidate and eight hard negative candidates.
  • Evaluation metrics: UMRB results report NDCG@10 for T→T, NDCG@5 for T→VD, Recall@10 for Fashion200K, FashionIQ, and OKVQA, and Recall@5 for other datasets.These metric choices follow the benchmark reporting conventions stated for Table 7.
  • Evaluation metrics: UMRB-partial reports NDCG@10 for T→T, NDCG@5 for T→VD, Recall@10 for FashionIQ, and Recall@5 for other datasets.The metric assignments are specified for Table 8.
  • Task design: UMRB contains 47 tasks with task-specific candidate types, and even one dataset can use different candidates across task types.WebQA contrasts text candidates in T→T with image-text candidates in T→IT.
  • Task design: Task-specific instructions guide the model through the retrieval process.The detailed instructions are provided in Table 13.

10. Fused-Modal Data Synthesis Details

The fused-modal synthesis pipeline transforms passages into image-text queries through staged question generation, entity replacement, image acquisition, and quality filtering. It combines generated and retrieved images to create multimodal training examples.

  • Synthesis pipeline: The synthesis goal is fused-modal candidate-to-query relevance data rather than conventional single-modality text relevance pairs.This adapts doc2query toward image-text retrieval training data.
  • Synthesis pipeline: The pipeline first generates a natural question and answer from a selected passage, then extracts the question’s main entity and refactors the question around it.The first step uses in-context learning; the second prepares the question for entity replacement.
  • Synthesis pipeline: The extracted entity is replaced with an image, either generated from an LLM-produced caption using FLUX or retrieved through the Google Image Retrieval API.Both routes produce the image component of a fused-modal query.
  • Quality control: A text retrieval model filters queries whose source passage does not rank within the top 20 results.This filtering follows the Promptagator framework.
  • Quality control: Google-retrieved images are filtered with CLIP using a relevance threshold of 0.2, selected as the median score to balance image quality and image availability.The threshold is intended to preserve quality while retaining sufficient images for most text queries.
  • Output data: The synthesis flow produces fused-modal queries from passages spanning 15 domains and includes both FLUX-generated and Google-retrieved images.The examples distinguish images generated by FLUX.1-dev from those obtained through Google Image retrieval.

11. Limitations

The paper’s evaluation scope is limited by its data and language settings. UMRB retains only single-image queries and candidates, and GME is trained and tested only in English.

  • Scope: The work presents UMRB as a benchmark for training and testing universal multimodal retrieval while explicitly identifying limitations in its evaluation scope.The stated limitations frame the benchmark and model results rather than eliminating their reported scope.
  • Single-image scope: UMRB retains a single image per query and candidate, so performance on interleaved inputs mixing multiple images and texts cannot be assessed.The restriction reflects model-training efficiency and limited relevant data.
  • Language scope: Although Qwen2-VL supports multiple languages, GME is trained and tested only in English, leaving performance in other languages unevaluated.The limitation concerns the paper’s training and testing configuration rather than the backbone’s language capability.
Loading 2412.16855v2…