PaperlayerView exact revision on arXiv

Source-linked AI summary

MegaPairs: Massive Data Synthesis For Universal Multimodal Retrieval

Junjie Zhou, Zheng Liu, Ze Liu, Shitao Xiao, Yueze Wang, Bo Zhao, Chen Jason Zhang, Defu Lian, Yongping Xiong

arXiv:2412.14475v1cs.CVcs.CL

TL;DR

Multimodal retrieval lacks scalable, diverse instruction-tuning data, especially beyond text-image matching. MegaPairs synthesizes heterogeneous open-domain image pairs and language-model-generated instructions, enabling MMRet to achieve state-of-the-art zero-shot performance across four CIR benchmarks and 36 MMEB datasets. The method also outperforms a MagicLens baseline trained on 70× more data and supports further downstream gains.

  • Problem

    Existing vision-language encoders and multimodal retrieval datasets provide insufficient coverage, scale, and diversity for universal multimodal retrieval.

  • Method

    MegaPairs mines heterogeneous image pairs from open-domain images with multiple similarity models and uses VLM and LLM annotators to generate open-ended retrieval instructions.

  • Results

    MMRet achieves state-of-the-art zero-shot performance across four composed image retrieval benchmarks and 36 MMEB datasets, while 500K MegaPairs instances outperform MagicLens trained on 36.7M instances.

  • Takeaways & Limitations

    MegaPairs provides a scalable synthetic dataset, trained retrievers, and production pipeline for advancing universal multimodal retrieval.

  • Takeaways & Limitations

    MegaPairs currently uses three pairing retrievers, leaving additional text-domain and image-text retrieval strategies for future exploration; its harmful-content screening may also be incomplete.

Abstract

from arXiv · show

Despite the rapidly growing demand for multimodal retrieval, progress in this field remains severely constrained by a lack of training data. In this paper, we introduce MegaPairs, a novel data synthesis method that leverages vision language models (VLMs) and open-domain images, together with a massive synthetic dataset generated from this method. Our empirical analysis shows that MegaPairs generates high-quality data, enabling the multimodal retriever to significantly outperform the baseline model trained on 70$\times$ more data from existing datasets. Moreover, since MegaPairs solely relies on general image corpora and open-source VLMs, it can be easily scaled up, enabling continuous improvements in retrieval performance. In this stage, we produced more than 26 million training instances and trained several models of varying sizes using this data. These new models achieve state-of-the-art zero-shot performance across 4 popular composed image retrieval (CIR) benchmarks and the highest overall performance on the 36 datasets provided by MMEB. They also demonstrate notable performance improvements with additional downstream fine-tuning. Our produced dataset, well-trained models, and data synthesis pipeline will be made publicly available to facilitate the future development of this field.

1 Introduction

MegaPairs addresses limited and low-diversity multimodal retrieval instruction data by synthesizing open-domain image-pair instructions with multiple similarity models and language-model annotators. Its data already outperforms MagicLens using 70× less training data, while full-scale MMRet models achieve state-of-the-art zero-shot results across major retrieval evaluations.

  • Motivation: Existing vision-language encoders are mainly trained for text-image matching, leaving them insufficient for composed image retrieval and other multimodal tasks.Instruction tuning can improve multi-task capacity, but multimodal retrieval instruction data remains scarce.
  • Motivation: Current synthesis methods are constrained by limited webpage scale, unrelated or near-duplicate image pairs, monotonous relationships, and restricted dataset availability.MagicLens synthesizes instructions from co-existing webpage images, but these limitations affect scalability, quality, diversity, and access.
  • MegaPairs: MegaPairs samples heterogeneous image pairs using CLIP vision, DINO vision, and CLIP text similarity, then uses VLM and LLM annotators to generate relationship descriptions and retrieval instructions.The three similarity models target visual-semantic, visual-pattern, and caption correlations.
  • Results: 500K MegaPairs instances surpass the full 36.7M MagicLens training set, delivering better fine-tuning results with 70× less training data.The comparison uses the same pretrained model in a pilot experiment.
  • Results: Using the full synthetic dataset, MMRet reaches state-of-the-art zero-shot performance on four composed image retrieval benchmarks and 36 MMEB datasets, with further gains after downstream fine-tuning.The dataset, trained models, and production pipeline are intended for public release.

2 Related Work

Multimodal retrieval increasingly combines image and text in queries or candidates across applications, but common VLMs and instruction datasets remain poorly suited to this breadth. Existing datasets are limited by human-annotation scale and diversity, while MagicLens is additionally constrained by multi-image webpages and private access.

  • Multimodal Retrieval: Multimodal retrieval covers image-text-integrated queries or candidates in applications including instruction-based image retrieval, multimodal documents, multimodal knowledge retrieval, and RAG.Most methods use pretrained VLMs, but image-text matching pretraining does not adequately support joint multimodal encoding and comprehension.
  • Instruction Tuning for Multimodal Retrieval: Instruction tuning improves multi-task capacity for language and embedding models, including multimodal retrieval models.Prior multimodal retrieval instruction datasets include image retrieval and multimodal document retrieval resources.
  • Instruction Tuning for Multimodal Retrieval: Human-annotated multimodal retrieval instruction datasets remain limited in scale and diversity, while MagicLens is constrained by scarce multi-image webpages and private availability.These constraints motivate creating and releasing high-quality instruction-tuning datasets.

3 Methodology

MegaPairs constructs universal multimodal retrieval training data by mining diverse correlated image pairs from open-domain corpora and automatically generating open-ended instructions. MMRet then combines pretrained VLM architectures with multimodal contrastive learning, achieving state-of-the-art zero-shot CIR results across model scales and benchmarks.

  • MegaPairs Data Synthesis: Open-domain image corpora address the challenges of sampling relevant, diverse image pairs and annotating their instructions at scale.The synthesis pipeline has two components: image-pair mining and open-ended instruction generation.
  • Mining Correlated Image Pairs: Multiple similarity models retrieve heterogeneous target images for each query, while other retrieved images serve as hard negatives during training.The correlations include visual-semantic and visual-pattern relationships, and the hard-negative strategy is empirically effective.
  • Generating Open-Ended Instructions: MLLMs describe common concepts and differences in each image pair, and LLMs refine those descriptions into multiple textual retrieval instructions.Generating multiple instructions increases textual instruction diversity.
  • Implementations: MegaPairs contains 26,235,105 image pairs mined from 20 million captioned images using EVA-CLIP and DINOv2 similarity models.The implementation uses Recap-DataComp-1B as the image corpus.
  • MMRet Model: MMRet integrates dual-encoder and MLLM-based VLM architectures to produce universal multimodal embeddings.The CLIP-based variant encodes composed image-text inputs by element-wise addition of image and text encoder outputs, while the MLLM variant processes interleaved multimodal tokens.

4 Experiments

MegaPairs is evaluated through zero-shot composed image retrieval, broader MMEB embedding tasks, supervised fine-tuning, and dataset-quality analyses. MMRet achieves leading performance across these evaluations while showing strong data efficiency and gains from mined hard negatives and diverse pairing strategies.

  • 4.1.3 Evaluation Results: MMRet-MLLM achieves leading results across CIR benchmarks, including 42.2% mAP@5 on CIRCO versus MagicLens-L’s 34.1%.It also exceeds current state of the art by 7.4% in CIRR R@1, 4.5% in CIRR Rs@1, and 3.7% in GeneCIS Rs@1.
  • 4.1.3 Evaluation Results: MMRet remains competitive across model scales: Base and Large exceed comparable models by 4.5% and 4.7% in CIRR R@1, and by 3.5% and 5.1% in CIRCO mAP@5.MMRet-Base also reaches 34.3% mAP@5 on CIRCO, surpassing many larger models, including models with dozens of times more parameters.
  • 4.2.1 Zero-shot Performance: MMRet-MLLM achieves state-of-the-art zero-shot performance on MMEB, including the highest overall score and strong generalization across its embedding meta-tasks.UniIR is not strictly zero-shot because it was trained on M-BEIR datasets overlapping with MMEB retrieval tasks.
  • 4.2.2 Supervised Fine-tuning Performance: After supervised fine-tuning, MMRet-MLLM reaches 64.1% overall Precision@1 on MMEB and improves downstream performance by 9.1% over LLaVA-1.6-based VLM2Vec.Its improvements over the two VLM2Vec versions are 11.6% and 7.1% on out-of-distribution datasets.
  • 4.3 Analysis: At 0.5M samples, MegaPairs significantly surpasses MagicLens across benchmarks despite MagicLens using 36.7M pairs, while mined hard negatives and combined pairing strategies further improve performance.Performance rises with MegaPairs data size; hard negatives improve all benchmarks, and combining pairing strategies consistently outperforms single strategies.

5 Conclusion

MegaPairs is a large-scale multimodal pairing dataset for universal multimodal retrievers, and MMRet models trained on it achieve state-of-the-art zero-shot results across composed image retrieval and MMEB.

  • MegaPairs provides diverse open-world image pairs annotated with open-ended instructions capturing visual and semantic relationships.
  • MMRet achieves state-of-the-art zero-shot performance on four composed image retrieval tasks and the 36-dataset MMEB benchmark.
  • Extensive experiments demonstrate MegaPairs' generalization capability and high-quality features.

Limitations

MegaPairs uses three distinct retrievers to generate richer image pairs, but additional pairing strategies remain unexplored.

  • MegaPairs' three distinct retrievers generate richer and more diverse image pairs.The paper suggests exploring advanced text retrievers such as BGE and varied image-text retrieval strategies.
  • The pairing pipeline could be expanded with advanced text-domain retrievers and alternative image-text retrieval methods.

Ethics Statement

MegaPairs uses screened open-domain images and automated multimodal-language annotation, while acknowledging that harmful-content screening may be incomplete.

  • MegaPairs images come from Recap-Datacomp-1B and were screened by the Datacomp team to remove harmful content.
  • The authors acknowledge that harmful-content screening may contain omissions and discourage using MMRet for sensitive content.
  • The annotation pipeline uses an MLLM to describe image-pair commonalities and differences, then an LLM refines those descriptions into textual instructions.

B Training Details of MMRet on MegaPairs

MMRet training uses prompt-based instruction generation alongside distinct CLIP-based and MLLM-based configurations, with fixed resolutions, hard negatives, and linear learning-rate decay.

  • CLIP-based MMRet uses one positive image and four hard negatives per query, with 224x224 inputs and 15,000 or 25,000 training steps.
  • WORD_NUM ranges from 60 to 100 in practical MLLM data generation to increase description diversity.
  • MLLM-based MMRet uses one positive image and three hard negatives, 512x512 inputs, LoRA rank 32, and 20,000 training steps.
  • Both model families start at a learning rate of 5 × 10^-6 and use linear decay.

C Detailed Information and Evaluation Metrics of Zero-Shot CIR Benchmarks

The zero-shot CIR evaluation covers four benchmarks with distinct datasets, query structures, and metrics, while MegaPairs uses instruction prompts built from image connections and demonstrations.

  • Query construction: MegaPairs query generation instructs an LLM to create concise queries from connected images while avoiding explicit source-image details.The practical process randomly selects five demonstrations from a pool of 50, although Figure 4 shows two.
  • CIRCO: CIRCO evaluates 800 composed image-text queries against 123,403 candidate images using mean Average Precision (mAP).Each query has multiple ground-truth images, making CIRCO the paper’s main zero-shot CIR benchmark.
  • CIRR: CIRR evaluates 4,148 queries over 2,315 images with standard and subset retrieval recall metrics.Each query has one annotated positive target, but false negatives remain possible.
  • FashionIQ: FashionIQ evaluates dress, shirt, and toptee retrieval on 6,016 validation queries and 15,536 images using average recall.The three subtasks are reported separately before averaging their recall values.
  • GeneCIS: GeneCIS contains four conditional image-similarity subtasks, reformulated as composed image-text queries using task-derived condition descriptions.Models retrieve the most similar images from dedicated subsets while changing or focusing an attribute or object.

D Full results on CIR Benchmarks

MMRet achieves state-of-the-art results across multiple CIR benchmarks and MMEB settings, while MegaPairs examples combine visual similarity with semantic relationships beyond appearance.

  • Full CIR results: MMRet achieves state-of-the-art performance across CIRCO, CIRR, and GeneCIS at various model sizes.The full benchmark results are reported in Tables 6, 7, and 9, respectively.
  • MMEB results: MMRet achieves state-of-the-art performance on MMEB in both zero-shot and fine-tuning settings, including stronger performance than the second-best model on OOD datasets.MMEB spans 36 datasets across four meta-task categories, with 20 in-distribution and 16 out-of-distribution datasets.
  • MegaPairs examples: MegaPairs examples associate each query image and alt-text caption with multiple targets that are visually similar or semantically related beyond visual features.The target set is designed to cover relationships such as shared style, attributes, objects, and concepts even when images look different.

G Qualitative Results of MMRet on Zero-shot CIR Tasks

Qualitative retrieval examples show MMRet handling compositional image-text queries by integrating objects, settings, attributes, poses, lighting, distance, actions, and scene context.

  • Qualitative comparison: MMRet retrieves images that satisfy multiple query constraints, whereas MagicLens can focus on an individual object while missing broader semantic context.The comparison uses CLIP-L backbones for both methods in zero-shot CIR examples.
  • Query capabilities: The examples test MMRet’s understanding of attire and setting, foreground and background entities, pose and angle, lighting and distance, and dynamic actions.These cases correspond to the blue ties, sweet beverage boats and sky, bench top, darker ground and closer distance, and wheel in the air queries.
  • Evaluation context: The reported benchmark tables cover CIRCO, CIRR, FashionIQ, GeneCIS, and MMEB results, with bold values indicating the best performance at each model scale where stated.The table captions identify the benchmark scope and model-scale comparison conventions.
Loading 2412.14475v1…