PaperlayerView exact revision on arXiv

Source-linked AI summary

mmE5: Improving Multimodal Multilingual Embeddings via High-quality Synthetic Data

Haonan Chen, Liang Wang, Nan Yang, Yutao Zhu, Ziliang Zhao, Furu Wei, Zhicheng Dou

arXiv:2502.08468v1cs.CVcs.AIcs.CL

TL;DR

Limited labeled multimodal data and concerns about synthetic-data quality motivate this work. It builds a broad, aligned, high-fidelity synthesis framework and trains mmE5, which achieves state-of-the-art results on MMEB and XTD.

  • Problem

    Labeled multimodal data is limited, and prior synthetic datasets often focus on narrow English retrieval settings, leaving synthetic-data quality insufficiently explored.

    Source: §1 · p. 1

  • Method

    The authors define scope, cross-modal alignment, and fidelity criteria, synthesize diverse data with an MLLM, and train the multimodal multilingual embedding model mmE5.

    Source: §1 · p. 2, §5 · p. 8

  • Results

    mmE5 achieves SOTA on MMEB’s 36 datasets using 45 times less training data than MMRet (560K compared to 26M) in zero-shot evaluation, and achieves the best XTD results.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The paper concludes that mmE5 achieves state-of-the-art performance on both the general MMEB benchmark and the multilingual XTD benchmark.

    Source: §5 · p. 9

    Data synthesis currently relies on the proprietary GPT-4o MLLM.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Multimodal embedding models have gained significant attention for their ability to map data from different modalities, such as text and images, into a unified representation space. However, the limited labeled multimodal data often hinders embedding performance. Recent approaches have leveraged data synthesis to address this problem, yet the quality of synthetic data remains a critical bottleneck. In this work, we identify three criteria for high-quality synthetic multimodal data. First, broad scope ensures that the generated data covers diverse tasks and modalities, making it applicable to various downstream scenarios. Second, robust cross-modal alignment makes different modalities semantically consistent. Third, high fidelity ensures that the synthetic data maintains realistic details to enhance its reliability. Guided by these principles, we synthesize datasets that: (1) cover a wide range of tasks, modality combinations, and languages, (2) are generated via a deep thinking process within a single pass of a multimodal large language model, and (3) incorporate real-world images with accurate and relevant texts, ensuring fidelity through self-evaluation and refinement. Leveraging these high-quality synthetic and labeled datasets, we train a multimodal multilingual E5 model mmE5. Extensive experiments demonstrate that mmE5 achieves state-of-the-art performance on the MMEB Benchmark and superior multilingual performance on the XTD benchmark. Our codes, datasets and models are released in https://github.com/haon-chen/mmE5.

1 Introduction

The introduction frames narrow synthetic-data scope as a generalization gap, proposes quality criteria and a synthesis framework, and reports mmE5 state-of-the-art performance on MMEB and XTD with less data.

1 Introduction

  • Human-labeled multimodal datasets are costly and scarce, while prior synthetic data mostly targets limited modality types of English retrieval, harming generalization.Source: §1 · p. 1
  • The framework covers three tasks, seven modality combinations, and 93 languages, using four image perspectives, single-pass MLLM generation, real images, self-evaluation, and refinement.Its criteria are broad scope, robust cross-modal alignment, and high fidelity.Source: §1 · p. 2
  • mmE5 achieves SOTA on all 36 MMEB datasets using 45 times less training data than MMRet (560K compared to 26M) in zero-shot evaluation, and achieves the best XTD results.The paper also reports the 45× data reduction in supervised settings.Source: §1 · p. 2

2 Related Work

Prior work develops unified multimodal embeddings and synthetic training data, while mmE5’s dataset comparison emphasizes broader language, task, and modality coverage.

2 Related Work

  • Prior embedding approaches include CLIP-like unified representations and MLLM fine-tuning, including VLM2Vec’s instruction-tuning approach.Source: §2 · p. 2
  • The dataset comparison describes mmE5’s synthetic data as incorporating 93 languages, two additional tasks, and more modality combinations than previous methods.The caption also notes single-pass MLLM synthesis, real images, and self-evaluation.Source: Table 1 · p. 3
  • Synthetic multimodal datasets include MagicLens image triplets, MegaPairs retrieval from different perspectives, and GME’s fused T→IT and IT→IT data.Source: §2 · p. 3

3 Methodology: mmE5

mmE5 uses a five-stage MLLM pipeline to synthesize multimodal embedding data and fine-tune an embedding model. The framework is guided by broad scope, robust cross-modal alignment, and high fidelity.

3 Methodology: mmE5

  • The five-stage method configures each sample, generates image descriptions and data, evaluates and revises samples, then fine-tunes an embedding model; stages two through four run in one MLLM pass.The single pass is intended to minimize potential information loss.Source: §3 · p. 3

3.1 Preliminaries

  • Each synthetic sample is a task instruction, query, positive document, and hard negative, with text, images, or text-image pairs available on either side.Depending on modality, a sample can contain up to seven elements.Source: §3.1 · p. 3

3.2 Data Synthesis Framework

  • The synthesis framework applies broad scope, robust cross-modal alignment, and high fidelity to generate data for transforming an MLLM for downstream embedding tasks.Source: §3.2 · p. 3

3.2.1 Data Configuration

  • Configuration spans classification, VQA, retrieval and benchmark modality combinations, uses LAION images with jina-clip-v2 to retrieve similar positives and hard negatives, and samples XLM-R languages with high-resource weighting and English instructions.Source: §3.2.1 · p. 4

3.2.2 One-pass Generation with MLLM

  • In one pass, the MLLM interprets images from multiple perspectives, synthesizes relevant text, evaluates relevance, hard-negative plausibility, instruction clarity, and diversity, then revises each sample.Source: §3.2.2 · pp. 4–5

3.3 Finetuning Embedding Model mmE5

  • Fine-tuning uses an instruction template, final-layer [EOS] representations, and InfoNCE over positive and negative documents, with cosine similarity scaled by temperature.Source: §3.3 · p. 5

4 Experiments

Experiments show mmE5 performs strongly on MMEB and multilingual XTD, and test its transfer to other MLLM backbones. Ablations, scaling tests, and hyperparameter analyses examine how data design and training choices relate to performance.

4.1 Experimental Setup

  • MMEB covers 36 tasks across four categories, while synthetic data covers classification, VQA, and retrieval in a 1:1:2 ratio; XTD evaluation reports Recall@10 across seven languages.Visual grounding data is not synthesized because the authors consider those tasks relatively simpler for MLLMs based on MMEB results.Source: §4.1 · pp. 5–6

4.2 Results on MMEB

  • mmE5 achieves the best MMEB performance in zero-shot and supervised settings, outperforming MMRet with 560K versus 26M training samples, though it trails MMRet on zero-shot retrieval.mmE5 generalizes across all four task categories; MMRet’s retrieval-only training performs well on retrieval but generalizes less well to other task types.Source: §4.1 · p. 5, §4.2 · p. 6

4.3 Multilingual Performance on XTD

  • On seven-language XTD text-to-image retrieval, mmE5 leads overall across all languages; multilingual synthetic training beats equal-sized English-only training zero-shot, while omitting labeled data lowers performance.The synthetic dataset covers 93 languages, with English as its largest component and even sampling across 75 low-resource languages.Source: §4.3 · p. 6, Table 3 · p. 6

4.4 Application to Other Base MLLM

  • Models trained with the method outperform baselines built on the same foundation MLLMs, indicating that the approach transfers to other MLLM backbones.The tested backbones include LLaVA-1.6 and Phi-3.5-V.Source: §4.4 · pp. 6–7, Table 4 · p. 7

4.5.1 Ablation Studies

  • MMEB performance declines when Visual Interpretation, Self-evaluation, task types, modality combinations, or hard negatives are removed from the synthesis and training setup.Removing retrieval data causes the smallest drop among task-type ablations; the authors suggest the Llama-3.2 Vision backbone may have stronger retrieval capabilities than Phi-3.5-Vision.Source: §4.5.1 · p. 7
  • English-only synthetic data gives a slight MMEB advantage over non-English-only data, but the difference is minimal and 280K non-English samples still yield satisfactory results.The authors report that their multilingual dataset supports generalization in both multilingual and English-only contexts.Source: §4.5.1 · pp. 7–8

4.5.2 Scaling Effect

  • MMEB performance consistently improves as synthetic data grows, following a linear-log relationship that can inform cost–performance balancing.The scaling experiments use zero-shot evaluation on MMEB.Source: §4.5.2 · p. 8, Figure 4 · p. 8

4.6 Hyperparameter Analysis

  • With 280K synthetic samples, MMEB performance first rises and then falls as LoRA rank or temperature increases, whereas larger batch sizes consistently improve performance.Larger batches require more GPU memory; the authors describe trade-offs for rank and temperature involving computation, underfitting, and preservation of pretrained multimodal reasoning.Source: §4.6 · p. 8, Figure 5 · p. 8

5 Conclusion

The paper defines high-quality multimodal synthetic data by broad scope, robust cross-modal alignment, and high fidelity, then uses those principles to build data and train mmE5. It reports state-of-the-art performance on both MMEB and XTD.

  • The paper defines high-quality multimodal synthetic data by three criteria: broad scope, robust cross-modal alignment, and high fidelity.Source: §5 · p. 8
  • A synthesis framework guided by those criteria produces data for training a multimodal multilingual embedding model, mmE5.Source: §5 · p. 8
  • mmE5 achieves state-of-the-art performance on the general MMEB benchmark and the multilingual XTD benchmark.Source: §5 · p. 9

Limitations

The authors identify reliance on proprietary GPT-4o, coverage limited to text and images, and constrained training-data volume as limitations.

Limitations

  • mmE5 data synthesis relies on proprietary GPT-4o, and the authors suggest aligning smaller MLLMs with GPT-like models for more efficient synthesis.Source: Limitations · p. 9
  • mmE5 covers text and image modalities; future models could extend coverage to audio and video.Source: Limitations · p. 9
  • Cost limits and an observed scaling effect constrained training-data volume; future work may increase data size while preserving diversity.Source: Limitations · p. 9

A Details about Synthetic Data

The synthetic dataset combines three tasks and seven modality combinations in 560K samples for training mmE5.

A Details about Synthetic Data

B.1 Data Synthesis

GPT-4o generates samples using real-world images from LAION-400m, with retrieved similar and hard-negative images added for image-document combinations.

B.1 Data Synthesis

  • GPT-4o-2024-08-06 generates samples around query images from LAION-400m; image-document combinations also use retrieved similar and hard-negative images.Corrupted or inaccessible corpus images are excluded.Source: §B.1 · p. 12

B.2 Finetuning Embedding Model

The paper describes mmE5's fine-tuning setup and prompts that vary by task and modality, with generated samples evaluated and revised before use.

B.2 Finetuning Embedding Model

  • mmE5 is fine-tuned with open-source Llama-3.2-11B-Vision on 64 A100 GPUs for one epoch, using LoRA rank 8 and one hard negative per sample.Training uses synthetic and labeled data; hard negatives are mined for each MMEB subset.Source: §B.2 · p. 12

C Prompts

  • Prompts vary by task and modality; absent text fields are empty, while clarity and difficulty settings are randomly selected.For visual descriptions, the MLLM covers four perspectives, evaluates generated data, suggests improvements, and outputs a revised sample; VQA uses a fixed instruction.Source: C Prompts · p. 12

D Data Examples

The examples span retrieval, classification, and visual question answering (VQA). Their prompts describe image content, generate task-specific examples, and evaluate or refine the resulting data.

D Data Examples

  • Classification examples begin with a detailed image description and task brainstorming, then produce a JSON example with instructions, input, labels, evaluation, and revisions.The input text may be empty or task-specified, and the example includes a correct label and a related misleading label.Source: D Data Examples · pp. 13–14
  • The prompts evaluate relevance, plausibility, clarity, and diversity, then request specific revisions that preserve consistency with the image and task.They caution against revisions that overly simplify the task, text, or labels.Source: D Data Examples · pp. 13–14
  • VQA examples pair a detailed image description with a question, positive answer, plausible hard negative, quality evaluation, suggested improvements, and revised answers.The prompt asks for visual details including objects, relationships, and contextual features.Source: D Data Examples · p. 14
  • Classification prompts constrain question length and language, require plausible but inferior hard negatives, and discourage explicit answer hints.They also specify clarity and education-level options for the question.Source: D Data Examples · p. 14
  • The cited passage associates Figure 8 with classification and Figure 9 with VQA, while mentioning Figures 6 and 7 without specifying their task.Source: D Data Examples · p. 14

E Detailed Results

The section reports MMEB baseline comparisons in zero-shot and supervised fine-tuning settings, while its retrieval prompts distinguish query-image-only tasks from tasks involving document images.

E Detailed Results

  • The MMEB results section compares baseline models in zero-shot and supervised fine-tuning settings and omits detailed partially supervised results; Table 7 covers each MMEB dataset.The supplied table caption identifies its scope but does not provide result values.Source: E Detailed Results · p. 14, Table 7 · p. 21
  • Query-image-only retrieval examples target image-to-text or (image,text)-to-text tasks, with task-specific queries and independently created documents, including useful but less comprehensive hard negatives.The prompt also constrains query length, specificity, clarity, and topic diversity.Source: E Detailed Results · p. 15
  • Retrieval prompts assess relevance, plausibility, clarity, and diversity, then request revisions that retain consistency with the images and task requirements.For image-to-text retrieval, the prompt asks that hard negatives remain similar enough to challenge while still being incorrect.Source: E Detailed Results · pp. 15–16
  • Retrieval tasks involving document images cover image-to-image and mixed text-image modality combinations, with the task requiring a content-based link between query and positive images.The prompt calls for visual descriptions before constructing a retrieval example.Source: E Detailed Results · p. 16
Loading 2502.08468v1…