PaperlayerView exact revision on arXiv

Source-linked AI summary

E5-V: Universal Embeddings with Multimodal Large Language Models

Ting Jiang, Minghui Song, Zihan Zhang, Haizhen Huang, Weiwei Deng, Feng Sun, Qi Zhang, Deqing Wang, Fuzhen Zhuang

arXiv:2407.12580v1cs.CLcs.CVcs.IR

TL;DR

Multimodal representation with MLLMs remains underexplored, especially for universal embeddings across visual and language inputs. E5-V uses prompts to unify modalities and trains the adapted MLLM only on text pairs, achieving strong results across four task types while reducing reliance on multimodal data and training.

  • Problem

    Universal multimodal embeddings remain challenging because existing approaches have limited language understanding, shallow visual-language fusion, costly scaling, and difficult multimodal data requirements.

    Source: §1 · p. 2, §2.1 · p. 3

  • Method

    E5-V adapts an MLLM with prompts that unify multimodal representations, then applies contrastive learning only to text pairs while removing the modality encoder and projector during training.

    Source: §1 · p. 2, §4 · p. 5

  • Results

    E5-V achieves strong performance across text-image retrieval, composed image retrieval, image-image retrieval, and sentence embeddings, including 12.2% and 15.0% Recall@1 gains over CLIP ViT-L on Flickr30K and COCO image retrieval.

    Source: §4.1 · p. 6, §6 · p. 9

  • Takeaways & Limitations

    Prompted MLLMs can transfer single-modality representation capability to multimodal embeddings and follow task-specific prompts not seen during training.

    Source: §1 · p. 2, §4.1 · p. 6

Abstract

from arXiv · show

Multimodal large language models (MLLMs) have shown promising advancements in general visual and language understanding. However, the representation of multimodal information using MLLMs remains largely unexplored. In this work, we introduce a new framework, E5-V, designed to adapt MLLMs for achieving universal multimodal embeddings. Our findings highlight the significant potential of MLLMs in representing multimodal inputs compared to previous approaches. By leveraging MLLMs with prompts, E5-V effectively bridges the modality gap between different types of inputs, demonstrating strong performance in multimodal embeddings even without fine-tuning. We propose a single modality training approach for E5-V, where the model is trained exclusively on text pairs. This method demonstrates significant improvements over traditional multimodal training on image-text pairs, while reducing training costs by approximately 95%. Additionally, this approach eliminates the need for costly multimodal training data collection. Extensive experiments across four types of tasks demonstrate the effectiveness of E5-V. As a universal multimodal model, E5-V not only achieves but often surpasses state-of-the-art performance in each task, despite being trained on a single modality.

1 Introduction

E5-V adapts MLLMs for universal multimodal embeddings by prompting them to unify modalities, then trains on text pairs alone. It reports competitive performance across text-image retrieval, composed image retrieval, sentence embeddings, and image-image retrieval.

  • E5-V directly adapts MLLMs rather than CLIP-like models to represent universal multimodal embeddings.The framework leverages MLLMs’ multimodal understanding, interleaved-input capability, and stronger language understanding and reasoning.Source: §1 · p. 2
  • Prompt-based representation instructs MLLMs to map multimodal inputs into words, initially unifying their embeddings and removing the modality gap [LZK+22].Source: §1 · p. 2
  • Text-pair-only training achieves robust multimodal embeddings while eliminating multimodal data collection and reducing training cost by removing unused visual components.The authors report that text pairs can outperform image-text pairs for multimodal embedding training.Source: §1 · p. 2
  • E5-V achieves competitive performance across text-image retrieval, composed image retrieval, sentence embeddings, and image-image retrieval.Source: §1 · p. 2
  • The study shows MLLMs can represent multimodal inputs without fine-tuning, while E5-V transfers single-modality representation ability through task-specific prompts.Source: §1 · p. 2

2 Related Work

Prior multimodal embedding systems rely on modality encoders, alignment stages, or interleaved training data, but face scaling, language-understanding, fusion, and data-collection challenges. E5-V’s single-modality training addresses these constraints by using text-pair contrastive learning and removing unused MLLM components during training.

2.1 Multimodal Large Language Models

  • MLLMs typically combine an LLM, a modality encoder, and a projector that connects raw multimodal inputs to the language model.Source: §2.1 · p. 3
  • Training multimodal embedding models is constrained by large batch-size requirements, with CLIP requiring 32k samples versus 2k for text embedding models such as E5.Source: §2.1 · p. 3

2.2 Multimodal Embeddings

  • CLIP aligns separate image and text encoders effectively for text-image retrieval but struggles with complicated text and interleaved visual-language inputs [RKH+21] [ZZD+24] [LROTG21].Source: §2.2 · p. 3
  • Universal multimodal approaches based on CLIP can harm text-image retrieval, poorly support zero-shot composed image retrieval, and require large interleaved datasets.Collecting such data may require complex annotation or GPT-4 synthesis [ZLX+24].Source: §2.2 · p. 3
  • E5-V’s single-modality training removes the modality encoder and projector, using only text-pair contrastive learning to improve multimodal embeddings.Source: Figure 2 · p. 4

3 E5-V

E5-V first uses prompts to align image and text representations by semantic meaning rather than modality, then transfers text embedding capability through contrastive training on text pairs. This design supports interleaved inputs while avoiding visual training inputs.

3.1 Unifying Multimodal Embeddings

  • Without fine-tuning, MLLM image and text embeddings still show a clear modality gap despite sharing an encoder.The authors visualize LLaVA-NeXT-8B representations of COCO images and captions with PCA.Source: §3.1 · p. 4
  • Prompting MLLMs to compress image and text meanings into next-token embeddings removes the modality gap and aligns semantically related inputs.For example, image and caption representations of a plane approach token embeddings such as “Plane,” “Air,” “Flying,” and “Above.”Source: §3.1 · p. 4

3.2 Single Modality Training

  • Single-modality training transfers text representation capability to multimodal embeddings because the prompt-based method places modalities in one embedding space.The approach uses no visual or interleaved inputs and avoids multimodal training-data collection.Source: §3.2 · p. 5
  • E5-V trains the MLLM with contrastive learning on sentence pairs, removes the visual encoder and projector, and uses NLI sentence pairs unrelated to multimodal tasks.Source: §3.2 · p. 5
  • Single-modality training achieves better multimodal retrieval performance than multimodal training while significantly reducing training cost.The objective uses temperature τ and batch size N in contrastive learning.Source: §3.2 · p. 5

4 Experiments

E5-V is evaluated as a single zero-shot universal multimodal embedding model across four tasks, using prompts and text-pair training. It achieves competitive or stronger performance across text-image, composed image, image-image, and sentence embedding evaluations.

4 Experiments

  • All four evaluations use the same E5-V model in a zero-shot setting without dataset-specific fine-tuning, while training uses text pairs and prompt-based last-token embeddings.The experimental backbone is LLaVA-NeXT-8B with a frozen CLIP ViT-L visual encoder, trained on about 273k NLI sentence pairs.Source: §4 · p. 5, §4.2 · p. 7

4.1 Text-Image Retrieval

  • 12.2% on Flickr30K and 15.0% on COCO: E5-V outperforms CLIP ViT-L on image retrieval at Recall@1 despite training only on text pairs.E5-V uses the same frozen CLIP ViT-L visual encoder as CLIP and also outperforms strong baselines on zero-shot text-image retrieval.Source: §4.1 · pp. 5–6

4.2 Composed Image Retrieval

  • 8.50% on CIRR Recall@1 and 10.07% on Recall@5: E5-V outperforms iSEARLE-XL without textual inversion or annotation.On FashionIQ, it also exceeds iSEARLE-XL by 2.56% on Recall@10 and 4.24% on Recall@50.Source: §4.2 · p. 7, Table 2 · p. 7, Table 3 · p. 7

4.3 Image-Image Retrieval

  • E5-V correctly represents text rendered as images and shows outstanding image-image retrieval results on I2I-Flickr30K and I2I-COCO.Baselines drop significantly because separate visual and language encoders struggle to understand textual information through images.Source: §4.3 · p. 8, Table 4 · p. 8

4.4 Sentence Embeddings

  • E5-V achieves the best performance on seven STS sentence-embedding tasks against SimCSE-RoBERTa [GYC21], PromptRoBERTa [JJH+22], SGPT, ST5-Enc [NÁC+21], and PromptEOL [JHL+23].The result demonstrates strong semantic representation of textual inputs despite E5-V being designed as a universal multimodal model.Source: §4.4 · p. 8, Table 5 · p. 8

5 Discussion

The discussion shows that prompt-based representation and single-modality training improve multimodal embedding quality while enabling efficient, instruction-sensitive use of E5-V.

5.1 Effect of the Representation Method

  • Our representation method significantly improves all tasks over Last and Prompt, including settings with and without fine-tuning.Prompt still drops notably on visual-input tasks despite differing from the proposed template by one word.Source: §5.1 · p. 8, Table 6 · p. 9

5.2 Effect of Single Modality Training

  • Single-modality training outperforms multimodal training across tasks and reduces training time from 34.9 hours to 1.5 hours on 32 V100 GPUs.It uses only 32 maximum text tokens and removes the visual encoder during training.Source: §5.2 · p. 9, Table 7 · p. 9

5.3 Zero-shot Instruction Following Ability on Multimodal Embeddings

  • E5-V follows unseen, task-specific instructions zero-shot, supporting detailed prompts for visual styles and interactions between images and language.FashionIQ prompts can specify product styles, while prompts can express edits such as changing a shirt’s style.Source: §5.3 · p. 9

6 Conclusion

E5-V adapts MLLMs into a universal multimodal embedding model that accurately represents interleaved visual and language inputs. Prompt-based representation unifies modalities, while single-modality training yields strong performance across four task types.

6 Conclusion

  • E5-V unifies multimodal representations through prompting and achieves strong performance on text-image retrieval, composed image retrieval, image-image retrieval, and sentence embeddings.The model is trained with single-modality data, and ablations validate the proposed method.Source: §6 · p. 9

A Implementation of Text Rendering in Image-Image Retrieval

The implementation renders text as an image for image-image retrieval, using PIL, a fixed font and resolution, and automatic line breaking for long text.

A Implementation of Text Rendering in Image-Image Retrieval

B Distruibution of Image and Text Embeddings with Different Representation Methods

The appendix visualizes how three representation methods distribute image and text embeddings, highlighting the modality-gap behavior of the proposed method.

B Distruibution of Image and Text Embeddings with Different Representation Methods

C Multimodal Embeddings with Different MLLMs

E5-V maintains strong performance across different MLLMs using the same prompts, with LLaVA-NeXT performing best when fine-tuned.

Related papers

Loading 2407.12580v1…