PaperlayerView exact revision on arXiv

Source-linked AI summary

VISTA: Visualized Text Embedding For Universal Multi-Modal Retrieval

Junjie Zhou, Zheng Liu, Shitao Xiao, Bo Zhao, Yongping Xiong

arXiv:2406.04292v1cs.IRcs.CLcs.CV

TL;DR

Text-oriented retrievers have limited ability to process visual information, while existing vision-language models are limited in text representation and joint image-text representation. VISTA combines a strong text encoder with visual tokens, generated composed image-text data, and staged training; it achieves state-of-the-art average performance across tasks, with more than 9% improvement on Recall@5.

  • Problem

    Most dense retrievers are text-oriented, while existing vision-language models are limited in text representation and joint representation of composed image-text data.

    Source: §1 · p. 1

  • Method

    VISTA extends a well-trained text encoder with visual tokens and trains it in stages using weakly labeled cross-modal data and generated composed image-text datasets.

    Source: §1 · pp. 1–2, §3.3 · p. 5

  • Results

    VISTA achieves state-of-the-art average performance across all tasks, with more than 9% improvement on Recall@5.

    Source: §4.1 · p. 6

  • Takeaways & Limitations

    Without task-specific optimization, VISTA can outperform or match the leading approach in every downstream evaluation scenario.

    Source: §1 · p. 2

    Generated IT2T images may have limited style diversity, and feeding all image tokens at uniform sequence length may increase computational load.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Multi-modal retrieval becomes increasingly popular in practice. However, the existing retrievers are mostly text-oriented, which lack the capability to process visual information. Despite the presence of vision-language models like CLIP, the current methods are severely limited in representing the text-only and image-only data. In this work, we present a new embedding model VISTA for universal multi-modal retrieval. Our work brings forth threefold technical contributions. Firstly, we introduce a flexible architecture which extends a powerful text encoder with the image understanding capability by introducing visual token embeddings. Secondly, we develop two data generation strategies, which bring high-quality composed image-text to facilitate the training of the embedding model. Thirdly, we introduce a multi-stage training algorithm, which first aligns the visual token embedding with the text encoder using massive weakly labeled data, and then develops multi-modal representation capability using the generated composed image-text data. In our experiments, VISTA achieves superior performances across a variety of multi-modal retrieval tasks in both zero-shot and supervised settings. Our model, data, and source code are available at https://github.com/FlagOpen/FlagEmbedding.

1 Introduction

VISTA addresses the gap between text-oriented retrievers and multimodal data with an embedding model designed to combine strong text encoding and image understanding. Its architecture, composed image-text data pipelines, and staged training support multimodal retrieval, with strong results across zero-shot and supervised settings.

1 Introduction

  • The architecture adds visual tokens from an image encoder to a well-trained text encoder, enabling in-depth text-image fusion while preserving original text-embedding performance.The text encoder remains fixed as visual tokens are incorporated.Source: §1 · pp. 1–2
  • Two automatic pipelines generate composed image-text datasets for training, covering retrieval scenarios where either the query or candidate contains image-text pairs.The pipelines are intended to provide large-scale, high-quality training data.Source: §1 · p. 2
  • The two-stage training first aligns visual token embeddings with the text encoder using massive weakly labeled cross-modal data, then builds multimodal representation capability with generated composed image-text data.Source: §1 · p. 2
  • VISTA outperforms or matches leading approaches in every downstream evaluation scenario without task-specific optimization.Its performance can improve further with continued fine-tuning for corresponding tasks.Source: §1 · p. 2

2 Related Work

General text embedding models have strong text-retrieval performance but are limited on multimodal data. Existing multimodal approaches often lack deep image-text fusion, focus on specific tasks, or use text encoders that lag recent general embedding models.

2.1 General Text Embedding

  • General text embedding models demonstrate generalizability and robust text-retrieval performance, but have limitations handling multimodal data as multimodal retrieval grows in practice.Source: §2.1 · p. 2

2.2 General Multi-Modal Embedding

  • Existing multimodal embedding models often lack deep image-text fusion or target specific tasks, while CLIP and BLIP text embeddings may underperform recent general models on text-heavy documents.Marvel uses pretrained text embeddings and a visual plugin but is task-specific to multimodal document retrieval and does not cover tasks such as composed image retrieval.Source: §2.2 · p. 2

3 VISTA Model

VISTA extends a pretrained text encoder to process image tokens and interleaved image-text data. It pairs this architecture with generated composed image-text datasets and two-stage training.

3.1 Model Architecture

  • VISTA uses ViT as an image tokenizer for a frozen pretrained text encoder, which encodes image tokens and interleaved image-text sequences into embeddings.Source: §3.1 · p. 3

3.2 Data Construction

  • Two data-generation pipelines target retrieval cases where either the query or candidate is composed of image and text, addressing the scale limits of human-annotated data.Source: §3.2 · p. 3

3.2.1 Image&Text To Image (IT2I) Dataset

  • The approach generates multiple editing instructions per source image, versus InstructPix2Pix’s single instruction; different edited images serve as hard negatives, and CLIP filtering yields 307K query-candidate pairs with hard negatives, discouraging image-to-image matching.Source: §3.2.1 · p. 4

3.2.2 Text To Image&Text (T2IT) Dataset

  • The T2IT pipeline generates over 213K query-document triples from image captions, and its data shows superior generalization to manually annotated WebQA when training multi-modal embedding models.Source: §3.2.2 · pp. 4–5

3.3 Two-Stage Training

  • Stage 1 aligns ViT image tokens with the text encoder using uni-modal cross-modal data; Stage 2 trains on generated IT2I and T2IT data for multi-modal encoding.Source: §3.3 · p. 5
  • 600 steps on generated multi-modal training data are reported sufficient to equip VISTA with robust multi-modal embedding capabilities.Source: §3.3 · p. 5

4 Experimental Results

Across multi-modal benchmarks, VISTA leads average zero-shot Recall@5 and remains robust after supervised fine-tuning; ablations examine generated stage-2 data and fusion.

4.1 Zero-Shot Retrieval Performance

  • Evaluation covers five datasets across four multi-modal retrieval tasks, using Recall@5 and dense retrieval over each dataset's entire corpus for zero-shot testing.The experiments include zero-shot and supervised evaluations, plus ablations of model design and stage-2 datasets.Source: §4.1 · p. 5, §4 · p. 5, Table 2 · p. 6
  • VISTA achieves state-of-the-art average zero-shot performance across tasks, improving Recall@5 by more than 9%.It significantly outperforms all baselines on WebQA, OVEN-QS, and ReMuQ, while its CIRR and FashionIQ results are slightly below task-specific pic2word.Source: Table 3 · p. 6, §4.1 · p. 6

4.2 Supervised Fine-Tuning Performance

  • Supervised fine-tuning on WebQA, CIRR, and ReMuQ shows VISTA's robustness and adaptability without task-specific modifications or additional training data.All VISTA parameters are trained during fine-tuning.Source: §4.2 · p. 6

4.2.1 Fine-Tuning Performance on WebQA

4.2.2 Fine-Tuning Performance on CIRR

  • 76.1% Recall@5, 75.7% Rsub@1, and 75.9% average on the CIRR test set place VISTA on par with state-of-the-art composed-image-retrieval models.VISTA uses no task-specific module.Source: Table 5 · p. 7, §4.2.2 · p. 7

4.2.3 Fine-Tuning Performance on ReMuQ

4.3.1 The Impact of Stage-2 Training Data

  • Dual-task training with IT2I and T2IT yields VISTA, which performs best on four of five benchmarks among the tested data configurations.IT2I improves CIRR, FashionIQ, and ReMuQ versus VISTAS1; T2IT improves WebQA and OVEN-QS, and outperforms WebQA training except on ReMuQ. IT2I includes generated hard negatives, which the authors say can foster understanding of image-text interplay and reduce reliance on image similarity over semantic correlation.Source: §4.3.1 · pp. 7–8, Table 7 · p. 8

4.3.2 Multi-Modal Fusion Methods

  • VISTA significantly outperforms score-fusion VISTA-SF, supporting its advantage in encoding interleaved text and visual sequences.VISTA-SF independently encodes image and text, then adds their embeddings element-wise; the authors attribute its weaker result to lack of deep image-text integration.Source: §4.3.2 · p. 8

5 Conclusion

VISTA combines a flexible text encoder architecture, generated composed image-text data, and two-stage training for universal multimodal retrieval. It achieves superior performance across varied retrieval tasks in zero-shot and supervised settings.

5 Conclusion

  • Its flexible architecture fuses text and image data while preserving the performance of general text embedding models.The conclusion identifies in-depth fusion and retention of text embedding performance as architectural contributions.Source: §5 · p. 8
  • VISTA uses two data-generation strategies and a two-stage training algorithm to develop multimodal representation capability.The stages align visual token embeddings with the text encoder, then train on generated composed image-text data.Source: §5 · p. 8
  • VISTA demonstrates superior performance across varied multimodal retrieval tasks in zero-shot and supervised fine-tuning settings.The conclusion also reports that it outperforms or matches leading approaches across downstream evaluation scenarios without task-specific optimization.Source: §5 · pp. 8–9

Limitations

The authors identify limited image-style diversity in the IT2T dataset and computational load from feeding all image tokens at a uniform sequence length as areas for refinement.

Limitations

  • Stable Diffusion generation may limit the range of image styles in the IT2T dataset.The authors identify greater image-style diversity as a potential refinement.Source: Limitations · p. 9
  • Feeding all image tokens directly into the text encoder may heighten computational load because sequence length is uniform.Source: Limitations · p. 9
  • Variable-length image-token sequences are proposed as a potential way to reduce sequence lengths and improve computational efficiency.Source: Limitations · p. 9

Ethics Statement

The authors report screening all training data for harmful content, while acknowledging that screening may not be comprehensive and discouraging use with sensitive content.

Ethics Statement

  • Training data, including LAION-5B and the authors’ own multimodal dataset, underwent screening to remove harmful content.Source: Ethics Statement · p. 9
  • The authors cannot fully guarantee that harmful-content screening was comprehensive or free of omissions.Source: Ethics Statement · p. 9
  • The authors strongly discourage using VISTA to encode or retrieve sensitive content.Source: Ethics Statement · p. 9

A More Details of Data Construction

The appendix describes two dataset-construction processes: image-editing instructions for IT2T and a two-step article-and-query process for T2IT.

A More Details of Data Construction

B More Training Details of VISTA

VISTA uses a two-stage training schedule, with different data and batch settings in each stage. The training setup also specifies a shared contrastive temperature and a decaying learning rate.

B More Training Details of VISTA

  • Stage one uses FLIP masking on 50% of image tokens for 116K steps, followed by 48K steps of unmasked tuning, with batch size 16K.The second stage uses three hard negatives from IT2I, runs for 600 steps, and has batch size 1920.Source: B More Training Details of VISTA · p. 12
  • Both stages use contrastive temperature τ=0.02 and a learning rate initialized at 2e−5 with linear decay.The learning-rate decay strategy is specified in the continuation of the training details.Source: B More Training Details of VISTA · p. 12

C Detailed Information of Benchmarks

The appendix describes the hybrid retrieval benchmarks and zero-shot baseline implementations, then reports that changing VISTA’s interleaved image-text token order does not significantly affect Recall@5.

C Detailed Information of Benchmarks

  • The five benchmarks cover four hybrid retrieval tasks, including document retrieval, composed image retrieval, knowledge retrieval with multimodal queries, and visually situated entity retrieval.WebQA’s deduplicated corpus contains 544,489 text documents and 403,277 image-caption pairs; ReMuQ is evaluated on its test split because it lacks a dev split.Source: C Detailed Information of Benchmarks · pp. 12–13
  • The benchmark appendix includes prompts for IT2T and T2IT data generation; T2IT varies article and query lengths randomly, typically about 50 and within 20 words, respectively.Zero-shot evaluation generally uses each dataset’s dev split as queries and its full corpus, except ReMuQ, which uses the test split.Source: Figure 3 · p. 12, C Detailed Information of Benchmarks · pp. 12–13, Figure 4 · p. 13

D Implementation details of Zero-Shot Baselines

  • CLIP and BLIP use score fusion for image-text pairs, while Pic2Word maps an image feature to a pseudo-language token and encodes it with the text.The text and image encoders in the score-fusion approach are denoted ΦT and ΦI; Pic2Word uses CLIP-L encoders.Source: D Implementation details of Zero-Shot… · p. 13

E The Impact of Token Order

  • Changing the interleaved order from image-text tokens to text-image tokens does not significantly affect zero-shot Recall@5.The comparison is reported in Table 8; prior experiments used image tokens followed by text tokens.Source: E The Impact of Token Order · pp. 13–14, Table 8 · p. 14

F Zero-Shot Qualitative Results

The figures show VISTA’s zero-shot retrieval examples on CIRR, FashionIQ, and WebQA. The authors note that full-corpus retrieval can return candidates that meet retrieval requirements but lack benchmark ground-truth labels.

F Zero-Shot Qualitative Results

Loading 2406.04292v1…