PaperlayerView exact revision on arXiv

Source-linked AI summary

MagicLens: Self-Supervised Image Retrieval with Open-Ended Instructions

Kai Zhang, Yi Luan, Hexiang Hu, Kenton Lee, Siyuan Qiao, Wenhu Chen, Yu Su, Ming-Wei Chang

arXiv:2403.19651v2cs.CVcs.AIcs.CLcs.IRcs.MM

TL;DR

Image retrieval must support diverse semantic relations that visual similarity and restricted instruction schemes do not capture well. MagicLens learns open-ended image-instruction-to-image retrieval from naturally paired web images and foundation-model-generated instructions, achieving strong benchmark performance with a much smaller model and supporting complex search intents. Its scope includes potential misuse for inappropriate image searches despite filtering efforts.

  • Problem

    Existing image retrieval methods have limited ability to express and handle diverse semantic relations beyond visual similarity, while many models rely on restricted domains or predefined instruction templates.

  • Method

    MagicLens trains lightweight dual encoders on web-mined image pairs whose implicit relations are made explicit through foundation-model-generated instructions.

  • Results

    MagicLens matches or exceeds prior state of the art across eight retrieval benchmarks, including outperforming it with a 50× smaller model on multiple benchmarks and satisfying complex intents in a 1.4M-image human evaluation.

  • Takeaways & Limitations

    Naturally occurring web image pairs provide effective self-supervised training signals for retrieval models that follow open-ended instructions and capture diverse search intents.

  • Takeaways & Limitations

    Despite filtering explicit and offensive training images, MagicLens’s image-relationship understanding could still be misused for inappropriate image searches.

Abstract

from arXiv · show

Image retrieval, i.e., finding desired images given a reference image, inherently encompasses rich, multi-faceted search intents that are difficult to capture solely using image-based measures. Recent works leverage text instructions to allow users to more freely express their search intents. However, they primarily focus on image pairs that are visually similar and/or can be characterized by a small set of pre-defined relations. The core thesis of this paper is that text instructions can enable retrieving images with richer relations beyond visual similarity. To show this, we introduce MagicLens, a series of self-supervised image retrieval models that support open-ended instructions. MagicLens is built on a key novel insight: image pairs that naturally occur on the same web pages contain a wide range of implicit relations (e.g., inside view of), and we can bring those implicit relations explicit by synthesizing instructions via foundation models. Trained on 36.7M (query image, instruction, target image) triplets with rich semantic relations mined from the web, MagicLens achieves results comparable with or better than prior best on eight benchmarks of various image retrieval tasks, while maintaining high parameter efficiency with a significantly smaller model size. Additional human analyses on a 1.4M-image unseen corpus further demonstrate the diversity of search intents supported by MagicLens. Code and models are publicly available at https://open-vision-language.github.io/MagicLens/.

1. Introduction

MagicLens addresses the difficulty of expressing and retrieving diverse semantic relations between images by learning from naturally occurring web image pairs and open-ended instructions. Its lightweight models match or exceed prior state of the art across multiple retrieval benchmarks while supporting complex search intents.

  • Motivation: MagicLens targets image retrieval settings where users may seek relations such as other attractions or interior views, not merely visual similarity.These intents require understanding both image content and the semantic relation expressed by the instruction.
  • Approach: MagicLens uses naturally co-occurring web image pairs as self-supervised signals and foundation models to convert their implicit relations into instructions.The resulting training data contains 36.7M query-image, instruction, target-image triplets.
  • Approach: MagicLens comprises lightweight dual encoders that jointly represent an image and instruction for retrieving a target image.The models are trained on triplets mined from web pages and curated with multimodal and language foundation models.
  • Results: MagicLens achieves results comparable with or better than prior state-of-the-art methods on eight multimodality-to-image and image-to-image retrieval benchmarks.It also retains or improves the text-to-image performance of its underlying single-modality encoders.
  • Results: With a 50× smaller model size than prior state-of-the-art methods, MagicLens outperforms them on CIRCO, Domain Transfer ImageNet, and GeneCIS.Human evaluation on a 1.4M-image unseen retrieval pool further finds that it satisfies complex and beyond-visual search intents where the prior state of the art fails.

2. Related Work

Prior retrieval research relies heavily on restricted visual relations, manually designed instructions, or synthetic pairs, limiting the breadth and efficiency of open-ended image retrieval. MagicLens instead mines natural web-page image relations and expands them into large-scale instructions using multimodal and language models.

  • Data Construction: Its data pipeline groups and cleans web-page images, annotates metadata with multimodal models, filters unqualified pairs, and generates open-ended instructions with language models.Metadata includes image content and background information used to connect paired images precisely.
  • Composed Image Retrieval: Existing composed-image-retrieval benchmarks usually begin with visually similar image pairs and therefore limit the richness of their relations.Prior methods also use manually defined relation domains, image-caption data, or predefined instruction templates.
  • Composed Image Retrieval: MagicLens mines naturally co-occurring image pairs from web pages, which can encode relations ranging from visual similarity to more nuanced associations.This provides a broader self-supervised signal than constructing pairs only from visually similar images.
  • Retrieval with Instruction: Compared with prior instruction-tuning efforts that use manually written task prefixes at hundred-scale, MagicLens uses million-scale instructions for multimodal retrieval.The approach is designed to cover diverse image relations rather than a small set of task identifiers.

3. MagicLens

MagicLens constructs open-ended image-retrieval training data from naturally co-occurring web images and trains lightweight dual encoders to match instructed queries with target images. Its pipeline combines metadata-based filtering, foundation-model instruction generation, multimodal encoding, and contrastive learning.

  • Data Construction: 36.7M training triplets are built from image pairs co-occurring on web pages, using LMMs and LLMs to expose their implicit semantic relations.Images are grouped, cleaned, enriched with Alt-texts, ICA labels, and captions, then paired and filtered using visual and non-visual relevance scores before instruction generation.
  • Instruction Generation: Open-ended instructions are generated from paired-image metadata with prompting techniques that connect the query image to its target image.The generation process uses instruction prompting, few-shot demonstrations, and chain-of-thought prompting with PaLM2.
  • Model Design: MagicLens uses a shared-parameter dual encoder with self-attention for modality integration and a multi-head attention pooler that produces a single matching embedding.The query combines an image and instruction, while the target image is paired with an empty text string; MagicLens-B and MagicLens-L use base and large checkpoints.
  • Model Training: The model is trained with contrastive loss against other targets in the batch and the query image encoded with empty text as an additional hard negative.All query and target negatives in the batch are used to increase the number of negative examples; similarity is cosine-based with temperature scaling.

4. Experiments

The experiments evaluate MagicLens across multimodality-to-image, image-to-image, and text-to-image retrieval settings using multiple benchmarks and a one-checkpoint protocol. MagicLens achieves strong benchmark performance, scales with model size, generalizes across image-to-image tasks, and improves text-to-image retrieval while causing only marginal image-to-text drops.

  • Experiment Setup: The evaluation covers composed image retrieval, domain transfer retrieval, and conditional image similarity across five benchmarks, including CIRCO with over 120K indexed natural images.CIRCO uses multiple ground truths per query and mean Average Precision, whereas the other tasks include domain or keyword-conditioned retrieval settings.
  • Multimodality-to-Image Retrieval: mAP@5 rose from 12.6 to 34.1 on CIRCO and R@10 from 12.9 to 48.2 on DTIN for CoCa-based MagicLens-L, while MagicLens outperformed prior state of the art across four open-domain benchmarks.These comparisons were reported with comparable model size.
  • Multimodality-to-Image Retrieval: MagicLens-L generally improved over MagicLens-B across five benchmarks, indicating that the constructed data benefits larger models and the dual-encoder architecture scales.The authors attribute scalability to the simple dual-encoder model architecture and contrastive loss.
  • Image-to-Image Retrieval: Using the same checkpoints across TU-Berlin, Sketchy, and QuickDraw, MagicLens significantly outperformed prior methods that used separate checkpoints for each benchmark.This one-checkpoint evaluation demonstrates cross-benchmark generalization in zero-shot sketch-based image retrieval.
  • Text-to-Image Retrieval: MagicLens training consistently improved text-to-image retrieval on Flickr30k and MSCOCO, while image-to-text retrieval showed marginal drops.The reported improvements occurred across all text-to-image metrics on both datasets.

5. Analysis

Analysis shows that naturally occurring web image pairs and template-free instructions improve retrieval quality, while MagicLens achieves strong parameter efficiency and supports complex search intents.

  • Comparison to Existing Training Data: Using naturally occurring image pairs and template-free instructions produces stronger retrieval models than IP2P-based alternatives across five benchmarks.MagicLens + IP2P also outperforms CompoDiff + IP2P and achieves better results than comparable-sized SOTA baselines on CIRCO, DTIN, and GeneCIS.
  • Data Scaling: Increasing training data from 0.2M to 36.7M triplets improves average performance, with especially strong gains before 10M triplets.The scaling experiment trains CoCa-based MagicLens-B on seven dataset sizes and evaluates five benchmarks.
  • Impacts of Instructions during Training: Template-free instructions consistently outperform template-based instructions across all benchmarks when training uses the same 1M image pairs.The result supports naturally expressed, diverse instructions for learning image relations and following instructions.
  • Model Size vs. Performance: MagicLens-B (267M parameters) outperforms CIReVL (14.6B) on GeneCIS, CIRCO, and DTIN, despite being 50× smaller.The advantage is especially significant on DTIN, attributed to parameter sharing and the strength of the constructed data.
  • Ablation on Contrastive Loss: Removing query images as hard negatives lowers performance across all benchmarks, especially CIRR, CIRCO, and DTIN.Without query negatives, the model often ranks the query image itself, although the full model can still retrieve identical images for unseen instructions.
  • Ablation on Model Architecture: Self-attention outperforms the explored cross-attention architecture, and freezing pretrained encoders performs worse than fully training MagicLens.These results indicate that adding layers above single-modality encoders is insufficient for the strongest performance.
  • Retrieval on 1.4M Open-Domain Image Corpus: MagicLens outperforms LinCIR on complex instructions and beyond-visual instructions, with human preference scores of 61.3 vs. 24.0 and 80 vs. 4.7, respectively.LinCIR handles simple instructions but nearly fails on beyond-visual instructions, whereas MagicLens handles diverse search intents.
  • Qualitative Study: MagicLens can interpret multiple conditions and temporal relations, but ambiguous instructions may yield qualified rather than preferred results.The qualitative study uses a 1.4M-image holdout index and includes examples involving cars, bags, muffins, and 3D anatomy.

6. Conclusion

MagicLens follows open-ended text instructions and achieves strong retrieval performance with a much smaller model than prior SOTA methods. Human evaluation shows especially strong handling of complex and beyond-visual search intents.

  • 6. Conclusion: MagicLens achieves better results than prior SOTA methods on multiple benchmarks, including CIRCO, GeneCIS, and DTIN, despite being 50× smaller.Human evaluation on a 1.4M-image retrieval pool indicates strong satisfaction of diverse open-ended search intents, especially complex and beyond-visual ones.

Impact Statement

MagicLens mines naturally occurring image pairs to train open-ended instruction-following retrieval models. The paper notes both broad application potential and a risk that image-relationship understanding could be misused.

  • Impact Statement: MagicLens may support diverse search scenarios and other vision-language tasks through open-ended instruction-following retrieval.The supplementary material lists implementation details, baselines, full results, and additional qualitative studies.
  • Impact Statement: Despite filtering explicit and offensive images from training data, MagicLens could still be misused for inappropriate image searches.The paper calls for careful consideration and mitigation strategies.

A. Implementation Details

The implementation constructs large-scale training data from web images and metadata, then trains MagicLens with lightweight self-attention layers and multimodal encoders.

  • A. Implementation Details: The data pipeline groups web images by identical URLs, removes near-duplicate images using CLIP similarity, and retains images at least 288×288 pixels.Groups with substantial duplication are also filtered by randomly removing one group.
  • A. Implementation Details: Instruction generation uses expanded metadata, including alt-text, image-content annotations, captions, and language-model processing with filtering for rare, sentiment-related, and pornographic content.The pipeline uses language and multimodal models to construct instructions from image metadata.
  • A. Implementation Details: The final pretraining set contains 36,714,118 triplets, and MagicLens adds four randomly initialized self-attention layers plus an attention-pooling layer over vision and language encoders.CoCa-based training uses 288×288 images with 18×18 patches.
  • A. Implementation Details: The study compares MagicLens with mapping-based retrieval baselines that convert reference images into pseudo-word tokens for text-to-image retrieval.Listed baselines include PALARVA, Pic2Word, SEARLE, ContextI2W, and LinCIR.
  • A. Implementation Details: The supplementary material provides detailed PaLM2 prompts and full comparisons between MagicLens trained on IP2P data and the constructed data.Tables 10 and 11 document prompt generation and same-scale training-data comparisons.

C.1. Results on Five Multimodality-to-Image Benchmarks

MagicLens outperforms prior methods across multimodality-to-image benchmarks, even when competing systems use larger encoders or elaborate retrieval pipelines.

  • MagicLens achieves better results than prior methods on the reported CIR, DT, and GeneCIS benchmarks despite some baselines using larger encoders or LLM/LMM retrieval pipelines.

C.2. Data Training Comparison

The section compares MagicLens training data against IP2P data and reports improved text-to-image retrieval after training.

  • MagicLens-B is compared when trained on IP2P data versus the authors’ data, with both settings evaluated at a 1M scale.
  • Text-to-image retrieval performance significantly improves for both base and large models, while image-to-text retrieval drops marginally.
  • Figure 10 illustrates template-based and template-free instructions describing the same image pair.

D. More Qualitative Study

Qualitative retrieval examples show MagicLens handling brand, multi-object, real-world-scene, and numerical relations that challenge LinCIR.

  • MagicLens retrieves same-brand bags without shared visible brand logos, whereas LinCIR fails on a house-and-gavel query where MagicLens finds relevant top-ranked results.
  • The gazebo example indicates that MagicLens can understand simple numerical relations.
  • The qualitative comparison attributes LinCIR’s failure on multi-object queries to the limited representation capacity of a single pseudo token.
  • The appendix reports full benchmark tables for FIQ, CIRR, CIRCO, DTIN, and GeneCIS, plus zero-shot image-text retrieval results.
Loading 2403.19651v2…