Source-linked AI summary
UniIR: Training and Benchmarking Universal Multimodal Information Retrievers
Cong Wei, Yang Chen, Haonan Chen, Hexiang Hu, Ge Zhang, Jie Fu, Alan Ritter, Wenhu Chen
TL;DR
Multimodal IR systems are often restricted to homogeneous, task-specific settings, motivating a retriever that can handle diverse queries, targets, and retrieval intentions. UniIR combines instruction-guided universal retrieval with the M-BEIR benchmark, and instruction tuning improves performance while supporting generalization across retrieval tasks and domains.
Problem
Existing multimodal IR systems are limited by homogeneous, task-specific retrieval settings that do not accommodate diverse cross-domain and cross-modal user needs.
Method
UniIR trains a single instruction-guided retriever on M-BEIR, which unifies 10 diverse datasets and specifies retrieval intent, domain, query modality, and target modality.
Results
Instruction tuning significantly improves same-architecture baselines on M-BEIR, increasing average Recall@5 by 12.8 and 10.9.
Takeaways & Limitations
UniIR provides a unified framework and M-BEIR benchmark for training and evaluating multimodal retrievers across retrieval tasks and domains.
Abstract
from arXiv · showhide
Existing information retrieval (IR) models often assume a homogeneous format, limiting their applicability to diverse user needs, such as searching for images with text descriptions, searching for a news article with a headline image, or finding a similar photo with a query image. To approach such different information-seeking demands, we introduce UniIR, a unified instruction-guided multimodal retriever capable of handling eight distinct retrieval tasks across modalities. UniIR, a single retrieval system jointly trained on ten diverse multimodal-IR datasets, interprets user instructions to execute various retrieval tasks, demonstrating robust performance across existing datasets and zero-shot generalization to new tasks. Our experiments highlight that multi-task training and instruction tuning are keys to UniIR's generalization ability. Additionally, we construct the M-BEIR, a multimodal retrieval benchmark with comprehensive results, to standardize the evaluation of universal multimodal information retrieval.
1. Introduction
Existing multimodal IR systems largely target homogeneous, task-specific retrieval settings, limiting support for diverse cross-domain and cross-modal user needs. UniIR addresses this gap with a unified instruction-guided retriever and the M-BEIR benchmark.
- Motivation: Existing multimodal IR literature focuses mainly on homogeneous formats and often single domains, despite diverse user information needs.Examples include retrieving Flickr images from captions and news images from titles.
- Motivation: Task-specific fine-tuning of CLIP-based systems does not adequately support varied retrieval demands across domains, modalities, and tasks.Users may search with text, images, or combined inputs for different target types.
- UniIR Framework: UniIR learns a single retriever that follows instructions to retrieve from heterogeneous candidate pools across multimodal tasks.The framework is trained using multimodal fusion mechanisms on M-BEIR instances.
- M-BEIR: M-BEIR unifies 10 diverse datasets spanning 8 multimodal retrieval tasks into a common benchmark formulation.The benchmark supports training and evaluation of universal multimodal retrievers.
- Contributions: UniIR models are introduced as universal retrievers trained on M-BEIR, providing a baseline for future multimodal retrieval research.The benchmark also evaluates zero-shot performance of existing vision-language models.
2. UniIR Framework
UniIR unifies heterogeneous multimodal retrieval by conditioning a single retriever on natural-language task instructions. It supports multiple fusion designs that combine query and candidate modalities into representations scored for retrieval.
- Unified retrieval formulation: UniIR models accept text, image, or image-text queries and retrieve text, image, or image-text candidates according to an instruction.The instruction specifies the retrieval intent, target type, and relevant domain.
- Unified retrieval formulation: The unified retriever scores heterogeneous candidates by maximizing dot-product similarity between instruction-conditioned query and candidate representations.The candidate pool may contain millions of items across diverse modalities.
- Fusion mechanisms: Score-level fusion separately encodes image and text inputs, then combines their vectors with weighted sums for queries and targets.The resulting representation includes within-modality and cross-modality similarity contributions.
- Fusion mechanisms: Feature-level fusion integrates modalities during encoding through mixed-modality attention, producing a single feature vector for each query or candidate.CLIP uses a multi-modal transformer, while BLIP uses cross-attention in an image-grounded text encoder.
- Training: Four UniIR model variants are fine-tuned on M-BEIR with query-target contrastive learning and instruction prefixes added to text queries.Padding tokens represent missing image or text modalities.
3. M-BEIR Benchmark
M-BEIR standardizes universal multimodal retrieval across eight tasks and ten datasets spanning multiple domains and image sources. It provides human-authored instructions, positive and negative candidates, and large-scale evaluation pools.
- Benchmark scope: M-BEIR contains 8 multimodal retrieval tasks and 10 datasets across varied domains, including everyday imagery, fashion, Wikipedia, and news.The benchmark repurposes datasets originally created for retrieval, captioning, image similarity, and visual question answering.
- Benchmark scope: The benchmark includes 1.5 million queries and 5.6 million retrieval candidates, with human-authored instructions accompanying each task.Each query instance includes a query, instruction, relevant candidates, and potentially available irrelevant candidates.
- Instruction design: M-BEIR unifies tasks by representing both queries and candidates with text and image modalities while using instructions to specify retrieval intent.Instructions describe intent, domain, query modality, and target-candidate modality.
- Instruction design: The benchmark authors four instructions for every query in each retrieval task to diversify instruction-tuning supervision.The instructions follow a schema inspired by TART and include examples such as fashion description to matching-image retrieval.
- Evaluation: Evaluation reports recall@5 for most datasets and recall@10 for Fashion200K and FashionIQ, following prior evaluation practice.Full Recall@1, Recall@5, and Recall@10 results are provided in the appendix.
4. Experiments
Experiments evaluate multimodal retrievers on heterogeneous M-BEIR candidate pools, focusing on instruction tuning, modality errors, fusion design, and generalization. UniIR improves retrieval intention understanding and generalizes better than non-instruction-tuned and zero-shot baselines.
- Evaluation setup: Models are evaluated on M-BEIR’s 5.6 million-candidate pool using zero-shot, fine-tuned baseline, and UniIR retrievers.The evaluation also considers task-specific candidate pools and held-out datasets.
- Zero-shot retrieval: Zero-shot retrievers achieve near-zero recall on most tasks because they cannot infer retrieval intention without instructions.On MSCOCO text-to-image retrieval, all zero-shot models retrieve text instances and obtain 0% recall; BLIP2’s WebQA recall falls from 35.2% to 0% for image-text candidates.
- Instruction tuning: Instruction tuning raises average Recall@5 over multi-task baselines by 12.8 and 10.9, with the largest gains on cross-modality tasks.Within-modality tasks improve less because they do not require intent understanding.
- Error analysis: Instruction finetuning reduces wrong-modality error rates from 58.8% and 50.9% to 2.7% and 15.2%.Visualization shows zero-shot and multi-task models retrieving distracting text-only candidates, whereas UniIR retrieves candidates in the intended modalities.
- Unseen-task generalization: UniIR outperforms zero-shot retrievers on held-out datasets and generalizes better than multi-task baselines without instructions.The held-out datasets are WebQA, OVEN, and CIRR, with results averaged over CLIPSF and BLIPFF.
- Fusion architecture: Aligning fusion architecture with pre-training improves average Recall@5 by 5.7 for CLIP and 2.8 for BLIP.Randomly initialized fusion layers can overfit; on OVEN/InfoSeek, the reported task-8 comparison is 52.4 versus 46.3.
- Overall comparisons: UniIR (CLIPSF) and UniIR (BLIPFF) reach average R@5 values of 50.6% and 46.8%, compared with 37.2% for SigLIP.Against single-task tuning, UniIR (BLIPFF) improves average R@5 by 9.7% and reaches 48.6% versus 13.9% on CIRR.
5. Related Work
Related work covers multimodal information retrieval, retrieval-augmented models, and instruction tuning. UniIR extends instruction-based retrieval beyond prior work focused on text-to-text content.
- Multimodal Information Retrieval: Multimodal information retrieval has largely emphasized image-to-text matching with benchmarks such as MSCOCO and Flickr30k.Prior work includes several pre-trained vision-language models evaluated on these datasets.
- Retrieval-augmented Models: Retrieval-augmented models commonly retrieve text or multimodal content to support language-model answering and image generation.Examples include systems that retrieve multimodal content from Wikipedia for visually informed questions.
- Retrieval-augmented Models: TART is the closest cited work, but it remains focused on text-to-text retrieval rather than multimodal retrieval.This contrast motivates a retriever spanning different content modalities.
- Instruction tuning: Instruction tuning has enabled language and vision-language models to follow user instructions and generalize to unseen tasks.The related work discusses FLAN, FLAN-T5, InstructGPT, and visual instruction tuning.
6. Conclusion
The paper presents UniIR and M-BEIR as a unified framework and benchmark for instruction-guided multimodal retrieval. Results show cross-task and cross-domain generalization, while performance remains imperfect and leaves room for stronger pre-training and vision-language backbones.
- Contributions: UniIR follows natural-language instructions to perform diverse retrieval tasks across modalities, supported by the M-BEIR training and evaluation benchmark.The benchmark enables training and evaluation of universal multimodal information retrieval models.
- Conclusion and limitations: Instruction tuning generalizes across different retrieval tasks and domains, but existing model performance remains relatively far from perfect.The authors identify large-scale pre-training and stronger vision-language backbones as directions for future improvement.
Information Retrievers
UniIR combines instruction-guided multimodal retrieval across heterogeneous queries, candidates, and datasets. Experiments show instruction tuning improves modality selection, while model fusion affects specialized tasks and zero-shot baselines vary by domain.
- Retrieval behavior: Instruction tuning helped UniIR retrieve candidates in the intended modalities, whereas multi-task baselines often matched query semantics without understanding task intent.Baselines were distracted by text mentioning entities such as “Obama,” “Gates,” or “Saint Peter.”
- Fusion mechanisms: BLIPFF outperformed CLIPSF on composed image retrieval, where text modifies a reference image, consistent with cross-attention merging image and text embeddings.The reported examples show BLIPFF-based models retrieving the target candidate accurately.
- Task analysis: InfoSeek was harder than OVEN: only UniIR BLIPFF retrieved the target within the top five, while CLIPSF found related lakes without the desired answer.OVEN mainly focuses on visual entity recognition, whereas InfoSeek requires details from Wikipedia articles.
- Unified retrieval: UniIR supports heterogeneous retrieval through human-authored instructions specifying task intent, with queries and candidates represented as text, images, or image-text pairs.M-BEIR instances include a query, positive candidates, possible negative candidates, and an instruction describing the retrieval intent.
- Zero-shot retrieval: Zero-shot SigLIP performed best on average, while BLIP varied substantially across domains, performing relatively well on MSCOCO but worse on news, fashion, and Wikipedia.The zero-shot evaluation used CLIP, SigLIP, BLIP, and BLIP2 as baseline feature extractors with score-level fusion.