PaperlayerView exact revision on arXiv

Source-linked AI summary

LamRA: Large Multimodal Model as Your Advanced Retrieval Assistant

Yikun Liu, Pingan Chen, Jiayin Cai, Xiaolong Jiang, Yao Hu, Jiangchao Yao, Yanfeng Wang, Weidi Xie

arXiv:2412.01720v1cs.CV

TL;DR

Existing multimodal retrieval methods often require cumbersome task-specific fine-tuning, while LamRA equips generative LMMs with unified retrieval and reranking through staged training and lightweight LoRA modules. Across more than ten tasks, it performs robustly in supervised and zero-shot settings, including unseen tasks, exceeding UniIR-CLIP by 26.3 points in InfoSeek Recall@5.

  • Problem

    Existing methods adapt vision-language models to increasingly complex retrieval tasks through cumbersome, task-specific fine-tuning.

  • Method

    LamRA adds lightweight LoRA modules to LMMs, using two-stage retrieval training and jointly trained pointwise and listwise reranking.

  • Results

    Across more than ten retrieval scenarios, LamRA performs robustly in supervised and zero-shot settings, exceeding UniIR-CLIP by 26.3 points in InfoSeek Recall@5.

  • Takeaways & Limitations

    LamRA can extrapolate to previously unseen retrieval tasks without additional training, including complex zero-shot scenarios.

  • Takeaways & Limitations

    Applying LMMs to retrieval and reranking has high inference costs, although feature pre-extraction and low-query-rate deployment can mitigate them.

Abstract

from arXiv · show

With the rapid advancement of multimodal information retrieval, increasingly complex retrieval tasks have emerged. Existing methods predominately rely on task-specific fine-tuning of vision-language models, often those trained with image-text contrastive learning. In this paper, we explore the possibility of re-purposing generative Large Multimodal Models (LMMs) for retrieval. This approach enables unifying all retrieval tasks under the same formulation and, more importantly, allows for extrapolation towards unseen retrieval tasks without additional training. Our contributions can be summarised in the following aspects: (i) We introduce LamRA, a versatile framework designed to empower LMMs with sophisticated retrieval and reranking capabilities. (ii) For retrieval, we adopt a two-stage training strategy comprising language-only pre-training and multimodal instruction tuning to progressively enhance LMM's retrieval performance. (iii) For reranking, we employ joint training for both pointwise and listwise reranking, offering two distinct ways to further boost the retrieval performance. (iv) Extensive experimental results underscore the efficacy of our method in handling more than ten retrieval tasks, demonstrating robust performance in both supervised and zero-shot settings, including scenarios involving previously unseen retrieval tasks.

1. Introduction

LamRA repurposes large multimodal models for universal retrieval and reranking, addressing the limitations of task-specific VLM fine-tuning. Its two-stage retrieval training is evaluated across more than ten scenarios, including unseen-task generalization.

  • Motivation: Existing VLM methods rely on task-specific fine-tuning, limiting adaptation as challenging tasks such as composed image retrieval and multimodal document retrieval emerge.These tasks include composed image retrieval [34], long-text image retrieval, and image/question to multimodal document retrieval [36].
  • Method: LamRA enhances LMMs with universal retrieval and reranking capabilities by inserting lightweight LoRA modules.The framework is designed to support sophisticated retrieval and reranking within LMMs.
  • Method: LamRA-Ret uses language-only pre-training followed by multimodal instruction tuning to progressively improve retrieval performance.The first stage trains text-only embedding outputs in response to summarization prompts, while the second stage uses instruction tuning.
  • Experiments: Experiments cover more than 10 retrieval scenarios, including text-to-image, text-image-to-image, and text-image-to-text-image retrieval.The reported performance indicates that LamRA handles both simple and complex retrieval tasks effectively.
  • Results: LamRA demonstrates strong task-level generalization by performing effectively on held-out retrieval tasks without prior exposure.This evaluates transferability to previously unseen retrieval challenges.

2. Related Work

Related work progresses from traditional cross-modal retrieval and dual-encoder representation learning toward more complex, universal retrieval settings. Recent methods use benchmarks, shared language spaces, and LMMs, but limitations on complex tasks motivate LamRA.

  • Multimodal Information Retrieval: Multimodal retrieval has expanded beyond MSCOCO [29] and Flickr30K [41] cross-modal benchmarks to composed image, long-text-to-image, and image/question-to-multimodal document retrieval [2] [34] [45] [13].
  • Multimodal Representation Learning: CLIP [42] and ALIGN [15] use dual-encoder contrastive learning [38] to align images and text, but this architecture faces challenges beyond straightforward image-text alignment.
  • Multimodal Representation Learning: UniIR benchmarks eight retrieval tasks, while E5-V [18] maps images and text into a shared language hidden space and achieves zero-shot retrieval through text-pair fine-tuning [11].These approaches improve generality and adaptability, but remain limited on certain complex retrieval tasks.
  • Large Multimodal Model: LMM research increasingly aligns visual and textual modalities through visual instruction tuning for diverse vision-language tasks, including segmentation, object detection, and general vision-language applications [21] [40].
  • Large Multimodal Model: Concurrent studies [19] [28] also investigate LMMs for universal retrieval, underscoring growing interest in this direction.

3. Method

LamRA unifies multimodal retrieval and reranking through a two-stage pipeline: embedding-based retrieval produces top-K candidates, which LamRA-Rank then refines using pointwise or listwise ranking.

  • Problem Formulation: LamRA first embeds arbitrary-format queries and candidates, ranks them by cosine similarity, and passes the top-K results to a subsequent reranking stage.Queries and candidates may contain images, text, or interleaved image-text formats.
  • Architecture and Feature Extraction: The model extracts generative embeddings with Explicit One-word Limitation prompts tailored to image-only, text-only, and multimodal inputs.The architecture comprises a vision encoder, vision projector, and language model, following approaches in [17] [18].
  • Retrieval Training: LamRA-Ret uses language-only pre-training followed by multimodal instruction tuning on M-BEIR’s eight retrieval tasks and ten datasets, with task-specific instructions and InfoNCE training.This staged scheme progressively improves embedding-based retrieval across diverse multimodal retrieval settings.
  • Reranking: LamRA-Rank adapts the retriever with a lightweight LoRA module and jointly trains pointwise YES/NO classification with listwise ground-truth position prediction.Reranking examples are constructed from the initial retriever’s top 100 candidates, and the final loss is Lrank = Lpoint + Llist.
  • Inference: At inference, LamRA-Ret computes cosine-similarity scores and selects top-K candidates, while LamRA-Rank offers pointwise or listwise reranking to produce the final ordering.The pipeline separates initial retrieval from subsequent reranking through Φret and Φrank.

4. Experiments

LamRA achieves strong retrieval performance across diverse M-BEIR tasks, generalizes effectively to held-out tasks and datasets, and extends competitively to zero-shot video retrieval. Ablations show that both training stages, model scaling, and reranking contribute materially to performance.

  • Versatility Across Various Retrieval Tasks: LamRA-Ret significantly outperforms UniIR-CLIP, improving Recall@5 by 24.2 points on InfoSeek and 8.5 points on CIRR.These gains span both multimodal and pure-text retrieval tasks on the M-BEIR benchmark.
  • Advanced Generalization Capability on Unseen Dataset: LamRA generalizes strongly to 10 unseen datasets, often exceeding or matching strong baselines and improving substantially on dialog-to-image and multi-round composed image retrieval.On Image-Text Matching, LMM-based methods outperform dual-encoder methods, which the authors attribute to stronger comprehension of complex text.
  • Exceptional Generalization on Unseen Retrieval Task: When trained on five tasks and evaluated on excluded tasks, LamRA achieves competitive or superior zero-shot results against supervised methods on complex unseen retrieval tasks.This evaluates generalization to held-out retrieval tasks rather than merely unseen datasets.
  • Ablation Studies: Removing pre-training lowers average M-BEIR performance by three points, while instruction tuning and pre-training both significantly affect retrieval quality.Scaling from Qwen2-VL-2B to 7B also improves performance, indicating benefits from larger LMMs.
  • Reranking Analysis: Both pointwise and listwise reranking significantly improve retrieval, although pointwise reranking generally costs more computation and listwise reranking is limited by context length.Reranking comparisons are conducted on the M-BEIR test set with reranking applied to the top five results.
  • Extending to Video Retrieval: Zero-shot LamRA outperforms InternVideo by 4.7 Recall@1 points on MSR-VTT and UMT-L by 2.5 points on MSVD.These results are obtained on text-to-video retrieval despite no exposure to the evaluated video datasets.

5. Conclusion

LamRA presents a universal LMM-based framework for retrieval and reranking, enabled by lightweight LoRA modules and designed for broad multimodal retrieval tasks. Comprehensive comparisons across more than ten diverse retrieval tasks demonstrate the method’s versatility and effectiveness.

  • LamRA equips LMMs with robust retrieval and reranking capabilities by inserting lightweight LoRA modules.
  • LamRA-Ret introduces a two-stage training strategy to progressively improve retrieval performance.
  • Comparisons across more than ten diverse retrieval tasks underscore LamRA’s versatility and effectiveness.

Supplementary Material · A. Analysis of Pre-training Dataset Selection · B. More Implementation Details

The supplementary material analyzes LamRA-Ret’s pre-training dataset choices and provides additional implementation details for feature extraction and reranking. It also documents the two-stage training setup used to incrementally improve retrieval performance.

  • A. Analysis of Pre-training Dataset Selection: LamRA-Ret uses language-only pre-training followed by multimodal instruction tuning to progressively enhance retrieval performance.
  • A. Analysis of Pre-training Dataset Selection: The pre-training analysis compares datasets containing only images, only text, or combinations of modalities.
  • A. Analysis of Pre-training Dataset Selection: Table 9 reports performance across the evaluated pre-training datasets.
  • B. More Implementation Details: The <emb> token is a newly added vocabulary item used primarily as a placeholder during feature extraction.
  • B. More Implementation Details: An arbitrary token can replace <emb> because the feature embedding comes from the hidden state immediately preceding that token.
  • B. More Implementation Details: The implementation supplement also discusses both pointwise and listwise reranking methods.

C. Details about M-BEIR Dataset

The M-BEIR benchmark applies additional processing to incorporated datasets, so its evaluations can differ from standard dataset evaluations. Table 10 summarizes the benchmark, while Table 12 presents the corresponding instructions.

  • Benchmark construction: M-BEIR applies additional processing to incorporated datasets, which may produce evaluations different from standard individual-dataset settings.The benchmark’s processing can alter the evaluation conditions relative to the original datasets.
  • Benchmark construction: For CIRR, M-BEIR includes training data in the candidate pool, increasing evaluation difficulty compared with the original CIRR dataset.This example illustrates how M-BEIR’s processing changes a dataset’s evaluation setup.
  • Benchmark overview: Table 10 provides a summary of the M-BEIR benchmarks.

D. Details about Unseen Dataset · E. Additional Experimental Results

The paper treats datasets as unseen when their captions or query formats differ substantially from familiar datasets, even when adapted from MSCOCO or FashionIQ. Table 11 summarizes these datasets, while Table 12 summarizes M-BEIR instructions.

  • D. Details about Unseen Dataset: Unseen datasets may be adapted from MSCOCO or FashionIQ but remain unseen because their captions or query formats differ significantly.The paper emphasizes that these differences create a substantial disparity compared with the source datasets.
  • D. Details about Unseen Dataset: Urban1K uses extended captions generated by GPT-4V, illustrating how caption construction can distinguish an adapted dataset as unseen.
  • D. Details about Unseen Dataset: CIRCO combines a reference image with a relative caption, illustrating how a changed query format can define an unseen retrieval setting.
  • D. Details about Unseen Dataset: Table 11 provides a summary of the unseen datasets.
  • D. Details about Unseen Dataset: Table 12 provides a summary of the M-BEIR instructions.

E.1. Experimental Results on the M-BEIR in Global Pool Setting … G. Limitations & Future Work

Across supplementary evaluations, LamRA outperforms UniIR-CLIP in the M-BEIR global-pool setting, improves KVQA retrieval over current SOTA, and supports both pointwise and listwise reranking, while retaining separate LoRA parameters for retrieval and reranking as a limitation.

  • E.1. Experimental Results on the M-BEIR in Global Pool Setting: LamRA achieves an average of 12.5 points higher than UniIR-CLIP on M-BEIR in the global pool setting.The global pool constructs candidates from all datasets collectively, unlike the dataset-restricted local pool used in the main paper.
  • E.1. Experimental Results on the M-BEIR in Global Pool Setting: Table 13 evaluates M-BEIR global-pool retrieval across text and image query/candidate types using Recall@10 for FashionIQ and Fashion200K and Recall@5 elsewhere.The task abbreviations are qt, qi, ct, and ci; VN, F200K, InfoS, and FIQ denote benchmark datasets.
  • E.2. Detailed Experimental Results on the Pointwise and Listwise Reranking: Both LamRA-Rank pointwise and listwise reranking improve performance across various applications, but listwise inference does not consistently outperform pointwise inference when candidates include images.Current LMMs often require hundreds of tokens to represent a single image, contributing to this discrepancy.
  • E.2. Detailed Experimental Results on the Pointwise and Listwise Reranking: Table 14 compares Recall@1 and per-query inference costs for LamRA-Rank(P) and LamRA-Rank(L) after reranking the top-5 results.Inference time is measured on eight A100 GPUs with batch size 32.
  • F. Exploration of RAG Applications: The KVQA experiments explore whether retrieval and generative capabilities can be integrated within the same LMM for retrieval-augmented generation.The motivation follows LamRA’s demonstrated potential as a universal retriever.
  • F. Exploration of RAG Applications: Jointly training retrieval and VQA with LoRA enables LamRA to surpass current SOTA retrieval performance on three knowledge-based visual question answering datasets.KVQA first retrieves relevant documents and then answers questions using the retrieved information.
  • G. Limitations & Future Work: LamRA currently requires separate LoRA parameter sets for retrieval and reranking tasks.Future work may jointly train these tasks or integrate retrieval training into the supervised fine-tuning stage.

H. More Qualitative Results · H.1. Successful Cases · H.2. Failure Cases

The qualitative results show that LamRA handles diverse retrieval tasks successfully across text, image, and multimodal inputs, while failures arise from false-negative candidates or inherently challenging queries.

  • H.1. Successful Cases: LamRA effectively handles a diverse range of retrieval tasks, as illustrated by successful examples in Figures 4–12.The examples include retrieval followed by reranking.
  • H.2. Failure Cases: Failure cases include errors caused by false-negative candidates and by inherently challenging queries that lead to retrieval failures.The latter difficulty is exemplified in the last row of Figure 13.
  • H.2. Failure Cases: The failure-case displays organize each example into query cues and retrieved results.This structure is explicitly shown in the failure-case presentation.
  • H.1. Successful Cases: The qualitative examples cover text-to-image, text-to-text, and text-to-text-image retrieval, with ground truths explicitly marked.Figures 4–6 use red boxes, red text, or both to identify the ground truth.
  • H.1. Successful Cases: Additional successful examples span image-to-text, image-to-image, and text-image-to-image retrieval tasks.Figures 7–9 identify the ground truth using red text or a red box.
  • H.1. Successful Cases: The results also demonstrate successful text-image-to-text and text-image-to-text-image retrieval.Figures 10 and 11 mark the ground truth with red text, a red box, or both.
  • H.1. Successful Cases: The qualitative section includes examples of retrieval followed by reranking, with the ground truth indicated by a red box.These examples are provided in Figure 12.
Loading 2412.01720v1…