PaperlayerView exact revision on arXiv

Source-linked AI summary

LamRA: Large Multimodal Model as Your Advanced Retrieval Assistant

Yikun Liu, Pingan Chen, Jiayin Cai, Xiaolong Jiang, Yao Hu, Jiangchao Yao, Yanfeng Wang, Weidi Xie

arXiv:2412.01720v1cs.CV

TL;DR

Retrieval methods often require cumbersome task-specific fine-tuning of vision-language models for increasingly varied tasks. LamRA equips generative LMMs for retrieval and reranking through staged retrieval training and pointwise/listwise reranking, with robust performance in supervised and unseen-task settings.

  • Problem

    Existing methods adapt vision-language models to specialized retrieval tasks through task-specific fine-tuning, which remains cumbersome and laborious.

    Source: §1 · p. 2

  • Method

    LamRA adds lightweight LoRA modules to LMMs, using two-stage training for retrieval and joint pointwise and listwise training for reranking.

    Source: §1 · p. 2

  • Results

    Across more than 10 retrieval scenarios, LamRA is robust in supervised and unseen-task settings; on unseen InfoSeek text-image-to-text, it beats UniIR-CLIP by 26.3 points in Recall@5.

    Source: §1 · p. 2, §4.2 · p. 6

  • Takeaways & Limitations

    The reported held-out-task results suggest LamRA may apply to unseen retrieval challenges without additional training.

    Source: §1 · p. 2

    A potential drawback is high inference cost, though pre-extracting candidate features or low-query-rate deployment can mitigate it.

    Source: §4.3 · p. 8

Abstract

from arXiv · show

With the rapid advancement of multimodal information retrieval, increasingly complex retrieval tasks have emerged. Existing methods predominately rely on task-specific fine-tuning of vision-language models, often those trained with image-text contrastive learning. In this paper, we explore the possibility of re-purposing generative Large Multimodal Models (LMMs) for retrieval. This approach enables unifying all retrieval tasks under the same formulation and, more importantly, allows for extrapolation towards unseen retrieval tasks without additional training. Our contributions can be summarised in the following aspects: (i) We introduce LamRA, a versatile framework designed to empower LMMs with sophisticated retrieval and reranking capabilities. (ii) For retrieval, we adopt a two-stage training strategy comprising language-only pre-training and multimodal instruction tuning to progressively enhance LMM's retrieval performance. (iii) For reranking, we employ joint training for both pointwise and listwise reranking, offering two distinct ways to further boost the retrieval performance. (iv) Extensive experimental results underscore the efficacy of our method in handling more than ten retrieval tasks, demonstrating robust performance in both supervised and zero-shot settings, including scenarios involving previously unseen retrieval tasks.

1. Introduction

LamRA equips LMMs for retrieval and reranking in a shared formulation, and the paper reports robust performance across more than ten retrieval tasks, including held-out tasks. Its retrieval and reranking components use staged training and support pointwise and listwise reranking.

1. Introduction

  • Although contrastively trained VLMs such as CLIP [42] and ALIGN [15] generalize well in cross-modal retrieval, emerging specialized tasks require cumbersome task-specific fine-tuning [33] [55].Examples include composed image retrieval [34], long-text image retrieval, and image/question to multimodal document retrieval [36].Source: §1 · pp. 1–2
  • LMMs’ language interface and LLM-based language comprehension and real-world knowledge motivate a shared formulation for retrieval tasks and extrapolation to unseen tasks without additional training.Source: §1 · p. 2
  • LamRA adds lightweight LoRA modules to LMMs, with LamRA-Ret for quick retrieval and LamRA-Rank for flexible pointwise and listwise reranking.LamRA-Ret uses text-only pre-training to produce embeddings from summarization prompts, followed by multimodal instruction tuning; LamRA-Rank uses hard-sample mining and joint training to handle single or multiple candidate inputs.Source: §1 · p. 2
  • LamRA performs on par with or significantly surpasses existing SOTA models across more than ten retrieval tasks, with robust supervised, zero-shot, and held-out-task performance.Experiments span simple and complex scenarios, and held-out tasks are evaluated without prior exposure during training.Source: §1 · p. 2

2. Related Work

Multimodal retrieval has expanded beyond traditional cross-modal benchmarks to complex tasks that are often addressed with task-specific fine-tuning, while universal retrieval remains challenging. Prior multimodal representations and LMM applications leave room for a framework focused on universal retrieval and reranking.

Multimodal Information Retrieval

  • Complex retrieval tasks now extend beyond MSCOCO [29] and Flickr30K [41], while task-specific fine-tuning [3] [37] leaves universal multimodal embeddings a significant challenge.Examples include composed image retrieval [2] [34] [45], long-text-to-image retrieval, and image/question-to-multimodal document retrieval [13].Source: §2 · p. 2

Multimodal Representation Learning

  • CLIP [42] and ALIGN [15] use dual-encoder contrastive learning [38] on image-text pairs, but struggle with interleaved inputs and complex text [12].The text encoder's limited understanding of complex text suggests traditional dual-encoder models may require refinement.Source: §2 · p. 2
  • UniIR [51] evaluates eight retrieval tasks and reports improved generality from fine-tuned CLIP, while E5-V [18], trained on text pairs [11], enables zero-shot retrieval but remains limited on some complex tasks.E5-V uses prompts to map images and text into a shared language hidden space.Source: §2 · pp. 2–3

Large Multimodal Model

  • LMMs have been applied to segmentation [21], object detection [40], and vision-centric tasks [47], but few studies address universal retrieval; LamRA adapts LMMs with lightweight LoRAs for retrieval and reranking.Concurrent studies [19] [28] also explore LMMs for universal retrieval.Source: §2 · p. 3

3. Method

LamRA uses generative LMMs to retrieve and rerank multimodal candidates through a shared pipeline. Its method combines embedding-based retrieval, staged retrieval training, joint reranker training, and inference-time score aggregation.

3.1. Problem Formulation

  • For each text, image, or mixed-format query, LamRA ranks candidates by embedding cosine similarity, then reranks the top-K subset.The initial retrieval produces C1, and reranking produces the final reordered set C2.Source: §3.1 · pp. 3–4

3.2. Architecture & Feature Extraction

  • LamRA extracts embeddings using an Explicit One-word Limitation prompt and the hidden state immediately preceding <emb>, following [17] [18].The prompts cover image-only, text-only, and mixed image-text inputs; the vision encoder and projector map images into language space for processing with text.Source: §3.2 · p. 4

3.3. Training for Retrieval

  • LamRA-Ret is trained in two stages: language-only NLI pre-training with LoRA, followed by instruction tuning across M-BEIR retrieval tasks.M-BEIR comprises 8 retrieval tasks and 10 datasets [51], with task-specific instructions; both stages use InfoNCE contrastive loss [38]. Direct retrieval with Qwen2-VL-7B [46] is described as inferior, motivating adaptation, and NLI pairs are from [11].Source: §3.3 · p. 4

3.4. Training for Reranking

  • LamRA-Rank trains jointly for pointwise YES/NO judgments and listwise ground-truth-position prediction, using top-100 retrieved candidates as hard negatives.Pointwise training pairs each positive with a randomly selected negative; listwise training inserts the positive among 2–5 randomly selected negatives. The final loss is the sum of pointwise and listwise cross-entropy losses.Source: §3.4 · pp. 4–5

3.5. Inference Pipeline

  • At inference, LamRA-Ret retrieves top-K candidates and LamRA-Rank reranks them pointwise or listwise; a weighted sum of their scores produces the final ordering.Pointwise reranking runs K inferences and scores each candidate by its YES probability; listwise reranking inputs all top-K candidates together and outputs the most relevant candidate’s serial number.Source: §3.5 · p. 5

4. Experiments

Experiments evaluate LamRA on M-BEIR and previously unseen datasets and tasks, finding stronger retrieval and reranking performance while also examining training, scaling, video transfer, inference costs, and qualitative examples.

4.1. Experimental Setup

  • The experiments use NLI for pre-training and M-BEIR’s eight tasks across 10 datasets for instruction tuning, evaluating standard metrics on M-BEIR and 10 unseen datasets.The default model is Qwen2-VL-7B, with a fixed vision side and LoRA fine-tuning of the LLM; the setup also includes training on five tasks and evaluating on three excluded tasks.Source: §4.1 · p. 5

4.2. Experimental Results

  • Qwen2-VL-7B’s M-BEIR average rises from 23.0 zero-shot to 63.7 with LamRA, while reranking adds 7.1 points on average across 16 retrieval tasks.LamRA-Ret also improves over UniIR-CLIP by 24.2 points in InfoSeek text-image-to-text Recall@5 and 8.5 points in CIRR composed-image Recall@5; on simpler cross-modal tasks, it is comparable to or slightly higher than UniIR-CLIP.Source: Table 2 · p. 6, §4.2 · p. 6
  • On held-out retrieval tasks, LamRA exceeds supervised UniIR-CLIP by 26.3 points in InfoSeek text-image-to-text Recall@5, demonstrating zero-shot generalization without training on those tasks.The held-out tasks are image-to-image, text-image-to-text, and text-image-to-text-image retrieval.Source: Table 3 · p. 6, §4.2 · p. 6
  • Across 10 unseen datasets, LamRA is comparable to or exceeds strong baselines, including over a 10-point gain on Urban-1K image-to-long-text retrieval.It also yields substantial gains on dialog-to-image and multi-round composed-image retrieval; on Image-Text Matching, LMM-based methods outperform dual encoders, which the authors attribute to complex-text comprehension.Source: §4.2 · p. 6, Table 4 · p. 7

4.3. Ablation Study & Analysis

  • Both pre-training and instruction tuning improve retrieval, with removing pre-training lowering average M-BEIR performance by three points; performance also rises from Qwen2-VL-2B to 7B.The scaling experiments cover only the 2B and 7B models due to computational limitations.Source: §4.3 · p. 7, Table 5 · p. 7, Table 6 · p. 7
  • Both pointwise and listwise reranking improve retrieval, but pointwise generally costs more computation while listwise is limited by the LLM context length.The paper notes that pre-extracting candidate features or using LamRA where query-per-second requirements are low can mitigate inference costs.Source: Table 7 · p. 7, §4.3 · pp. 7–8

4.4. Qualitative Results.

  • Qualitative examples show LamRA retrieving a target image for a composed-image query and matching a relevant Wikipedia snippet with an image similar to the query image.The figure presents results across six retrieval tasks.Source: Figure 3 · p. 8, §4.4 · p. 8

4.3. Ablation Study & Analysis

  • Zero-shot video retrieval is competitive without video fine-tuning: LamRA beats InternVideo by 4.7 Recall@1 on MSR-VTT and UMT-L by 2.5 on MSVD, but trails InternVideo2.The authors identify video-data integration as future work to address the gap to InternVideo2.Source: Table 8 · p. 8, §4.3 · p. 8

5. Conclusion

LamRA applies LMMs to diverse retrieval tasks using lightweight LoRA modules, with separate strategies for retrieval and reranking. Comparisons across more than ten tasks underscore the framework’s versatility and effectiveness.

5. Conclusion

  • LamRA-Ret uses a two-stage training strategy to progressively enhance retrieval performance.The framework applies lightweight LoRA modules to equip LMMs with retrieval and reranking capabilities.Source: §5 · p. 8
  • LamRA-Rank supports both pointwise and listwise reranking to further optimize results.Source: §5 · p. 8
  • Comparisons across more than ten diverse retrieval tasks underscore LamRA’s versatility and effectiveness.The paper reports comprehensive quantitative and qualitative comparisons.Source: §5 · pp. 8–9

Supplementary Material · A. Analysis of Pre-training Dataset Selection · B. More Implementation Details

The supplementary material analyzes pre-training dataset selection and adds implementation details for feature extraction, listwise reranking, and baseline evaluation.

A. Analysis of Pre-training Dataset Selection

B. More Implementation Details

  • The <emb> token is only a placeholder: feature extraction uses the hidden state at the preceding position, so any token can replace it.Source: B. More Implementation Details · p. 12
  • Listwise reranking scores each candidate by the probability that the LMM outputs its serial number, analogous to pointwise scoring by the probability of YES.Source: B. More Implementation Details · p. 12
  • Baseline evaluation uses official checkpoints and reports results from the corresponding papers; because MagicLens Coca is unavailable, experiments use its CLIP version.Source: B. More Implementation Details · p. 12

C. Details about M-BEIR Dataset

M-BEIR applies additional processing to its component datasets, so its benchmark results can differ from standard evaluations; CIRR’s candidate pool, for example, includes training data and is therefore more difficult than the original evaluation.

C. Details about M-BEIR Dataset

  • M-BEIR’s additional dataset processing can change standard evaluations; its CIRR candidate pool includes training data, making evaluation more difficult than in original CIRR [51].The paper directs readers to UniIR [51] for a fuller account of these differences.Source: C. Details about M-BEIR Dataset · p. 13, Table 10 · p. 13

D. Details about Unseen Dataset · E. Additional Experimental Results

The paper treats datasets adapted from MSCOCO or FashionIQ as unseen when their captions or query formats differ substantially from the originals. It gives Urban1K and CIRCO as examples and points to tables summarizing the unseen datasets and M-BEIR instructions.

D. Details about Unseen Dataset

  • Urban1K and CIRCO are treated as unseen because their captions or query formats differ substantially from the original COCO dataset.Urban1K uses extended captions generated by GPT-4V, while CIRCO queries combine a reference image with a relative caption.Source: D. Details about Unseen Dataset · p. 13
  • Table 11 summarizes the unseen datasets.Source: Table 11 · p. 13
  • Table 12 summarizes the M-BEIR instructions.Source: Table 12 · p. 14

E.1. Experimental Results on the M-BEIR in Global Pool Setting … G. Limitations & Future Work

Supplementary experiments report LamRA’s performance in the global-pool setting, compare pointwise and listwise reranking, and explore joint retrieval and generation for KVQA. The sections also identify limitations in training, candidate capacity, and model size.

E.1. Experimental Results on the M-BEIR in Global Pool Setting

  • Unlike the local-pool results in the main paper, this evaluation draws candidates from all datasets collectively rather than only the dataset under evaluation.The global and local settings differ in candidate-pool composition.Source: §E.1 · p. 15
  • In the global-pool M-BEIR setting, LamRA averages 12.5 points higher than UniIR-CLIP.Table 13 reports the global-pool comparison; FashionIQ and Fashion200K use Recall@10, while other evaluations use Recall@5.Source: §E.1 · p. 15, Table 13 · p. 15

E.2. Detailed Experimental Results on the Pointwise and Listwise Reranking

  • Pointwise and listwise LamRA-Rank both improve performance across various applications, but listwise reranking does not consistently outperform pointwise reranking in inference time.Table 14 compares Recall@1 and per-query inference costs when reranking the top five results.Source: §E.2 · p. 15, Table 14 · p. 15
  • Listwise reranking can be slower when candidates include images because representing each image often takes hundreds of tokens, extending context length and inference time.The paper notes that research is investigating ways to reduce visual-token counts.Source: §E.2 · p. 15

F. Exploration of RAG Applications

  • On three KVQA tasks, jointly trained retrieval and VQA using LoRA surpasses current SOTA in retrieval and achieves comparable or better answer accuracy.The experiments retrieve relevant documents before answering questions based on them; Table 15 compares methods on KVQA tasks.Source: F. Exploration of RAG Applications · p. 16, Table 15 · p. 16

G. Limitations & Future Work

  • LamRA currently requires separate LoRA parameter sets for retrieval and reranking; future work may train both jointly or integrate retrieval training into SFT.These are proposed strategies to address the separate-task training limitation.Source: G. Limitations & Future Work · p. 16
  • Listwise LamRA-Rank currently accepts only 2–5 candidates because of context-length and computational constraints, and supporting more candidates remains future work.Source: G. Limitations & Future Work · p. 16
  • Current models are limited to 2B and 7B parameters, leaving further exploration of LMMs for multimodal retrieval as an improvement opportunity.Source: G. Limitations & Future Work · p. 16

H. More Qualitative Results · H.1. Successful Cases · H.2. Failure Cases

LamRA’s qualitative examples span diverse retrieval tasks and retrieval followed by reranking, while the reported failures arise from false-negative candidates or challenging queries.

H. More Qualitative Results

  • Examples in Figures 4–12 illustrate LamRA handling a diverse range of retrieval tasks.Source: §H.1 · p. 16

H.2. Failure Cases

  • Some retrieval failures result from false-negative candidates, while others arise from inherently challenging queries.The last row exemplifies a challenging query that leads to retrieval failure.Source: §H.2 · p. 16, Figure 13 · p. 21

H.1. Successful Cases

Related papers

Loading 2412.01720v1…