PaperlayerView exact revision on arXiv

Source-linked AI summary

MLLM Is a Strong Reranker: Advancing Multimodal Retrieval-augmented Generation via Knowledge-enhanced Reranking and Noise-injected Training

Zhanpeng Chen, Chengjin Xu, Yiyan Qi, Jian Guo

arXiv:2407.21439v2cs.AIcs.CLcs.LG

TL;DR

MLLMs rely on static training data, and multimodal RAG inevitably introduces multi-granularity noisy correspondence that hinders accurate retrieval and generation. RagVL contrasts logit distributions conditioned on original and Gaussian-noised images to address visual noise. RagVL improves multimodal retrieval and generation performance across WebQA and MultimodalQA, with consistent retrieval gains and stronger robustness.

  • Problem

    MLLMs rely on static training data, and multimodal RAG inevitably introduces multi-granularity noisy correspondence that hinders accurate retrieval and generation.

  • Method

    RagVL contrasts logit distributions conditioned on original and Gaussian-noised images to address visual noise.

  • Results

    RagVL improves multimodal retrieval and generation performance across WebQA and MultimodalQA, with consistent retrieval gains and stronger robustness.

  • Takeaways & Limitations

    Knowledge-enhanced reranking and noise-injected training provide an effective framework for handling noisy multimodal retrieval-augmented generation.

Abstract

from arXiv · show

Multimodal Large Language Models (MLLMs) have demonstrated remarkable capabilities in processing and generating content across multiple data modalities. However, a significant drawback of MLLMs is their reliance on static training data, leading to outdated information and limited contextual awareness. This static nature hampers their ability to provide accurate and up-to-date responses, particularly in dynamic or rapidly evolving contexts. Though integrating Multimodal Retrieval-augmented Generation (Multimodal RAG) offers a promising solution, the system would inevitably encounter the multi-granularity noisy correspondence (MNC) problem, which hinders accurate retrieval and generation. In this work, we propose RagVL, a novel framework with knowledge-enhanced reranking and noise-injected training, to address these limitations. We instruction-tune the MLLM with a simple yet effective instruction template to induce its ranking ability and serve it as a reranker to precisely filter the top-k retrieved images. For generation, we inject visual noise during training at the data and token levels to enhance the generator's robustness. Extensive experiments on the subsets of two datasets that require retrieving and reasoning over images to answer a given query verify the effectiveness of our method. Code and models are available at https://github.com/IDEA-FinAI/RagVL.

1 INTRODUCTION

Multimodal RAG supplies external visual knowledge for MLLMs but introduces multi-granularity noisy correspondence that can undermine retrieval and generation. RagVL addresses this with knowledge-enhanced reranking and noise-injected training, and experiments demonstrate effectiveness on multimodal knowledge-seeking QA and retrieval tasks.

  • 1 INTRODUCTION: Multimodal RAG retrieves external visual information to improve the contextual grounding of MLLM responses.It extends retrieval-augmented generation beyond text by incorporating relevant multimodal data before generation.
  • 1 INTRODUCTION: MNC arises when coarse captions retrieve similar but negative images and fine-grained image discrepancies disrupt reasoning and answering.These mismatches affect both retrieval precision and the correspondence needed for accurate responses.
  • 1 INTRODUCTION: RagVL instruction-tunes an MLLM to rank query-image relevance and uses it to select more relevant top-N candidates.The ranking ability is induced with a simple instruction template, leveraging the MLLM’s cross-modal understanding.
  • 1 INTRODUCTION: The framework uses a three-stage pipeline and extensive experiments to improve robust multimodal retrieval-augmented generation.The reported evaluation covers multimodal knowledge-seeking QA and retrieval tasks.
  • 1 INTRODUCTION: RagVL addresses multi-granularity noisy correspondence with knowledge-enhanced reranking and noise-injected training.The framework targets both coarse-grained and fine-grained noise in multimodal retrieval-augmented generation.

2 RELATED WORK

Related work develops retrieval-augmented language and multimodal systems that incorporate external knowledge for generation and retrieval. Existing approaches span dense passage retrieval, joint retriever-generator training, image-patch retrieval, multimodal memory, and cross-modal candidate modeling.

  • 2 RELATED WORK: Retrieval-augmented language models improve task performance by incorporating relevant information from diverse external knowledge sources.DPR, REALM, and RAG exemplify retrieval-based approaches for open-domain question answering and language generation.
  • 2 RELATED WORK: Multimodal retrieval-augmented systems extend this paradigm to images, patches, and external multimodal memories.Plug-and-play retrieves image patches, while MuRAG accesses a non-parametric multimodal memory to augment generation.
  • 2 RELATED WORK: SKURG models relationships among multimodal candidates using an entity-centered fusion encoder to align sources across modalities.This line of work focuses on connecting candidates and modeling their relations during retrieval.

3 METHODOLOGY

RagVL uses a staged multimodal retrieval pipeline: CLIP/faiss retrieves candidate images, an instruction-tuned MLLM reranks them, and a generator answers using the selected images. Its training addresses noisy correspondence through relevance-aware reranking and visual noise injection at both data and token levels.

  • 3.2 MULTIMODAL RETRIEVER: CLIP and faiss retrieve top-K images with Maximum Inner Product Search before downstream reranking.The retriever encodes queries and images with CLIP and selects nearest image neighbors from the external memory.
  • 3.3 INDUCING RANKING ABILITY OF MLLMS: CLIP can miss proper text-image matches for long-tail or domain-specific terms, motivating a reranking stage.This limitation can require larger K values to preserve recall, increasing time and resource costs.
  • 3.3 INDUCING RANKING ABILITY OF MLLMS: The reranker is instruction-tuned on relevant and hard-negative image pairs to judge query-image relevance using caption-aware prompts.The model generates “Yes” for relevant pairs and “No” for irrelevant pairs, then uses the probability of “Yes” to rank candidates.
  • 3.3 INDUCING RANKING ABILITY OF MLLMS: An adaptive threshold removes reranked images whose relevance probability falls below η, reducing distraction from irrelevant visual evidence.The natural threshold is η = 0.5, while the adaptive threshold is selected from validation-set exact-match and mismatch curves.
  • 3.4 NOISE-INJECTED TRAINING: Noise-injected training improves robustness by adding hard-negative images at the data level and contrasting logits from original and distorted images at the token level.The token-level procedure contrasts distributions conditioned on original and Gaussian-noise-distorted visual inputs, then reweights the language-model loss using visual correlation weights.

4 EXPERIMENTS AND ANALYSIS

Experiments on WebQA and MultimodalQA show that RagVL improves multimodal retrieval and retrieval-augmented generation through knowledge-enhanced reranking and noise-injected training. The method generalizes across datasets and low-resource settings, while natural thresholds and both noise-injection components provide the strongest reported results.

  • Retrieval Results: R@2 on WebQA improves by an average of 40% across five MLLM rerankers, while MultimodalQA reaches 98.26% Recall@20 with selected models.These results compare knowledge-enhanced rerankers against the retriever baseline across both datasets.
  • Generalizability: With only 2.5% of WebQA training data, the reranker significantly outperforms InternVL-G on R@2, demonstrating low-resource robustness.As training data increases, correct-recall probability stabilizes around 20% and recall follows a similar trend.
  • Reranking with Thresholds: At η=0.5, WebQA F1 increases by 29.75%, while MultimodalQA identifies all ground-truth images among retrieved candidates.The adaptive threshold filters irrelevant images, improving accuracy and F1, but natural thresholds provide the stronger generation results.
  • Retrieval-augmented Generation: RagVL configurations on InternVL2-1B and InternVL2-2B approach or surpass Oracle performance, with natural thresholds improving recalled-image accuracy and downstream metrics.Adaptive thresholds improve accuracy but substantially reduce recall, making natural thresholds the more efficient choice for generation.
  • Ablation Studies: Ablations show that removing the reranker, noise-injected data, or noise-injected logits contrasting decreases performance across metrics, especially for multi-image inference.Noise at both data and token levels helps distinguish candidate images and mitigate multi-granularity noisy correspondence; NIT also produces more focused attention on query-relevant image regions.

5 CONCLUSION

RagVL addresses multi-granularity noisy correspondence in multimodal RAG through knowledge-enhanced reranking and noise-injected training, improving retrieval accuracy and generation robustness.

  • RagVL combines knowledge-enhanced reranking with noise-injected training to address coarse- and fine-grained noise in multimodal RAG.The framework instruction-tunes MLLMs for reranking and injects visual noise during training to improve robustness on noisy data.

A DATA STATISTICS AND EVALUATION METRICS

The evaluation uses WebQA, MultimodalQA, Flickr30K, and MS-COCO, with dataset-specific splits and metrics for multimodal retrieval and question answering.

  • The paper reports overall dataset statistics and Recall@20 for different retrievers in dedicated tables.
  • WebQA contains queries requiring 1–2 images or text snippets, and Accuracy measures overlap between generated and ground-truth key entities.Multiple knowledge sources are needed for 44% of image-based and 99% of text-based queries.
  • MultimodalQA includes questions over tables, text, and images; the evaluation focuses on image-only ImageQ pairs using Exact Match.Each selected ImageQ pair is attached to one image.
  • Flickr30K uses 1,000 images for validation, 1,000 for testing, and the remainder for training from 31,000 captioned images.
  • MS-COCO uses 113,287 images for training, 5,000 for validation, and 5,000 for testing from 123,287 captioned images.

B IMPLEMENTATION DETAILS

The implementation evaluates RagVL with several frozen-backbone MLLMs, uses instruction templates for ranking and generation, and addresses reranking efficiency as a practical constraint.

  • RagVL evaluates LLaVA-v1.5-13B, mPLUG-Owl2, Qwen-VL-Chat, and InternVL while using frozen CLIP-ViT-L/14-336px as the vision-text encoder.
  • The MLLM reranker requires more inference time than CLIP Top-K retrieval because it evaluates candidate-image relevance before generation.The efficiency issue is identified as a major limitation for further development.
  • The instruction template supports multi-image retrieval-augmented QA, while ranking tasks evaluate one image at a time.
  • Inference acceleration methods such as FlashAttention and PagedAttention are proposed as ways to address the efficiency issue.

D EFFECT OF CAPTIONS

Caption-aware instruction tuning improves MLLM reranking over the vanilla baseline, supporting captions as useful signals for selecting relevant image sources.

  • Ranking-task models outperform vanilla LLaVA-v1.5-13B on WebQA ranking and QA, with the caption-aware model performing particularly well.The result supports instruction templates as a means of inducing MLLM ranking ability.
  • The WebQA setup retrieves top-20 images with CLIP and reranks them into top-2 candidates using instruction-tuned rerankers.
  • Each reranker inference involves 20 image-relevance evaluations and one answer-generation step, increasing cost relative to CLIP Top-K retrieval.

E PERFORMANCE ON CAPTION-TO-IMAGE RETRIEVAL

RagVL's knowledge-enhanced reranker generalizes to caption-to-image retrieval, outperforming most existing retrievers across metrics except InternVL-G.

  • RagVL outperforms the majority of existing retrievers across all caption-to-image retrieval metrics except InternVL-G.The evaluation covers Flickr30K and MS-COCO, where InternVL-G is specifically designed for image-text matching.

F MORE EVALUATIONS ON LLAVA-V1.5-13B

Additional LLaVA-v1.5-13B experiments show that RagVL remains effective under low-resource reranker training, adaptive thresholding, ablations, and multimodal question answering.

  • Low-resource Settings on WebQA: With only 2.5% of the original data, the reranker surpasses InternVL-G in R@2 and nearly reaches the performance peak.This result supports robust reranker training under low-resource settings.
  • Reranking Performance with Thresholds: Correct-recall relevance probabilities concentrate at the highest range, enabling the adaptive threshold to filter most incorrect candidates.The thresholding analysis uses LLaVA-v1.5-13B trained with 20% of the data.
  • The reranker outperforms direct CLIP top-N retrieval across all metrics in both blended and ranking-only training settings.With η=0.5, ranking-only training increases WebQA F1 by 25.17%, while MultimodalQA retrieves all ground-truth images from the top-K candidates.
  • Results of Retrieval-augmented Generation: RagVL outperforms baselines across all reported LLaVA-v1.5-13B retrieval-augmented generation configurations on MultimodalQA and WebQA.The adaptive threshold is more effective for LLaVA-v1.5-13B because correct-recall probabilities are more concentrated at high values.
  • Ablation Studies: Ablations show significant benefits from reranking and noise injection, while a blended reranker for retrieval and generation is promising but still requires optimization.The blended reranker combines ranking and QA training with noise-injected data and vanilla MLE loss.

G MORE CASE STUDIES

The paper provides additional attention-heatmap visualizations for WebQA cases requiring either single-image or multiple-image reasoning.

  • Additional visualizations cover WebQA examples requiring single-image and multi-image inference.The cases are presented as attention heatmaps, including a comparison involving homes and a church on Enniskillen's Main Shopping Street.
Loading 2407.21439v2…