PaperlayerView exact revision on arXiv

Source-linked AI summary

MRAG-Bench: Vision-Centric Evaluation for Retrieval-Augmented Multimodal Models

Wenbo Hu, Jia-Chen Gu, Zi-Yi Dou, Mohsen Fayyaz, Pan Lu, Kai-Wei Chang, Nanyun Peng

arXiv:2410.08182v2cs.CVcs.AIcs.CL

TL;DR

Existing multimodal retrieval benchmarks are largely text-centric, despite cases where visual information is more useful for answering questions. MRAG-Bench introduces a nine-scenario benchmark of vision-centric retrieval and finds that image augmentation helps LVLMs more than textual knowledge, while GPT-4o uses retrieved knowledge far less effectively than humans.

  • Problem

    Existing retrieval-augmented LVLM benchmarks remain text-centric even though some questions benefit more from retrieved visual information.

    Source: §1 · p. 2

  • Method

    MRAG-Bench evaluates 14 LVLMs on 16,130 images and 1,353 human-annotated questions spanning nine perspective and transformative scenarios.

    Source: §1 · p. 2

  • Results

    All LVLMs improve more with image than textual augmentation, while GPT-4o gains only 5.82% from ground-truth knowledge versus 33.16% for humans.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    MRAG-Bench highlights the need to improve LVLMs’ ability to utilize retrieved visual knowledge effectively.

    Source: §6 · p. 10

Abstract

from arXiv · show

Existing multimodal retrieval benchmarks primarily focus on evaluating whether models can retrieve and utilize external textual knowledge for question answering. However, there are scenarios where retrieving visual information is either more beneficial or easier to access than textual data. In this paper, we introduce a multimodal retrieval-augmented generation benchmark, MRAG-Bench, in which we systematically identify and categorize scenarios where visually augmented knowledge is better than textual knowledge, for instance, more images from varying viewpoints. MRAG-Bench consists of 16,130 images and 1,353 human-annotated multiple-choice questions across 9 distinct scenarios. With MRAG-Bench, we conduct an evaluation of 10 open-source and 4 proprietary large vision-language models (LVLMs). Our results show that all LVLMs exhibit greater improvements when augmented with images compared to textual knowledge, confirming that MRAG-Bench is vision-centric. Additionally, we conduct extensive analysis with MRAG-Bench, which offers valuable insights into retrieval-augmented LVLMs. Notably, the top-performing model, GPT-4o, faces challenges in effectively leveraging retrieved knowledge, achieving only a 5.82% improvement with ground-truth information, in contrast to a 33.16% improvement observed in human participants. These findings highlight the importance of MRAG-Bench in encouraging the community to enhance LVLMs' ability to utilize retrieved visual knowledge more effectively.

1 INTRODUCTION

MRAG-BENCH addresses the text-centric focus of prior retrieval-augmented LVLM benchmarks by evaluating scenarios where visual knowledge is more useful. It contains diverse vision-centric questions and shows that image augmentation improves LVLMs, while GPT-4o still uses retrieved knowledge less effectively than humans.

  • MRAG-BENCH targets scenarios where retrieving visual information is more beneficial than textual knowledge, such as recognizing a car from a front-facing view rather than a top-down view.Source: §1 · p. 2
  • The benchmark contains 16,130 images and 1,353 human-annotated multiple-choice questions across perspective and transformative scenarios, with 9,673 human-selected ground-truth images.Source: §1 · p. 2
  • All evaluated LVLMs improve more with image augmentation than textual knowledge, confirming that MRAG-BENCH is vision-centric.Source: §1 · p. 2
  • GPT-4o reaches 68.68% without RAG and 74.5% with ground-truth RAG, outperforming LLaVA-OneVision by 15.39% and 15.52%, respectively.Source: §1 · p. 2
  • GPT-4o improves only 5.82% with ground-truth knowledge, whereas humans improve 33.16%, highlighting limited use of retrieved visual knowledge.Source: §1 · p. 3

2 MRAG-BENCH

MRAG-BENCH is organized around perspective and transformative visual challenges, then populated with human-annotated questions and curated image knowledge. Its collection process combines public datasets, web scraping, human selection, and multi-stage quality control.

2.1 BENCHMARK OVERVIEW

  • Perspective challenges involve varying viewpoints, scopes, or visibility, whereas transformative challenges involve fine-grained physical changes that make entities unfamiliar.Source: §2.1 · p. 3
  • MRAG-BENCH evaluates a retriever that returns relevant images for a query image and textual question, which an LVLM uses to produce the final answer.Source: §2.1 · p. 3
  • The dataset contains 16,130 images and 1,353 multiple-choice questions, designed around real-world visual retrieval, cleaned ground-truth images, and deterministic evaluation settings.Source: §2.1 · p. 3

2.2 BENCHMARK COMPOSITION

  • The benchmark covers nine scenarios: four perspective dimensions, four transformative dimensions, and one geographic-origin category, with scenario shares ranging from 7.5% to 23.8%.Source: §2.2 · pp. 4–5
  • Perspective scenarios test angle, partial visibility, long-range scope, and occlusion, while transformative scenarios test temporal change, deformation, incompleteness, and biological transformation.Source: §2.2 · pp. 4–5

2.3 DATA COLLECTION

  • Perspective data came from ImageNet, Oxford Flowers102, and StanfordCars, with manually selected representatives and cropped partial-image queries; transformative data came from Bing Image Search and human filtering.Source: §2.3 · p. 5
  • Quality control combined automatic format, validity, and redundancy checks with expert review of question-image correspondence and ambiguity.Source: §2.3 · p. 6

3 EXPERIMENTS

MRAG-BENCH evaluates 14 LVLMs on multimodal retrieval, showing that visual knowledge improves performance while exposing differences in how models handle noisy retrieved images.

3.1 EXPERIMENTAL SETUP

  • MRAG-BENCH evaluates 14 LVLMs, including 4 proprietary and 10 open-source models, using multiple-image inputs and accuracy-based multiple-choice evaluation.The setup uses default model generation settings and automated answer extraction with GPT-3.5-turbo only when predefined rules fail.Source: §3.1 · p. 7

3.2 MAIN RESULTS

  • 68.68% without multimodal RAG and 74.5% with ground-truth knowledge mark the benchmark as challenging, while open-source models trail advanced proprietary LVLMs.Open-source mean accuracies range from 26.83% to 53.29% without RAG and from 28.90% to 59.28% with ground-truth knowledge.Source: §3.2 · p. 7
  • Ground-truth image knowledge improves all models, but retrieved images often reduce open-source performance while proprietary models continue to improve.The reported ground-truth improvements range from 2.07% to 11.31% for open-source models and from 5.64% to 9.69% for proprietary models.Source: §3.2 · p. 7
  • GPT-4o leads most categories, but performance declines on incomplete scenarios where humans achieve 58.82% and 83.33% with ground-truth knowledge.GPT-4o exceeds human performance on all perspective scenarios and on temporal and deformation scenarios, but not on partial, incomplete, and biological scenarios.Source: §3.2 · pp. 7–8
  • Gemini Pro can use correct retrieved examples despite noise, whereas LLaVA-Next-Interleave follows bad examples and answers incorrectly.The qualitative case contains two correct and three false retrieved examples.Source: §3.2 · p. 8, Figure 4 · p. 8

4 ANALYSIS

The analysis finds that visual knowledge is more helpful than textual knowledge, that retriever quality strongly relates to LVLM accuracy, and that performance peaks with a limited number of image examples.

4 ANALYSIS

  • The analysis examines visual-versus-textual benefit, retriever effects, and the number of ground-truth images LVLMs can effectively leverage.These questions organize the three quantitative analyses in the section.Source: §4 · p. 8

4.1 HOW MUCH CAN VISUAL KNOWLEDGE BENEFIT MORE THAN TEXTUAL KNOWLEDGE?

  • 2.36% and 2.34% image-over-text improvements occur with retrieved knowledge for LLaVA-Next-Interleave and GPT-4-Turbo, respectively.With ground-truth knowledge, the corresponding image-over-text improvements are 11.90% and 3.87%.Source: §4.1 · pp. 8–9

4.2 HOW DOES RETRIEVER PERFORMANCE AFFECT LVLMS?

  • 95% positive correlation links multimodal retriever Recall@5 with LLaVA-Next-Interleave accuracy when retrieved images are used.Despite similar Recall@5 scores for CLIP and VISTA, their overall accuracies differ by 2.07%, suggesting retrieved-example order may also matter.Source: §4.2 · p. 9, Figure 5 · p. 9

4.3 HOW MANY GROUND-TRUTH IMAGE EXAMPLES ARE NEEDED?

  • 5.64% is the greatest improvement from one ground-truth image example, while performance peaks at 10 examples and is 0.29% higher than with 20.The experiment averages LLaVA-Next-Interleave results across three random seeds and evaluates 1, 2, 3, 5, 10, and 20 examples.Source: §4.3 · p. 9, Figure 5 · p. 9

5 RELATED WORK

Related work spans multimodal RAG benchmarks, LVLMs, and retrieval-augmented multimodal language models, while MRAG-BENCH targets a missing vision-centric evaluation setting.

5.1 MULTIMODAL RETRIEVAL-AUGMENTED GENERATION BENCHMARKS

  • Existing benchmarks such as OK-VQA and A-OKVQA primarily evaluate visual questions requiring external textual or world knowledge.Later benchmarks expand knowledge bases for knowledge-intensive and information-seeking visual questions.Source: §5.1 · p. 10

5 RELATED WORK

  • MRAG-BENCH fills this gap by focusing on scenarios where retrieving visual information is more helpful than retrieving text.This focus distinguishes it from benchmarks centered on external textual knowledge.Source: §5.1 · p. 10, §5.3 · p. 10

5.2 LARGE VISION LANGUAGE MODELS

  • LVLM research has developed models and interleaved image-text training that support in-context learning and processing information from multiple images.Examples include Flamingo, Emu, Idefics, VILA, MMC4, OBELICS, Mantis-Instruct, LLaVA-Next-Interleave, and LLaVA-OneVision.Source: §5.2 · p. 10

5.3 RETRIEVAL-AUGMENTED MULTIMODAL LARGE LANGUAGE MODELS

  • Retrieval-augmented multimodal language-model research uses external multimodal knowledge, but a systematic vision-centric benchmark for leveraging visual knowledge remains lacking.Prior work includes multimodal generation enhancement and direct use of web images.Source: §5.3 · p. 10

6 CONCLUSION

MRAG-BENCH evaluates vision-centric retrieval-augmented multimodal models and shows that visual augmentation yields larger gains than textual knowledge. GPT-4o still uses retrieved knowledge less effectively than human participants.

6 CONCLUSION

  • MRAG-BENCH provides a vision-centric benchmark for retrieval-augmented multimodal models, emphasizing visually augmented knowledge.Source: §6 · p. 10
  • Visually augmented knowledge produces greater improvements on MRAG-BENCH than textual knowledge across the evaluated LVLMs.Source: §6 · p. 10
  • GPT-4o improves only 5.82% with relevant retrieved information, versus a 33.16% improvement for human participants.Source: §6 · p. 10
  • The findings motivate developing LVLMs that can utilize retrieved visual knowledge more effectively.Source: §6 · p. 10

A.1 DATASET CURATION DETAILS

The dataset curation process constructs transformative visual examples from manually filtered web searches and applies automatic and expert quality controls. Search terms pair an image object with a condition, while ground-truth searches remove the condition except in the biological scenario.

Dataset collection of transformative aspect

  • Search keywords combine an image object with a condition, then ground-truth searches retain only the image object except for biological queries.Human selection is applied to downloaded search results to ensure quality.Source: §A.1 · pp. 18–19
  • The curated transformative examples cover temporal, deformation, incomplete, and biological changes across diverse objects.Examples include animals or structures under construction, damaged vehicles, keyboards missing keys, and oxidized fruit or vegetables.Source: §A.1 · pp. 19–21

Quality control

  • Automatic checks verify MCQA format, metadata, image accessibility, and removal of redundant corpus images.Source: §A.1 · p. 21
  • Two field experts manually verify query–ground-truth correspondence and filter or revise ambiguous questions and uncorrelated images.Source: §A.1 · p. 22

A.2 HUMAN EVALUATION PROTOCOL

Human evaluation compares answering without retrieval-augmented knowledge against answering with ground-truth retrieval examples. The two conditions are presented through separate evaluation interfaces.

A.2 HUMAN EVALUATION PROTOCOL

  • Three domain human annotators conduct evaluations with and without retrieval-augmented knowledge.Source: §A.2 · p. 22
  • Figure 6 shows the human evaluation interface without RAG examples, while Figure 7 shows the interface with ground-truth RAG examples.Source: Figure 7 · p. 22, Figure 6 · p. 22

B EXPERIMENT SETTING DETAILS

The experiment prompts contain an instruction, image input, question, answer choices, and an answer prefix. RAG evaluation adds retrieved examples after the input image and asks the model to answer directly with an option letter.

B.1 MODEL PROMPTS

  • The common prompt structure includes an instruction, question, choices, and an answer prefix, with images inserted at an image placeholder.Source: §B.1 · p. 23
  • The no-RAG prompt presents one image followed by the question, four choices, and a direct option-letter answer request.Source: §B.1 · p. 23
  • The RAG prompt presents the input image first and additional retrieved examples afterward before the question and choices.It instructs the model to answer directly with the chosen option’s letter.Source: §B.1 · p. 23

B.2 EVALUATION TOOL

The evaluation tool uses a prompt template to extract multiple-choice answers from model responses.

  • The prompt template extracts a multiple-choice answer from the model’s response.Source: Table 5 · p. 23
  • The template includes in-context examples for answer extraction.Source: Table 5 · p. 23
  • The evaluation tool operationalizes answer extraction rather than describing the benchmark’s retrieval procedure.Source: Table 5 · p. 23

C MORE RESULTS

The additional results report retriever recall across MRAG-BENCH scenarios and examine how retriever choice affects LLaVA-Next-Interleave accuracy.

Related papers

Loading 2410.08182v2…