MRAMG-Bench: A Comprehensive Benchmark for Advancing Multimodal Retrieval-Augmented Multimodal Generation

1 Introduction

• Users increasingly favor visual content, and combining images with text can improve clarity in scenarios such as recipes and tourist-spot descriptions.The introduction also notes that an image alone can be the most direct answer when asked about an appearance.Source: §1 · pp. 1–2
• MRAMG targets the gap between multimodal retrieval and text-only answers by generating responses that integrate text and images.The task aims to leverage multimodal information in the corpus rather than describe retrieved images only in text.Source: §1 · pp. 1–2
• The benchmark addresses the lack of suitable datasets and rigorous metrics for evaluating integrated text-image answers.The paper identifies this evaluation gap as a limitation for progress on the task.Source: §1 · p. 2
• MRAMG-Bench comprises six curated datasets with 4,346 documents, 14,190 images, and 4,800 QA pairs across Web, Academia, and Lifestyle.The benchmark includes hierarchical difficulty and order-based reasoning; its challenges include determining image number, selection, and ordering.Source: §1 · p. 2
• The paper proposes statistical and LLM-based evaluation alongside a flexible generation framework that combines rule-based and model-based approaches for LLMs and MLLMs.It also reports a comprehensive evaluation of 11 generative models.Source: §1 · pp. 2–3

2.1 Multimodal RAG

• MuRAR [51] and M2RAG [25] address multimodal answer generation, but M2RAG's repeated model calls can be computationally costly; this work uses a single generation invocation.Other cited approaches retrieve image-text pairs or associate image regions with external knowledge, while often producing text outputs.Source: §2.1 · p. 3

2.2 RAG Benchmarks

• Text-based RAG benchmarks such as MS-MARCO, TriviaQA [17], HotpotQA [46], Natural Questions, and SQuAD [33] do not evaluate tasks requiring both textual and visual information.The paper introduces a benchmark for evaluating MRAMG to address this gap.Source: §2.2 · p. 3

2.3 Multimodal RAG Benchmarks

• Existing multimodal benchmarks cover tasks including multimodal reasoning, image retrieval, and joint text-table reasoning, but WebQA and M2RAG [25] have limitations for evaluating integrated answers.WebQA provides textual answers without textual support for reasoning over retrieved images; M2RAG has 200 questions, web-only data, automated generation without manual verification, and no query ground truth.Source: §2.3 · p. 3

3 Task Formulation

• The knowledge base contains multiple documents, each represented as interleaved text blocks and images.Each text block is a semantic paragraph, and each image belongs to the document's sequence.Source: §3 · p. 3
• The answer is generated from the query, its top-k relevant documents, a generation framework, and a foundation model that may be an LLM or MLLM.The framework selects appropriate images from retrieved documents and integrates them with generated text.Source: §3 · p. 3
• MRAMG's output is a multimodal answer formed by choosing images from retrieved documents and combining them with generated text.The task takes a text-form user query and its associated multimodal information as input.Source: §3 · p. 3

• The datasets cover Web, Academia, and Lifestyle, including MRAMG-Wit, MRAMG-Wiki, MRAMG-Web, MRAMG-Arxiv, MRAMG-Recipe, and MRAMG-Manual.Source: §4 · p. 3
• The benchmark construction pipeline has three stages: data selection and preprocessing, QA generation and refinement, and data quality check.Source: §4 · p. 3, Figure 2 · p. 4
• Annotators and experts conduct a three-stage review as part of the data quality check.Source: Figure 2 · p. 4

4.1.1 Data Collection.

• The three Web datasets use mostly short, single-image documents and are classified as level 1: easy.Source: §4.1.1 · p. 4
• Academic data uses 110 selected arXiv papers and is level 2: medium, with QA instances generally containing 1 to 4 images.Source: §4.1.1 · pp. 4–5
• Lifestyle data is level 3: difficult, with dense images, varied answer types, and challenging image-insertion ordering.Source: §4.1.1 · p. 5

4.1.2 Data Filtering and Context Enhancement.

• Web contexts are filtered for length, image information, and image size, deduplicated, and supplemented with crawled or GPT-4o-extended text.Source: §4.1.2 · p. 5
• Academic annotators select text and corresponding figures from papers to create aligned interleaved contexts.Source: §4.1.2 · p. 5
• Lifestyle contexts combine recipe-step images with text and preserve manual image-text order, followed by annotation edits and GPT-4o and human verification.Manuals are parsed into markdown while preserving original order; annotators remove sensitive or irrelevant content, adjust image placements, and correct text errors.Source: §4.1.2 · p. 5, §4.1.3 · p. 5

4.2 QA Generation and Refinement

• The stage comprises question generation, answer generation, and QA pair refinement.Source: §4.2 · p. 5

4.2.1 Question Generation.

• Question creation varies by dataset: original questions are retained for MRAMG-Web, GPT-4o generates questions for MRAMG-Wiki, MRAMG-Wit, and MRAMG-Recipe, and annotators create questions for MRAMG-Manual and MRAMG-Arxiv.Questions are based on their contexts and should be natural, practical, and answerable through text-image integration.Source: §4.2.1 · p. 5

4.2.2 Answer Generation.

• Web answers are generated from validated questions and extracted context evidence, then enriched with images in appropriate positions.Academic annotators incorporate images or tables; Recipe answers differ for recipe-specific versus step-specific questions, while Manual answers are formulated by annotators.Source: §4.2.2 · p. 5

4.2.3 QA Pair Refinement.

• QA refinement extracts supporting context evidence, checks its alignment with answer keywords, resolves contradictions, and removes irrelevant information.Source: §4.2.3 · pp. 5–6

• The quality-check process has three steps: annotator review, correction of identified issues, and a comprehensive final recheck.Source: §4.3 · p. 6
• Reviewers assess QA correctness and consistency with contexts and image descriptions, with GPT-4o also used for QA evaluation.Source: §4.3 · p. 6
• A separate annotator group corrects issues in image insertion, question formulation, context alignment, image descriptions, and answer correctness.Expert annotators additionally review and correct MRAMG-Arxiv.Source: §4.3 · p. 6

4.4 Data Statistics

• The six datasets cover Web, Academia, and Lifestyle and contain 4,800 QA pairs, 4,346 documents, and 14,190 images.The datasets are MRAMG-Wit, MRAMG-Wiki, MRAMG-Web, MRAMG-Arxiv, MRAMG-Recipe, and MRAMG-Manual.Source: §4.4 · p. 6
• Tasks are categorized into three difficulty levels; Recipe and Manual have longer texts and higher image density than other datasets.Multimodal Element Density reflects the proportion of images to text within a document.Source: §4.4 · p. 6
• Single-image questions test direct image-text associations, while two-image and three-or-more-image questions target comparative or multi-step reasoning.Text-only questions test textual reasoning and whether models exclude irrelevant images.Source: §4.4 · p. 6

5 Generation Framework for MRAMG Task

• The framework has document retrieval and answer generation stages, with LLM-based, MLLM-based, and rule-based strategies for generating multimodal answers.The rule-based strategy first generates text and then inserts selected images.Source: §5 · p. 6

5.1 Retrieving Relevant Documents

• Retrieval selects the top-k relevant documents by query-document cosine similarity, then concatenates their text and image blocks as generation input.BGE-M3 supplies the document embeddings.Source: §5.1 · p. 6

5.2.1 LLM-Based Method.

• LLM-based generation uses image placeholders in retrieved text and treats surrounding context and image captions as textual substitutes for images.The resulting answer represents output images with placeholders.Source: §5.2.1 · p. 6

5.2.2 MLLM-Based Method.

• MLLM-based generation supplies retrieved text and the top-N query-similar images, since MLLMs impose limits on the number of input images.Output images are represented as placeholders.Source: §5.2.2 · p. 6

5.2.3 Rule-Based Method.

• Rule-based generation matches answer sentences to retrieved images using BLEU string similarity and BGE-M3 semantic similarity, retaining edges above either threshold.Each edge combines the two similarity scores using a weighted sum.Source: §5.2.3 · p. 7
• The insertion optimization allows each image once; under one constraint several images may follow a sentence, while the other permits at most one image per sentence.The second constraint uses weighted maximum bipartite matching solved with Edmonds’ Blossom algorithm [10].Source: §5.2.3 · p. 7
• The Graph Matching Algorithm performed better overall than the alternative matching algorithm and was selected for subsequent large-scale evaluations.The comparison covers Web and Manual datasets.Source: Figure 3 · p. 7, §5.2.3 · p. 7

6.1 Experimental Baselines and Settings

• The evaluation covers four closed-source and seven open-source models, with method testing varying by model group; retrieval uses BGE-M3 top-k = 10 chunks and GPT-4o judges LLM-based metrics.Documents are chunked at size 256 after images are replaced with placeholders.Source: §6.1 · p. 7

6.2 Evaluation Metric

• Retrieval uses Context Recall and Visual Recall, while generation metrics include image precision, recall, F1, ordering, text similarity, relevance, effectiveness, position, and comprehensive quality.Image Ordering Score is applied only to Lifestyle data.Source: §6.2.1 · p. 8, §6.2.2 · p. 8

6.3.1 Retrieval Performance.

• MRAMG-Web achieves Visual Recall of 0.99 and Context Recall of 0.95; both retrieval metrics decline on the three other datasets, though overall retrieval remains strong.The paper attributes the decline to longer text passages and higher image counts.Source: §6.3.1 · p. 8

6.3.2 Generation Performance.

• Overall generation performance decreases as dataset complexity increases; rule-based methods struggle on simpler datasets with one image per document but align better as image density rises.The paper links higher image counts to better sentence-image alignment for the rule-based method.Source: §6.3.2 · p. 9
• Advanced models outperform roughly 7B open-source models across domains and methods; on Lifestyle, Qwen2-VL-72B-Instruct scores 26.25 Image Recall with MLLM-based generation, versus Gemini’s 57.4.Larger open-source models narrow the gap on simpler datasets.Source: §6.3.2 · p. 9
• No model or method passes the Lifestyle image-ordering threshold; Gemini’s LLM-based score is highest at 54.62.GPT-4o leads among the reported rule-based and MLLM-based scores, but both remain below the passing threshold.Source: §6.3.2 · p. 9
• Across generation methods, the overall performance trend is LLM-based > MLLM-based > rule-based; on Web Data, Gemini averages 85.85, 83.3, and 41.21, respectively.Table 4 reports comprehensive benchmark results.Source: Table 4 · p. 8, §6.3.2 · p. 9
• LLM-based insertion uses image context; rule-based generation avoids context-window and image-input limits and reduces costs, while MLLM performance deteriorates on harder datasets.Rule-based costs are reported as up to one-third lower than LLM-based methods and half those of MLLM-based methods.Source: §6.3.2 · p. 9

7 Conclusion

• Evaluations of 11 popular generative models highlight limitations on challenging datasets and in selecting the correct image order.Source: §7 · p. 9
• MRAMG-Bench contains 4,800 QA pairs spanning diverse domains and difficulty levels.Source: §7 · p. 9
• Its evaluation strategy combines statistical and LLM-based metrics to assess retrieval and generation performance.Source: §7 · p. 9
• The proposed general MRAMG framework enables models to produce interleaved text-image responses.Source: §7 · p. 9