Multi-modal Retrieval Augmented Multi-modal Generation: Datasets, Evaluation Metrics and Strong Baselines

Motivation

• Text-only RAG responses can lack clarity in instructional or spatial-reasoning scenarios, motivating responses that combine text with relevant images.The paper airplane example contrasts text instructions alone with explanations incorporating illustrative images.Source: §1 · p. 1

Task challenges

• M2RAG requires models to understand multimodal content and cross-modal relationships and to generate coherent responses integrating text and images.Source: §1 · p. 2

Contributions

• The paper contributes a 10-topic benchmark, text-only and multimodal evaluation metrics, single- and multi-stage strategies, and a filtered training dataset.The training samples are selected using the designed multimodal evaluation metrics.Source: §1 · p. 2

Key findings

• Fine-tuned 7B-8B models on the curated dataset outperform GPT-4o, while the designed metrics correlate strongly with human judgments.The reported findings also show that LLMs consistently outperform MLLMs and that multi-stage generation integrates more relevant images while exhibiting better overall quality.Source: §1 · p. 2

Text-modal RAG

• Text-modal RAG incorporates external knowledge into language-model generation, commonly retrieved with BM25, dense retrieval, or a search engine.A newer approach concatenates retrieved data into the model context.Source: §2 · p. 2

Multimodal RAG

• Multimodal RAG and multimodal models can handle multimodal inputs, yet their outputs remain text-only and cannot provide rich multimodal responses.Source: §2 · pp. 2–3

Multimodal generation

• M2RAG selects multimodal content from multiple inputs to construct responses, rather than generating visual elements.The passage distinguishes it from work that retrieves one image, generates images, or initially produces text without incorporating corresponding multimodal data.Source: §2 · p. 3
• The proposed task emphasizes understanding content and relationships across multimodal inputs, unlike approaches that do not model this content holistically.Source: §2 · p. 3

Task formulation

• M2RAG produces a multimodal response to a query by summarizing a retrieved knowledge base of multimodal documents, whose ordered text and image elements retain their original order.The preserved order keeps images paired with their associated textual context.Source: §3 · p. 3

In-Doc Retrieval

• In-Doc Retrieval selects the top-k relevant elements from the knowledge base and includes an image’s associated text to form a concise refined knowledge base.Source: §3 · p. 3

Generation

• The generative model processes the refined knowledge base with the user query to produce the final response.Source: §3 · p. 3

Pipeline scope

• The framework separates data curation into Steps 1–3 and generation strategies into Step 4; without visual elements, the task defaults to traditional RAG.Source: §3 · p. 3, Figure 2 · p. 4

4.1 Benchmark Construction

• The benchmark construction pipeline comprises query collection, data preparation, and retrieval.These steps establish the data resources used for M2RAG evaluation.Source: §4.1 · p. 4

4.1.1 Query Collection

• Queries from ELI5 are filtered for rich references and visual information, then classified into 11 topics.The topic classification is intended to support dataset balance and investigation of topic sensitivity.Source: §4.1.1 · p. 4

4.1.2 Data Preparation

• Web content is cleaned and segmented, while images are supplemented, quality-filtered, and deduplicated.The pipeline crawls web pages and auxiliary images, removes irrelevant text, and filters images by size and CLIP similarity before deduplication with PHash.Source: §4.1.2 · pp. 4–5

4.1.3 Element Evaluation

• DeepSeek-Chat and MiniCPM-V-2.6 evaluate in-document text and image elements, correlating more strongly with human evaluations than CLIP-based embedding models.The evaluations are used to select relevant elements for generation.Source: §4.1.3 · p. 5

4.2 Evaluation Metrics

• Evaluation uses four text-modal and four multi-modal metrics, with the overall score averaging all metrics on a 0–100 scale.Text metrics cover fluency, relevance, context precision, and faithfulness; multi-modal metrics cover image coherence, helpfulness, reference, and recall.Source: §4.2 · p. 5

4.3 Generation Strategy for M2RAG

• Single-stage and multi-stage strategies jointly model text-image relationships instead of generating text and images independently.Single-stage places selected images into placeholders in one prompt; multi-stage generates text, identifies segments for image insertion, then refines those segments using selected images. Prompts use a structured template, representing images as embeddings for MLLMs or textual descriptions for LLMs.Source: §4.3 · pp. 5–6

4.4 Training Dataset Construction

• A training dataset is created from 3K GPT-4o multi-stage triplets and filtered to 1.6K instances using multi-modal metrics.Text-modal metrics were omitted because evaluating Context Precision and Faithfulness on long generations was costly.Source: §4.4 · p. 6

5.1 Validate Reliability of Evaluation Metrics

• 0.57 > 0.53 on helpfulness: GPT-4o's average correlation exceeded the human average, while MLLMs were otherwise comparable to annotator inner-correlation.The evaluation used 200 randomly selected benchmark samples and three independent annotators.Source: §5.1 · pp. 6–7, Table 1 · p. 7

5.2 Overall Experimental Results

• Qwen2.5-72B-Instruct and GPT-4o were the strongest LLMs under multi-stage generation; among MLLMs, Step-1o scored 83.0 versus Qwen2-VL-72B-Instruct's 79.9.The passage also identifies GPT-4o and Step-1o as outperforming all open-source MLLMs.Source: §5.2 · p. 8
• Reasoner models significantly outperformed standard LLMs in single-stage generation, but that advantage narrowed with multi-stage generation.The authors attribute the narrowing to explicit image-interleaving stages compensating for standard LLMs' lack of built-in reasoning.Source: §5.2 · p. 8
• Llama-3.2-Vision's weaker performance was linked tentatively to single image-text pretraining, unlike Qwen2-VL's multi-image pretraining [7] [32].The authors say multi-image pretraining supports multi-image reasoning and more coherent output in multimodal RAG.Source: §5.2 · p. 8
• Open-source MLLMs generally lagged similarly sized LLMs, with multi-image confusion identified as a challenge for MLLMs [22].LLMs receive detailed image descriptions rather than processing raw visual input, avoiding that issue.Source: §5.2 · p. 8
• Larger open-source models generally scaled better, while separate modeling [41] scored 74.1 versus 74.9 for the weakest joint approach.An exception was Image Recall in single-stage LLMs: Llama-3.1-8B-Instruct scored 79.5 > 66.0 for Llama-3.1-70B-Instruct.Source: §5.2 · p. 8
• Multi-stage approaches outperformed single-stage methods for both LLMs (82.6 > 71.1) and MLLMs (77.1 > 64.3).Table 2 reports overall M2RAG experiment results.Source: Table 2 · p. 7, §5.2 · p. 8

5.3 Improvements from Fine-tuning

• Fine-tuned 7B–8B models improved on all multimodal metrics and overall score, averaging a 19.2% overall-score gain, but underperformed baselines on some textual metrics, especially fluency.Text-modal metrics were not used to construct the training dataset.Source: §5.3 · p. 8

6.1 Does Topic Affect Performance?

• Politics & Government is the most challenging topic, while Society & Culture and Sports are significantly easier.The authors hypothesize that this reflects differences in the prevalence and usefulness of images, and the need for accurate tables and charts in technical fields.Source: §6.1 · p. 9

6.2 Is Description Quality Important?

• Removing context from image descriptions significantly reduces performance under most metrics, while removing detail notably lowers Image Recall.The ablation attributes measurable contributions to both context and detail in GPT-4o’s image descriptions.Source: §6.2 · p. 9

6.3 Are Auxiliary Images Helpful?

• Removing auxiliary images significantly reduces generative performance on multimodal metrics.The authors note the challenge of integrating auxiliary images without contextual information; Table 3 does not report Recall or Overall scores for the no-auxiliary-image condition.Source: §6.3 · p. 9, Table 3 · p. 9

7 Conclusion

• M2RAG asks foundation models to process mixed-modality web pages and generate multimodal answers to user queries.The paper constructs a benchmark to evaluate these capabilities using four text-only and five multimodal fine-grained metrics.Source: §7 · p. 9
• The paper proposes strong baselines for existing models to solve the M2RAG task.Source: §7 · p. 9
• The experiments report phenomena intended to facilitate future research on M2RAG.Source: §7 · p. 9

A Limitations

• The benchmark and training data focus on text and images, while speech, video, and mixed modalities remain future extensions.Source: A Limitations · p. 12
• The proposed metrics may miss aspects of M2RAG evaluation, and evaluating long multimodal outputs across eight dimensions is costly.The authors use a quantity-first approach with 1,000 evaluation samples and plan to develop more efficient metrics.Source: A Limitations · p. 12

C Query Collection

• Queries come from ELI5 and are filtered for detailed responses and visual-information needs, then classified into 11 topics.The classification uses Yahoo Answers Topics; the benchmark’s topic volumes are listed in Table 4.Source: C Query Collection · pp. 12–14, Table 4 · p. 15

D Retrieval of Collected Elements

• LLM- and MLLM-based retrieval scores text and visual elements for relevance, replacing embedding-based methods to better suit online documents that cannot be pre-indexed.Element scores range from 0 to 10, with higher scores indicating greater relevance.Source: D Retrieval of Collected Elements · p. 15

E.1 Basic Statistics

• The appendix points to tables reporting element-score averages and standard deviations, and numerical statistics for the full benchmark.Source: §E.1 · p. 16, Table 5 · p. 17, Table 6 · p. 17

E.2 Data Distribution

• The data-distribution figures cover non-empty web pages per query, valid text pieces per page, and text-piece scores.Source: §E.2 · p. 17, Figure 9 · p. 18, Figure 10 · p. 18
• Separate figures cover web-page image counts per query and scores for web-page images.Source: §E.2 · p. 17, Figure 12 · p. 18, Figure 13 · p. 18
• The appendix also plots auxiliary-image counts per query and auxiliary-image scores.Source: §E.2 · p. 17, Figure 14 · p. 18, Figure 15 · p. 18

F.1 Prompt Templates

• GPT-4o prompt templates are provided for Fluency and Relevance evaluations.Source: §F.1 · p. 19, Figure 16 · p. 19, Figure 17 · p. 20
• Prompt templates are also provided for Image Coherence, Image Helpfulness, and Image Reference evaluations.Source: §F.1 · p. 19, Figure 18 · p. 21, Figure 19 · p. 22

• Table 7 reports average evaluation costs for experiments using the joint modeling approach presented in Table 2.Source: §F.2 · p. 23, Table 7 · p. 24
• Text-modal metric costs cover an entire generated response, whereas multi-modal metric costs are per output image.Source: Table 7 · p. 24
• Context Precision and Faithfulness use GPT-4o-mini; all other evaluation metrics use GPT-4o.Source: Table 7 · p. 24

G Experimental Setup

• The evaluation includes advanced open- and closed-source LLMs and MLLMs, plus a separate MuRAR approach using GPT-4o [41].Source: G Experimental Setup · p. 24, Table 8 · p. 24

H Implementation Details of Model Distillation

• Qwen2.5-7B-Instruct, Llama-3.1-8B-Instruct, and Qwen2-VL-7B-Instruct are fine-tuned with LoRA using a 32,768-token context.Training used 2 Nvidia A100-SXM4-80GB GPUs, a global batch size of 64, and 3 epochs.Source: H Implementation Details of Model… · p. 24

I Results of Representative Models on the Entire Benchmark Dataset

• Table 9 presents overall experiment results for representative models on the entire benchmark dataset.Source: Table 9 · p. 25, I Results of Representative Models on… · p. 25

J Case Study

• The case studies compare single- versus multi-stage approaches, large versus small models, and separate versus joint modeling.The comparisons are illustrated in Figures 21, 22, and 23, respectively.Source: J Case Study · p. 25, Figure 21 · p. 25, Figure 22 · p. 26