Source-linked AI summary
Seeing Through the MiRAGE: Evaluating Multimodal Retrieval Augmented Generation
Alexander Martin, William Walden, Reno Kriz, Dengjia Zhang, Kate Sanders, Eugene Yang, Chihsheng Jin, Benjamin Van Durme
TL;DR
Text-centric RAG evaluation does not readily assess information grounded in audiovisual sources or verify attribution across multiple sources. MiRAGE evaluates factuality, coverage, and citation quality through subclaims. It outperforms text-centric metrics on text while being the only metric reported to generalize to multimodal sources.
Problem
Text-centric RAG metrics do not verify information against non-text sources, while multimodal RAG also requires attributing information to the correct source.
Source: §1 · p. 1Method
Results
Takeaways & Limitations
MiRAGE provides consistent subclaim-level assessment of factuality, information coverage, and citation quality across text, image, audio, and video.
Source: §8 · p. 9Automatic MiRAGE variants have weaker agreement on audiovisual sources because claim verification against video remains a bottleneck.
Source: §6.2 · p. 9
Abstract
from arXiv · showhide
We introduce MiRAGE, an evaluation framework for retrieval-augmented generation (RAG) from multimodal sources. As audiovisual media becomes a more prevalent source of information online, RAG systems must integrate such media into generation. Yet, existing evaluation methods for RAG are largely text-centric and do not readily transfer to multimodal settings. MiRAGE is a claim-centric approach to multimodal RAG evaluation, consisting of InfoF1, which assesses factuality and information coverage, and CiteF1, which assesses citation support and completeness. We show that, when applied by humans, MiRAGE strongly aligns with extrinsic judgments of output quality. We additionally introduce an automatic implementation of MiRAGE and compare it to multimodal variants of three prominent text-centric RAG metrics---ALCE, ARGUE, and RAGAS---finding that MiRAGE outperforms all three on text while being the only one to generalize to multimodal sources. We release open-source implementations and outline evaluation methods for multimodal RAG.
1 Introduction
Text-centric RAG metrics do not verify information against non-text sources or provide fine-grained subclaim-level factuality and attribution across multimodal evidence. MiRAGE evaluates factuality, coverage, and citation quality at the subclaim level, and is reported to outperform text-native metrics on text while generalizing to audiovisual sources.
1 Introduction
- Text-centric metrics do not verify information against non-text sources, while RAG evaluation must also attribute information to the correct text, image, audio, or video sources.Existing metrics operate at sentence level, limiting fine-grained factuality judgments; multimodal adaptation also raises computational and cross-modal grounding challenges.Source: §1 · pp. 1–2
- MiRAGE decomposes generated information into subclaims and uses INFOF1 for factuality and coverage and CITEF1 for citation support and attribution.The framework extends subclaim-level factuality assessment to multimodal, multi-source, citation-attributed RAG.Source: §1 · p. 2
- MiRAGE has the highest agreement with human judgments across three datasets and modalities, outperforming text-native metrics on text and generalizing to audiovisual sources.On audiovisual sources, subclaim-level precision (INFOP, CITEP) is the only automatic metric reported to consistently correlate with grounding judgments.Source: §1 · p. 2
- The paper evaluates three RAG datasets and three types of human judgments, finding MiRAGE outperforms text-native metrics on text and generalizes to other modalities.The authors describe this extension across modalities as coming without cost to text-domain performance.Source: §1 · p. 2
2 Related Work
Prior work extends RAG from text to visual documents, speech, and video, while claim-decomposition metrics assess facts against single sources. MiRAGE targets multi-source generation with attribution to the correct source.
2 Related Work
- VideoRAG is the focus because it mirrors long-form TextRAG generation from multiple cited videos, and video combines visual and audio modalities as a challenging testbed.The paper also validates its evaluation independently on text, audio, and video.Source: §2 · p. 2
- Pyramid, FActScore, and VeriScore motivate atomic fact assessment, but prior factuality metrics do not assess citations across multiple sources as MiRAGE does.FActScore and VeriScore verify subclaims against a single text source; MiRAGE addresses multi-source generation where each subclaim must be factual and cited by the correct source.Source: §2 · p. 2
3 TextRAG Metrics
The section reviews text-only RAG metrics and their multimodal adaptations, then identifies limitations in sentence-level assessment, source concatenation, and exhaustive extraction. MiRAGE addresses these issues with subclaim-level factuality and citation evaluation.
3 TextRAG Metrics
- A document is one unit of evidence in any modality, and a prediction is text decomposed into atomic declarative subclaims that may or may not be true.A support function scores a premise-hypothesis pair through entailment or LLM judgment.Source: §3 · p. 3
3.1 ALCE
- ALCE assesses correctness and citation quality with recall and precision, but its sentence-level citation checks combine support across sources, which is infeasible for long multimodal documents.The implementation omits exact-match precision as unsuitable for long-form generation and does not explore fluency.Source: §3.1 · p. 3
3.2 ARGUE
- ARGUE measures information coverage through nugget question-answer pairs and citation support through individual sentence-citation pairs.Sentence Support still evaluates sentences rather than subclaims, so a sentence containing multiple subclaims poses the same support-assessment challenge as ALCE.Source: §3.2 · p. 3
3.3 RAGAS
- RAGAS measures faithfulness, answer relevance, and context relevance; its video adaptation extracts query-relevant observations and uses a detailed VLM summary to approximate the context denominator.Faithfulness checks statements against context documents individually, owing to context limitations.Source: §3.3 · pp. 3–4
3.4 Summary
- ALCE and ARGUE assess information at sentence level, ALCE concatenates sources, and RAGAS assumes exhaustive source-observation extraction; MiRAGE instead evaluates subclaims across dimensions.INFOF1 and CITEF1 can be computed in reference-based or reference-free settings.Source: §3.4 · p. 4
4 MIRAGE
MiRAGE evaluates information quality and citation use by decomposing text into verifiable subclaims. Its metrics assess factuality, coverage, citation support, and proper attribution in reference-based or collection settings.
4 MIRAGE
- MiRAGE evaluates predicted subclaims with INFOF1 for information and CITEF1 for citation use, and supports both collection and reference settings.Reference evaluations use text-based proxies for document content, which reduces multimodal inference but can penalize factual claims omitted from non-exhaustive references.Source: §4 · pp. 4–5
4.1 INFOF1: Evaluating Information Quality
- INFOF1 combines INFOP, which assesses subclaim factuality, and INFOR, which assesses information coverage.Source: §4.1 · p. 5
4.1.1 Information Factuality (INFOP)
- INFOP measures how many predicted subclaims are supported by evidence, weighted by support strength.Collection Precision checks against relevant documents, while Reference Precision checks against reference text or subclaims.Source: §4.1.1 · p. 5
4.1.2 Information Coverage (INFOR)
- INFOR measures the proportion of reference subclaims attested in the prediction; the experiments use its reference version because the collection version has impractical computational costs and task complexity.Source: §4.1.2 · p. 5
4.2 CITEF1: Evaluating Citations
- CITEF1 is an F1 score combining citation support precision with proper-attribution recall over subclaims.Source: §4.2 · p. 5
4.2.1 Citation Support (CITEP)
- CITEP assesses whether predicted subclaims are supported by their cited documents, using either document content or associated reference subclaims.Unlike sentence-level citation checks, subclaim-level evaluation avoids requiring an entire multi-premise sentence to be supported by each citation.Source: §4.2.1 · pp. 5–6
4.2.2 Proper Attribution (CITER)
- CITER measures whether reference subclaims expressed in the prediction are attributed to sources that attest them.It uses the best support across documents and sentences citing each document, and has no collection-versus-reference variant.Source: §4.2.2 · p. 6
5 Grounding Claims in Multimodal Data
The evaluation tests claim-grounding methods across text, audio, video, and audiovisual settings. Most methods struggle with claim verification, while modality-specific and omnimodal approaches show different coverage and quality patterns.
5 Grounding Claims in Multimodal Data
- The evaluation covers text, audio, video, and audiovisual settings using audio-language, vision-language, and omnimodal models.The text datasets include UNLI and PeopleProfile; the audio datasets include CLOTHO and WIKIVIDEO-Audio, with WIKIVIDEO used for audiovisual evaluation.Source: §5 · p. 6
- Most claim-grounding methods struggle with verification; modality-specific models perform decently in their modalities but lack general-purpose coverage, while omnimodal models provide broader coverage at varying quality.Source: §5 · p. 6
- CLUE (omni) and Qwen3.5-27B (vl) achieve strong performance on their respective modalities.Source: §5 · p. 6
6 Alignment With Human Judgments
The evaluation compares automatic RAG metrics with human judgments across text and audiovisual datasets. Human-computed MIRAGE scores align with holistic quality judgments, while automatic MIRAGE is strongest overall and extends to video grounding.
6 Alignment With Human Judgments
- The study spans RAGTIME, WIKIVIDEO, and WIKIVIDEO-25, combining judgments collected by the authors with task-organizer annotations.WIKIVIDEO judgments include holistic quality, intrinsic claim support, and video grounding; organizer judgments differ across the other datasets.Source: §6 · pp. 6–7
6.1 Human-Annotated MIRAGE
- Human-computed INFOF1 strongly aligns with holistic quality judgments, while grounding judgments show no-to-low agreement with them.The authors conclude that quality or coverage measures may miss source groundedness unless they explicitly verify against sources.Source: §6.1 · pp. 7–8
6.2 Automatic Evaluations
- Across three datasets and human-judgment types, MIRAGE shows the strongest, most consistent agreement and is the only metric with positive video-grounding agreement.It extends to multimodal RAG without sacrificing text performance.Source: Table 3 · p. 7, §6.2 · p. 8, Table 6 · p. 8
- Sentence-level evaluation reduces agreement with human judgments across modalities; on RAGTIME, MIRAGE’s subclaim-level CITEP and INFOR outperform ARGUE’s sentence support and nugget coverage.On WIKIVIDEO, ALCE Citation Quality and ARGUE Sentence Support correlate more weakly with EQJs than MIRAGE counterparts and negatively with grounding judgments.Source: §6.2 · p. 8
- Surface metrics are brittle: they fail to assess grounding, depend on high-quality references, and lose agreement across tasks.ROUGE and BERTScore have negative agreement with some human judgments on RAGTIME and near-zero EQJ agreement on WIKIVIDEO-25.Source: §6.2 · p. 9
- Automatic MIRAGE does not match human-level agreement on audiovisual sources because its verifier must ground claims in video.The authors identify improving verification against multimodal evidence as future work.Source: §6.2 · p. 9
7 How to Evaluate Multimodal RAG
The paper recommends different evaluation variants for quick diagnostics versus comprehensive assessment. It also identifies useful complementary metrics and reporting practices.
7 How to Evaluate Multimodal RAG
- For comprehensive evaluation, collection-based INFOF1 and CITEF1 verify claims directly against source documents.For quick or compute-constrained checks, reference-based versions verify claims against a text proxy.Source: §7 · p. 9
- ROUGE and BERTScore remain reasonable proxies when high-quality references exist, while RAGAS Faithfulness and ALCE Claim Recall closely mirror MIRAGE components.Source: §7 · p. 9
- Report Precision and Recall alongside F1, and retain answer relevance as an evaluation dimension.RAGAS Answer Relevance is described as useful, though imperfect.Source: §7 · p. 9
8 Conclusion
MIRAGE evaluates multimodal RAG at the subclaim level through INFOF1 and CITEF1. It outperforms text-centric metrics on text and is the only metric reported to generalize to multimodal sources.
8 Conclusion
- MIRAGE provides consistent subclaim-level assessment of factuality, information coverage, and citation quality across text, image, audio, and video.Source: §8 · p. 9
- MIRAGE outperforms text-centric and sentence-level metrics on text while remaining the only metric to generalize to multimodal sources, aligning with human quality judgments where others do not.Source: §8 · p. 9
Limitations
The limitations concern evaluating refusals and constraints on multi-video inference. The appendix also introduces details about the experimental setup and claim-grounding setup.
Limitations
- MIRAGE does not evaluate refusals; identifying absent information in a large multimodal collection requires strong prior knowledge or perfect retrieval and grounding.Refusal claims are also out-of-distribution for the entailment judges.Source: Limitations · p. 9
- Without multi-video inference, extreme frame-rate reduction can limit VLM verifier performance, while single-video judgments cannot assess cross-source inferences.Source: Limitations · p. 9
A Experimental Setup Details
- The appendix expands on the section 5 and 6 experiments, including data setup, prediction models, and hyperparameters.Source: A Experimental Setup Details · p. 13, §A.1 · p. 13
A.1.1 Data
The appendix details claim-verification data and models, human-alignment prediction settings, MIRAGE variants, and weighted evaluation choices. Its single-source precision variant is more computationally feasible but cannot evaluate cross-source inferences.
A.1.1 Data
- Evaluation covers four claim-verification modality combinations—text, audio, video, and audiovisual; text-only evaluation uses UNLI and PeopleProfile, with labels binarized as supported above 0.5 and unsupported below 0.5.WIKIVIDEO mixes unsupported subclaims into the data to provide a balanced 50/50 label distribution, using unique, randomly selected claims for each modality instance.Source: §A.1.1 · p. 13
- Audio evaluation uses CLOTHO and audio-supported WIKIVIDEO claims, while video-only and audiovisual evaluations use corresponding WIKIVIDEO subclaims.CLOTHO retains contradictory and entailment instances; WIKIVIDEO modality claims are used unmodified.Source: §A.1.1 · p. 13
A.1.2 Models
- Models span audio-only, language-only, vision-language, and omnimodal capability profiles.Source: §A.1.2 · p. 13
A.1.3 Hyperparameters
- The appendix lists model hyperparameters in Table 7.Source: §A.1.3 · p. 13, Table 7 · p. 13
A.2 Human Alignment
- WIKIVIDEO-25 human evaluations provide scalar prediction-quality scores and an overall best-system selection.For WIKIVIDEO, predictions are generated in LLM-only, oracle-video, and RAG settings across 10 topics.Source: §A.2 · p. 14
B MIRAGE ++
- MIRAGE variants address single-source inference and weighted evaluation, with tradeoffs between reference and collection precision.Source: B MIRAGE ++ · p. 14
B.1.1 Single-Source Precision
- Single-source precision compares a subclaim separately with each evidence item and takes the maximum score, making it more computationally feasible but unable to assess cross-source inferences.Source: §B.1.1 · p. 14
B.1.2 Weighted Variations
- Weighted precision and recall use subclaim importance, defined by the prediction author for precision and by evaluation data or consumers for recall.Source: §B.1.2 · p. 14
B.1.3 Implementation Details
The implementation supports CLUE and zero-shot VLM verification, including single-source scoring and source-quality weighting. The section also contrasts reference-based evaluation with collection-based evaluation, noting limitations of non-exhaustive references.
B.1.3 Implementation Details
- CLUE and zero-shot VLM backends support INFOF1 and CITEF1, with scalar or binary prompts selected by modality and a 0.5 support threshold for CLUE.Collection Precision checks each predicted subclaim against each document individually and takes the maximum; Information Coverage reverses prediction and reference roles.Source: §B.1.3 · p. 14, §B.2.3 · p. 15
B.2.1 Single-Source Precision
- Single-source precision scores a subclaim against each cited source separately and takes the maximum, with the document set configurable by first mention or any mention.Subclaim deduplication is recommended.Source: §B.2.1 · p. 15
B.2.2 Weighted Variations
- CITEF1 can weight citation precision and recall by source quality, allowing evaluation preferences to distinguish counterfactual sources or favor informative or preferred source types.Examples include preferring news over social media or the most informative sources so fewer need to be consulted for validation.Source: §B.2.2 · p. 15
B.2.3 Implementation Details
- CITEF1 also supports CLUE and zero-shot VLM verification, using scalar prompts for CLUE and binary prompts for the VLM backend.Source: §B.2.3 · p. 15
B.3 Reference vs. Collection Precision
- Reference precision for CITEF1 can penalize factual subclaims absent from non-exhaustive references, although CITEF1 is intended to assess source grounding rather than topic relevance.Reference-based evaluation is motivated by computational efficiency, while collection precision evaluates against documents related to the topic.Source: §B.3 · p. 15
C Additional Human Alignment Results
This appendix reports additional human-alignment results and details implementations of ALCE, ARGUE, and RAGAS, alongside human-judgment procedures. It describes how these metrics and judgments were adapted or collected for the evaluation setting.
C Additional Human Alignment Results
- Tables 8–10 report additional human-alignment results with CLUE and Qwen3.5, while Table 11 reports alignment between human-annotated INFOF1 precision and recall and extrinsic quality judgments.Source: C Additional Human Alignment Results · p. 15, §E.4 · p. 18
D.1 ALCE
- ALCE uses Claim Recall without exact match; its sentence-level citation checks pose subclaim-verification and multi-video constraints, addressed by downsampling video until it fits GPU memory.The implementation verifies subclaims against cited video, beginning at 1 fps and halving the framerate after out-of-memory errors.Source: §D.1 · pp. 15–16
D.2 ARGUE
- ARGUE evaluates sentence citation support and nugget coverage; this implementation derives question-answer nuggets from reference subclaims and changes sentence verification from text documents to video.Nugget Coverage is similar to ALCE Claim Recall or INFOF1 Recall, while the implementation uses ARGUE’s LLM critic for that score.Source: §D.2 · p. 16
D.3 RAGAS
- RAGAS Faithfulness is adapted to verify one subclaim against one video at a time, while Answer Relevance remains text-based and compares query embeddings.The implementation uses only videos used in generation for Faithfulness.Source: §D.3 · pp. 16–17
- For RAGAS Context Relevance, the implementation extracts query-relevant video information but uses a detailed video summary as the denominator because exhaustive extraction is challenging.On WIKIVIDEO, VLMs often label videos “insufficient information” despite human-annotated relevant claims grounded in those videos.Source: §D.3 · p. 17
E.1 Extrinsic Quality Judgments
- Extrinsic quality judgments ask three native or fluent English-speaking annotators to rate factuality, adequacy, coherence, relevancy, and fluency on a 1–5 scale.Each annotator receives the 10 evaluation instances; the annotations are redundant.Source: E Collection of Human Judgments · p. 17, §E.1 · p. 17
E.2 Intrinsic Claim Judgments
- Intrinsic claim judgments ask annotators whether predicted subclaims are supported by the reference text, mirroring INFOF1-Ref with human verification.Three annotators independently receive the 10 evaluation instances, and the annotations are redundant.Source: E Collection of Human Judgments · p. 17, §E.2 · p. 17
E.3 Grounding Judgments
- Grounding judgments ask annotators whether predicted subclaims are supported by relevant videos; five annotators provide non-redundant judgments, with rankings assessed using Kendall’s Tau.These judgments correspond to INFOP and CITEP with human verification.Source: E Collection of Human Judgments · p. 17, §E.3 · p. 17