Source-linked AI summary
Multimodal Fact-Level Attribution for Verifiable Reasoning
David Wan, Han Wang, Ziyang Wang, Elias Stengel-Eskin, Hyunji Lee, Mohit Bansal
TL;DR
Existing multimodal grounding evaluations rarely test whether claims in complex, multi-step reasoning are supported across heterogeneous sources. MuRGAT introduces a fact-level benchmark and decomposed scoring protocol for explicit, modality- and time-specific citations; experiments show strong reasoning can coexist with hallucinated grounding, while structured grounding may reduce answer accuracy.
Problem
Existing multimodal grounding benchmarks focus on simplified observation-based settings or limited modalities, leaving attribution in complex multimodal reasoning insufficiently assessed.
Method
MuRGAT evaluates answers grounded in video, audio, and figures through verifiable-claim identification, atomic fact decomposition, and citation-quality assessment.
Results
MLLMs often reason correctly yet hallucinate citations, while explicit grounding improves attribution by 9.6 MURGAT-SCORE on average but can reduce reasoning performance.
Takeaways & Limitations
Reasoning ability does not guarantee verifiable attribution, and attribution quality remains distinct from question-answering accuracy.
Takeaways & Limitations
Programmatic grounding improves attribution but reduces answer accuracy by an average of 7.4 points.
Abstract
from arXiv · showhide
Multimodal large language models (MLLMs) are increasingly used for real-world tasks involving multi-step reasoning and long-form generation, where reliability requires grounding model outputs in heterogeneous input sources and verifying individual factual claims. However, existing multimodal grounding benchmarks and evaluation methods focus on simplified, observation-based scenarios or limited modalities and fail to assess attribution in complex multimodal reasoning. We introduce MuRGAt (Multimodal Reasoning with Grounded Attribution), a benchmark for evaluating fact-level multimodal attribution in settings that require reasoning beyond direct observation. Given inputs spanning video, audio, and other modalities, MuRGAt requires models to generate answers with explicit reasoning and precise citations, where each citation specifies both modality and temporal segments. To enable reliable assessment, we introduce an automatic evaluation framework that strongly correlates with human judgments. Benchmarking with human and automated scores reveals that even strong MLLMs frequently hallucinate citations despite correct reasoning. Moreover, we observe a key trade-off: increasing reasoning depth or enforcing structured grounding often degrades accuracy, highlighting a significant gap between internal reasoning and verifiable attribution.
1. Introduction
MURGAT targets fact-level attribution in multimodal reasoning settings that go beyond direct observation, requiring precise modality- and time-specific citations. Its evaluation decomposes responses into verifiable claims, atomic facts, and attribution quality, revealing strong reasoning models often hallucinate grounding and face trade-offs between citation quality and task accuracy.
- Motivation and benchmark: MURGAT evaluates multimodal answers that synthesize evidence from video, audio, and figures while requiring precise citations for each claim.Citations specify the supporting modality and temporal segments.
- Evaluation protocol: The evaluation protocol identifies verifiable claims, decomposes them into atomic facts, and assesses whether cited multimodal evidence entails each fact.The protocol avoids penalizing unobservable reasoning statements while measuring citation recall and precision.
- Evaluation protocol: Pearson correlation reaches 0.84 across evaluation steps, versus r = 0.59 for the next-best LLM-as-judge baseline.The automatic evaluation is calibrated against human annotations on WorldSense and Video-MMMU.
- Findings: Models often achieve high question-answering accuracy but frequently produce hallucinated grounding through incorrect citations.This exposes a gap between answering correctly and attributing claims to valid evidence.
- Findings: Citation generation acts as a reasoning tax on simple recognition tasks but scaffolds performance on complex reasoning benchmarks.The effect of citation generation depends on task complexity.
2. Related Work
Prior multimodal attribution work addresses grounding and retrieval, but commonly emphasizes direct evidence matching, limited modalities, answer accuracy, or temporal localization rather than fine-grained attribution of generated claims.
- Attribution and grounding: Text attribution research evaluates citation quality and often decomposes outputs into atomic facts for finer-grained assessment.Atomic decomposition addresses sentences containing multiple factual claims.
- Attribution methods: Existing attribution methods use direct generation, post-retrieval evidence selection, or post-generation claim verification and revision.These approaches organize text-domain attribution around when and how evidence is incorporated.
- Multimodal grounding: Multimodal methods improve retrieval, reasoning, or temporal grounding but typically evaluate answer accuracy or localization rather than fine-grained attribution of generated claims.This leaves fact-level multimodal attribution underassessed.
3. Task and Evaluation
MURGAT evaluates whether multimodal models support fact-level claims with precise citations while distinguishing observable content from reasoning. Its protocol identifies verifiable sentences, decomposes them into atomic facts, evaluates citation entailment, and combines coverage with attribution quality.
- MURGAT: MURGAT gives models heterogeneous inputs and requires explicit reasoning plus citations tied to specific modalities and timestamped segments.Citations may refer to sources such as video, audio, or figures, and verifiable claims must be strictly entailed by them.
- Evaluation protocol: The evaluation pipeline identifies verifiable, citation-covered sentences while filtering out reasoning statements that cannot be directly grounded.The pipeline retains sentences with non-empty citation sets before further decomposition.
- Evaluation protocol: Each retained sentence is decomposed into minimal independently verifiable atomic facts, with its citation set propagated to every derived fact.Decontextualization resolves pronouns to specific entities before attribution evaluation.
- Attribution quality: Attribution quality uses citation precision and recall to test whether cited multimodal evidence is relevant and sufficient to entail each atomic fact.The two-sided verification captures both missing evidence and incorrect or unnecessary citations.
- Evaluation metrics: MURGAT-SCORE combines citation coverage with attribution quality, penalizing systems that ground only a small subset of factual claims.Coverage measures cited verifiable sentences, while attribution quality is summarized as the harmonic mean of precision and recall.
4. Automatic Evaluation
The automatic evaluation pipeline decomposes multimodal attribution into verifiable-claim identification, atomic fact decomposition, and attribution quality, then combines specialized models into MURGAT-SCORE. Fine-grained evaluation correlates strongly with human judgments, while explicit decontextualization and sentence-level processing improve decomposition quality.
- Pipeline and human evaluation: Human annotations evaluate verifiable claims, atomic facts, and attribution quality on Video-MMMU and WorldSense outputs from strong MLLMs.The annotated examples cover multimodal inputs and complex queries requiring reasoning beyond direct observation.
- Subtask 1: Verifiable Claim Identification: 84.2 BAcc is achieved by Gemini-3-Pro with the JSON prompt for verifiable claim identification, followed by 81.4 BAcc with its CoT prompt.Balanced Accuracy is used because the task has binary decisions and unbalanced labels.
- Subtask 2: Atomic Fact Decomposition: Atomic fact decomposition evaluates generated-reference overlap with F1 and checks whether citations are exactly propagated from source sentences to derived facts.Citation lists must contain neither missing nor additional citations relative to the original sentence.
- Subtask 2: Atomic Fact Decomposition: Sentence-level generation consistently outperforms response-level generation, while omitting or internalizing explicit decontextualization hurts performance.These results support breaking the complex decomposition problem into smaller explicit subtasks.
- Subtask 3: Attribution Quality: 72.9 F1 from Gemini-2.5-Flash with the Simple prompt is only 0.2 points below the 73.1 F1 best achieved by Gemini-3-Pro with JSON prompting.Gemini-2.5-Flash is selected as the default entailment model to maximize efficiency.
- End-to-end metric validation: MURGAT-SCORE achieves r = 0.97 for coverage and consistently outperforms prompting-based baselines across coverage, attribution precision, attribution recall, and the overall score.Sentence-level baselines correlate better than response-level approaches, with coverage at r = 0.76 versus r = 0.58.
5. Generation Experiments
Generation experiments show that multimodal attribution remains substantially harder than answering correctly, with citation effects, post-hoc grounding, model modality, and reasoning effort varying by task. Programmatic grounding improves attribution but introduces an accuracy trade-off.
- 5.2. Main Results: Models reach peak MURGAT-S scores of 69.2 on WorldSense and 56.9 on Video-MMMU, while failing to ground roughly 30–35% of claims.Coverage is generally high, but precise temporal attribution remains the bottleneck.
- 5.2. Main Results: Citation requirements reduce Gemini-3-Pro accuracy from 71.4% to 70.0% on WorldSense but improve it from 85.3% to 86.0% on Video-MMMU.Qwen3-VL-Thinking rises from 51.0% to 60.0% on Video-MMMU, indicating task-dependent citation effects.
- 5.2. Main Results: Gemini-3-Pro and Gemini-3-Flash match 86.0 accuracy on Video-MMMU, but Gemini-3-Flash has higher MURGAT-S than Gemini-3-Pro, 56.9 versus 41.8.Correct answers therefore do not guarantee faithful citations.
- 5.2. Main Results: Post-hoc attribution improves Gemini-3-Pro’s WorldSense MURGAT-S from 51.7 to 65.2 but lowers Gemini-2.5-Flash’s VideoMMMU score from 41.5 to 38.0.On reasoning tasks, post-hoc methods can force-align abstract reasoning steps to unrelated segments.
- 5.3. Impact of Reasoning Effort: Gemini-3-Flash attribution declines from 69.7 to 64.4 as reasoning effort increases, whereas Gemini-3-Pro gains 6.1 MURGAT-SCORE points and 7.4 accuracy points.The models therefore exhibit diverging scaling trends under increased thinking effort.
- 5.4. Programmatic Multimodal Grounding: Program-aided methods improve attribution by an average of 9.6 MURGAT-SCORE points, with Logic Imperative reaching 76.4, but reduce answer accuracy by an average of 7.4 points.Imperative grounding outperforms Declarative grounding, 76.4 versus 74.3.
6. Conclusion
The paper introduces MURGAT and MURGAT-SCORE to evaluate fact-level multimodal attribution in complex reasoning settings. Experiments expose a gap between reasoning accuracy and faithful grounding, alongside failure modes from post-hoc and programmatic methods.
- 6. Conclusion: MURGAT evaluates whether MLLMs synthesize answers from video, audio, and figures while providing precise evidentiary support.MURGAT-SCORE is a decomposed, fine-grained automatic evaluation pipeline designed to correlate with human judgments.
- 6. Conclusion: Experiments show that the capability to reason does not imply the capability to ground.The paper identifies hallucinated mappings in post-hoc methods and a trade-off between programmatic rigor and narrative accuracy.
- 6. Conclusion: The human annotation protocol covers atomic decomposition, verifiable claim identification, and attribution quality for fine-grained evaluation.This protocol provides the annotation basis for assessing attribution beyond sentence-level judgments.
A.1. Data and Models
The appendix describes the multimodal datasets, model inputs, annotation workflow, and agreement patterns used to support evaluation. It emphasizes atomic-fact verification and separate assessment of citation support.
- Data and Models: The evaluation samples Video-MMMU and WorldSense, which require visual and audio reasoning, and evaluates Gemini and Qwen3-Omni MLLMs.Video-MMMU emphasizes figures and graphs with audio, while WorldSense emphasizes video and audio interpretation.
- Annotation Guidelines: Annotators decompose complex sentences into independent atomic units, resolve pronouns using prior context, and strip meta-talk.These guidelines reduce information leakage and make claims self-contained.
- Annotation Guidelines: Verifiable claim identification retains facts describing specific visual or audio events, dates, locations, or object absence while filtering reasoning, definitions, and chitchat.The judgment is performed at atomic-fact level for granular coverage.
- Annotation Details: Annotators use a Union Strategy because verifiable-claim identification has 73.7% inter-annotator agreement and disagreements reflect differing sensitivity thresholds.Any atomic fact marked verifiable by at least one annotator is retained.
- Over-citation Analysis: Only 3 of 37 non-verifiable sentences contain model citations, yielding an over-citation rate of 8%.The analysis suggests over-citation affects a small portion of non-verifiable content.
- Attribution Evaluation: Attribution evaluation reaches 86.1% inter-annotator agreement, exceeding the 82.2% pairwise agreement reported for a comparable verification benchmark.Annotators evaluate citation support across recall and precision dimensions.
- Attribution Evaluation: Citation evaluation checks each citation against the atomic fact it supports rather than penalizing all facts in a multi-citation sentence.This produces separate recall and precision values for the relevant facts.
A.6. Citation Placement Analysis
Citation placement and atomic-fact decomposition analyses motivate a staged evaluation pipeline. Inline citations can be assigned selectively, while response-level decomposition performs worse than sentence-level processing.
- Citation Placement: Only 14.8% of sentences contain multiple citations; among those, 79.8% are end-dumped and 20.2% are inline.The placement analysis supports special handling for inline citations and propagation of end-dumped citation sets.
- Evaluation Statistics: The annotation statistics summarize evaluated sentences and facts alongside coverage, attribution recall, precision, F1, MURGAT-S, and accuracy.These measures provide the reporting framework for the annotation and model results.
- Claim Identification: Verifiable claim identification is evaluated at both sentence and atomic-fact granularity using balanced accuracy.The comparison includes multiple model families beyond Gemini.
- Atomic Fact Decomposition: Response-level atomic-fact generation performs noticeably worse than sentence-level generation.The full decomposition results compare the complete pipeline with ablations, including omitted decontextualization and single-pass generation.
- Atomic Fact Decomposition: Explicit decontextualization gives Gemini-3-Pro a 2-point F1 gain, while separating decontextualization and decomposition gives a 3-point gain over single-pass generation.The results support a two-stage pipeline for higher-quality atomic facts.
- Metric Validation: Correlation tables report evaluator agreement with human judgments, bootstrap confidence intervals, and subsampling stability for MURGAT-SCORE.The reported correlations are significant at p < 0.001.
B.3. Robustness to Evaluator Choice
The evaluation framework is tested for evaluator bias and supported by a programmatic grounding design that separates reasoning from attribution. The design explores reasoning paradigms, grounding mechanisms, and runtime verification, while exposing trade-offs in transparency and robustness.
- Evaluator robustness: The multi-model evaluation pipeline performs comparably to single-evaluator alternatives and avoids end-to-end self-evaluation by distributing subtasks across models.Its ranking agreement with human judgments is perfect (τ = 1.00), whereas GPT-5 reverses the top two rankings (τ = 0.33).
- Evaluator robustness: Bootstrap analyses indicate stable rank-level correlations across evaluator configurations and annotation sample sizes.The median Spearman correlation remains approximately 0.84 from 30 to 80 samples, while confidence intervals narrow as sample size increases.
- Evaluator robustness: Manual inspection found 47 of 50 entailment judgments matching the automated pipeline, with disagreements concentrated at ambiguous temporal boundaries.This corresponds to 94% agreement in the inspected sample.
- Programmatic grounding: The framework decouples reasoning from attribution through a plan-then-execute program whose executable modules link claims to retrieved sources.Runtime checks verify that each function output is entailed by its input evidence, preserving attribution standards for atomic operations.
- Programmatic grounding: The design space varies both the reasoning paradigm and grounding mechanism, contrasting logic-centric versus narrative-centric programs and planner-defined versus executor-discovered localization.Planner-defined grounding gives the planner control but depends on its localization ability, whereas executor-discovered grounding shifts performance to retrieval tools.
- Programmatic grounding: Logic-centric programs support verifiable objective queries but can obscure reasoning through opaque intermediate data structures, while narrative-centric programs produce human-readable explanatory traces.Narrative-centric programs prioritize contributive attribution by making intermediate reasoning outputs part of the verification trace.
C.5. Results
Results compare full benchmark, program-aided, reasoning-level, and evaluator-correlation metrics. Structured grounding can improve verifiability while reducing accuracy on complex video reasoning tasks.
- C.5. Results: Programmatic variants reduce Video-MMMU accuracy from 90.0% to 84.7% for Gemini-3-Flash.The authors attribute this penalty to excessive constraints on holistic video understanding despite curbing correct-for-the-wrong-reasons behavior.
- C.5. Results: Table 13 reports Coverage, Attribution Precision, Attribution Recall, Attribution F1, MURGAT-SCORE, and answer accuracy across WorldSense and Video-MMMU variants.The table highlights the best result within each method.
- C.5. Results: Table 14 provides program-aided WorldSense results for Gemini-3-Flash, while Table 15 reports reasoning results across different thinking levels.These tables separate program-aided performance from reasoning-depth comparisons.
- C.5. Results: Table 16 correlates Coverage, Attribution Precision, Attribution Recall, and MURGAT-SCORE with human judgments using Pearson, Spearman, and Kendall coefficients.The table reports correlations for the authors’ evaluation protocol and marks the best results.
- C.5. Results: The appendix provides complete main results, detailed reasoning and program-aided metrics, evaluator correlations, and modality-level attribution precision.These results are distributed across Tables 13–17.
D.2. Qualitative Analysis
Qualitative analyses identify a recurring tension between richer reasoning and precise attribution. Post-hoc and program-aided strategies improve some grounding dimensions but can introduce precision or accuracy costs.
- D.2. Qualitative Analysis: Post-hoc attribution improves recall on perceptual WorldSense tasks by recovering scene details missed during initial reasoning.A second perceptual pass helps ground the descriptive narrative more completely.
- D.2. Qualitative Analysis: Modality-level attribution precision is reported separately for visual, audio, and combined evidence, excluding BASE because it produces no citations.The breakdown includes citation counts for each modality.
- D.2. Qualitative Analysis: On knowledge-intensive VideoMMMU tasks, post-hoc attribution adds grounding overhead that compromises precision.The divergent effect shows that attribution strategies behave differently across perceptual and deductive benchmarks.
- D.2. Qualitative Analysis: The Logic Imperative variant reaches 78.7 F1 attribution but only 60.0 accuracy, below BASE + POST-HOC accuracy of 67.0.This accuracy-attribution gap indicates that stronger verification can coincide with weaker task performance.
E. Prompts
The appendix documents prompts and qualitative examples for automatic evaluation and response generation. These materials illustrate model differences and the contrasting effects of citation and post-hoc attribution strategies.
- E.1. Automatic Evaluation: The automatic-evaluation appendix includes prompts for atomic fact decomposition, decontextualization, verifiability evaluation, and attribution entailment.Verifiability prompts cover Simple, CoT, and JSON variants.
- E.1. Automatic Evaluation: Gemini-2.5-Flash receives a score of 1.0 in the shown automatic-evaluation comparison, versus 0.61 for Gemini-3-Pro.The displayed comparison reports scores for the two Gemini models.
- E.1. Automatic Evaluation: The example citations identify the song title through audio evidence around 0:06–0:07.Both example outputs cite the utterance in the audio modality.
- E.1. Automatic Evaluation: Figure 6 contrasts Gemini 2.5 Flash’s minimalist observation-first approach with Gemini 3 Pro’s higher-level narrative synthesis and lower grounding precision.Pro attempts spatial-layout and song-title synthesis but grounds less precisely than Flash.
- E.2. Response Generation: The response-generation appendix supplies prompts for baseline generation, citation-conditioned generation, and post-hoc refinement.These prompts correspond to the three response-generation configurations.
- E.2. Response Generation: On WorldSense, post-hoc attribution fixes missing recall by adding grounded descriptive scene elements overlooked by the citation baseline.The qualitative heading and comparison label distinguish citation recall failure from the grounded post-hoc result.
- E.2. Response Generation: The WorldSense example adds separate visual citations for preparing the soil and planting seeds at their respective timestamps.The post-hoc output cites preparation at 0:03 and 0:08, then planting at 0:22.
- E.2. Response Generation: Figure 7 shows that post-hoc attribution improves WorldSense recall but can create Citation Salad on VideoMMMU by mapping technical steps to generic introductory frames.The comparison highlights opposite effects across perceptual and technical tasks.