PaperlayerView exact revision on arXiv

Source-linked AI summary

Signal or Noise? Modality Contribution and Cooperation in Multimodal GraphRAG

Antonios Georgakopoulos, Paul Groth, Lise Stork

arXiv:2609.35304v1cs.IR

TL;DR

The paper examines whether multimodal evidence contributes equally or cooperates in GraphRAG question answering. It adds modality-aware provenance and uses SHAPE to analyze contributions and cooperation in DocVQA. Text and tables often contribute most, while modality combinations often show redundancy, supporting task-aware evidence selection.

  • Problem

    How modalities contribute and whether their evidence is complementary or redundant across questions, models, and tasks remain underexplored in multimodal GraphRAG.

    Source: §1 · p. 1

  • Method

    The authors extend a KG-based QA framework with modality provenance and use SHAPE to analyze modality contributions and cooperation.

    Source: §1 · p. 1, §4 · p. 3, §5.5.2 · p. 5

  • Results

    Across the evaluated benchmarks and models, text and tables often contribute most, while combinations often show redundancy rather than synergy, especially for pairs involving text; cooperation also depends on question intent and task type.

    Source: §6.1.1 · p. 7, §7.1 · p. 11, §9 · p. 13

  • Takeaways & Limitations

    The findings support selective, modality-aware retrieval that filters modalities according to the downstream task rather than retrieving them uniformly.

    Source: §9 · p. 13

    The analysis uses RAG-Anything and focuses on relative modality contributions and cooperation rather than absolute retrieval performance.

    Source: §8 · p. 13

Abstract

from arXiv · show

Multimodal knowledge graphs (KGs) integrate information from text, figures, tables, and other modalities into a unified structured representation, with the promise that richer evidence enables better inference. In GraphRAG systems built over such graphs, it is commonly assumed that retrieving evidence from more modalities at inference time improves downstream performance. Yet, redundant or overlapping multimodal evidence may distract language models in question answering (QA), and whether each modality contributes equally across questions, models, and tasks remains poorly understood. In this work, we study how modality-aware retrieval affects downstream inference in a multimodal GraphRAG pipeline, using document visual question answering (DocVQA) as a testbed. We extend an existing KG-based QA framework to be modality-aware, leveraging the graph structure to track which modality supports which facts and to selectively filter evidence at the edge level. This enables us to investigate whether providing all available multimodal evidence at inference time benefits QA, and to evaluate the contribution and cooperation of modalities across question, task, and model characteristics. Through a controlled analysis within a state-of-the-art multimodal GraphRAG pipeline, five multimodal LLMs and two DocVQA benchmarks, we find that tables and text provide the strongest contributions, and that combining modalities frequently produces redundancy rather than synergy, particularly for pairs involving textual information. Positive cooperation appears mainly between non-text modalities and depends on question intent and task type. Our findings argue for selective, modality-aware retrieval in the design of more effective GraphRAG systems, where modalities are filtered according to the downstream task rather than retrieved uniformly.

1. Introduction

Multimodal GraphRAG commonly assumes that adding modalities improves performance, but how modalities contribute and interact across questions, models, and tasks remains underexplored. This work traces modality provenance in a GraphRAG system and evaluates contributions and cooperation across DocVQA benchmarks and models.

1. Introduction

  • Whether modalities provide complementary or redundant evidence across models, domains, question intents, and reasoning tasks remains underexplored beyond modality ablations.Source: §1 · p. 1
  • Because question type may change which evidence is useful, graph-linked modality provenance can help analyze when additional modalities benefit QA and when they offer little or negative value.Source: §1 · p. 1
  • The study extends multimodal GraphRAG to trace each knowledge-graph triple to its source modality and evaluates modality contributions and cooperation across two DocVQA benchmarks and five multimodal LLMs.Source: §1 · p. 1

2. Related Work

Prior work studies multimodal retrieval, modality reliability and dominance, and ways to quantify modality importance. The paper uses SHAPE to distinguish individual contributions from cross-modal cooperation, addressing a limitation of ablation-based analyses.

2.1. Modality contributions in multimodal RAG.

  • Multimodal RAG studies report benefits from heterogeneous evidence, while CMRAG finds that naive textual-visual combinations can perform worse than a single modality and GraphRAG studies report synergies.Source: §2.1 · p. 2

2.2. Modality robustness in multimodal learning.

  • Research on multimodal learning examines reliability, modality dominance, and sample-level variation, including cases where a second modality helps only when it provides useful answer information.Source: §2.2 · p. 2

2.3. Metrics for modality contribution analysis.

  • SHAPE uses Shapley values to quantify both marginal modality contribution and cross-modal cooperation, separating these effects beyond standard ablations.Source: §2.3 · pp. 2–3
  • Ablations combine a modality’s usefulness with its interaction with remaining modalities, whereas SHAPE evaluates modality combinations to distinguish contribution from cooperation.Source: §2.3 · pp. 2–3

3. Task Definition

The task is document question answering over a multimodal knowledge graph, with retrieval restricted to a selected subset of modalities. Each document graph represents content elements and their relations, and each fact is linked to the modalities from which it was extracted.

3. Task Definition

  • Each document is represented as a multimodal knowledge graph whose nodes correspond to document-content elements and whose edges represent relations.Source: §3 · p. 3
  • Modality-aware question answering retrieves and infers over a document graph restricted to a selected modality subset, with each fact associated with the modalities of its source content.Source: §3 · p. 3

4. Modality-Aware Framework

The framework adds modality provenance to an existing knowledge-graph QA system and uses that provenance to constrain retrieval. For each selected modality subset, it retains supported graph edges and corresponding evidence chunks before prompting an LLM to answer.

4. Modality-Aware Framework

  • The framework extends an existing state-of-the-art KG-based QA system with modality-aware provenance tracking and retrieval constraints.Source: §4 · p. 3
  • For each graph edge, the framework records the document chunks from which it was extracted and their modality labels, enabling edge filtering by supporting modality.Source: §4 · p. 3
  • For a selected modality subset, retrieval retains edges with at least one matching support modality and uses only evidence chunks from the selected modalities to form the LLM context.Source: §4 · p. 3

5. Experimental Setup

The experimental setup extends a graph-based multimodal QA pipeline with modality-aware retrieval, then evaluates it across five answer-generation models and two DocVQA benchmarks. The analysis retains questions supported by exactly two modalities so their contributions and interactions can be compared.

5.1. GraphRAG QA System

  • The study uses RAG-Anything, recording modality provenance on graph edges so retrieval can be restricted to selected modalities before relevant chunks are ranked and passed to an MLLM.The framework is described as model-agnostic.Source: §5.1 · p. 3

5.2. LLMs for Answer Generation

  • Five answer-generation models span GPT-4o-mini, two Qwen3-VL sizes, and two Gemma 3 sizes, covering proprietary and open-weight models.The selected model sizes are 8B and 30B for Qwen3-VL and 4B and 27B for Gemma 3.Source: §5.2 · p. 3

5.3. DocVQA Benchmarks

  • The benchmarks are MMLongBench-Doc and LongDocURL, covering multipage documents with evidence in text, figures, and tables; together they contain 327 documents.Their gold-standard modality annotations support evaluation with restricted evidence subsets.Source: §5.3 · p. 4
  • The analysis retains 1,051 questions with exactly two gold modalities and evaluates all three nonempty modality subsets across five models, yielding 15,765 QA runs.Single-modality questions are excluded because they do not permit measuring modality interaction; three-modality questions are too sparse for reliable analysis.Source: §5.3 · p. 4

5.4. Question Intent Taxonomy

The study categorizes benchmark questions by intent using a web-search taxonomy as a proxy, then applies zeroshot classification and consensus review to validate labels.

5.4. Question Intent Taxonomy

  • The analysis adopts a web-search question-intent taxonomy as a proxy because both web search and DocVQA questions seek information grounded in external evidence.The taxonomy classifies intent from question wording and what the question aims to ask.Source: §5.4 · p. 4
  • GPT-5 classifies questions zero-shot, and labels change only when GPT-5.5 and Claude Opus 4.7 agree on an alternative taxonomy label.The judges had Cohen’s kappa of 0.87 and raw agreement of 90.1%; their consensus revisions changed 66 labels.Source: §5.4 · p. 4

5.5. Evaluation Metrics

The evaluation combines benchmark-defined answer accuracy with SHAPE measures of modality contributions and cooperation. Bootstrap resampling is used to quantify uncertainty in aggregate comparisons.

5.5.1. Answer Accuracy

  • Answer generation is first scored per question using each benchmark’s defined accuracy metric.The resulting per-question score is denoted v.Source: §5.5.1 · p. 4

5.5.2. SHAPE Metric

  • SHAPE [14] measures each modality’s marginal contribution across modality combinations and the cooperation between modalities.A positive cooperation score indicates complementary behavior, while a negative score indicates redundancy or interference.Source: §5.5.2 · pp. 5–6
  • The pairwise contribution difference D is S_m2 minus S_m1, so its sign indicates which modality contributes more.The no-modality baseline is estimated with a majority-class predictor using the most frequent gold answer within each modality group.Source: §5.5.2 · p. 6

5.5.3. Bootstrap Sampling Schemes

  • The main bootstrap scheme preserves each benchmark’s modality-group distribution, and reported intervals are percentile-based 95% confidence intervals from 100,000 samples.Additional schemes produced largely consistent intervals, indicating robustness to the sampling strategy.Source: §5.5.3 · p. 6

5.6. Experimental Analyses

The analyses examine overall performance, compare modality contribution and cooperation across models, and assess how those scores vary by question intent and process type.

5.6. Experimental Analyses

  • Overall performance scores across questions, benchmarks, and models provide context for interpreting SHAPE scores.Table 4 reports total accuracy and accuracy by modality group across benchmarks.Source: §5.6 · p. 6, Table 4 · p. 6
  • Model comparisons assess whether modality contribution and cooperation scores differ across LLMs, using bootstrap confidence intervals for aggregate comparisons..Source: §5.6 · p. 6
  • Question-type analyses compare contribution and cooperation scores by question intent and by the process required to answer..Source: §5.6 · p. 6

6. Results

The analysis uses per-question scores to compare modality contributions and cooperation overall and across question intent and task requirements. Absolute accuracy differences between benchmarks are outside its scope, and task-type analysis uses LongDocURL questions.

6.1.1. Descriptive Overview

  • Table and Text provide the strongest marginal contributions, with Table contributing most; Image-Text and Text-Table show negative cooperation, while pairs involving Table can cooperate positively across models.Layout contributes little, and Image contributions vary more by model; Image-Table and Layout-Table have the strongest positive cooperation in the macro-averaged overview.Source: §6.1.1 · p. 7, Figure 3 · p. 7

6.1.2. Bootstrap Analysis

  • Layout contributes consistently less than other modalities; Table contributes more than Text, while Text edges Image in Image-Text and Table also exceeds Image.The Table-Text gap has stronger cross-model agreement than the smaller Text-Image and Table-Image gaps.Source: §6.1.2 · p. 7, Figure 4 · p. 8
  • Image-Text and Text-Table show negative cooperation across models, while Image-Layout and Layout-Table show little interaction and Image-Table shows slight cooperation, mainly driven by GPT-4o-mini.The negative effects have moderate cross-model agreement (mean pairwise Spearman correlation 0.372); evidence of complementary interactions is otherwise limited.Source: §6.1.2 · p. 7, Figure 4 · p. 8

6.2.1. Question Intent

  • Text contributes more than Layout across analysed intents; Quantity questions show near-equal contributions in Text-Image and Table-Text, while List and Entity patterns differ by modality pair.For Entity questions, Text and Table are generally strongest, with little difference between them in Table-Text; List results vary substantially across some pairs and models.Source: §6.2.1 · p. 8
  • For Quantity questions, non-text pairs show positive cooperation while text-involving pairs are negative or near zero; Text-involving pairs also tend toward negative cooperation for List and Entity questions.For List and Entity questions, Text can already provide sufficient information, limiting the benefit of combining it with other modalities.Source: §6.2.1 · pp. 8–9, Figure 8 · p. 10

6.2.2. Question Task Requirement

  • For Understanding, Text contributes more in Text-Image and Text-Layout pairs, while Table dominates Table-Text; for Locating, Text and Table contribute most, and for Reasoning, Text generally leads in Table-Text and Text-Layout.The analysis uses LongDocURL task labels and includes only task-type groups with at least 10 questions.Source: §6.2.2 · p. 9
  • Understanding questions show lower cooperation than Reasoning for Image-Layout, Image-Text, and Text-Table; Locating questions show positive Image-Table cooperation.For Locating, Table usually contributes more, while the image helps identify the relevant information's location on the page.Source: §6.2.2 · pp. 9–10, Figure 8 · p. 10

7. Discussion

The discussion finds that modality value varies with the evidence retrieved, question intent, task type, and inference model. It recommends modality-aware retrieval that prioritizes useful evidence while accounting for complementarity and redundancy.

7.1. Observations

  • Cooperation is negative for pairs involving Text and mostly limited among non-text pairs; Image-Table positivity is mainly driven by GPT-4o-mini.Image-Layout and Layout-Table are close to zero, while every negative cross-model mean involves Text.Source: §7.1 · pp. 10–11
  • Cooperation varies with question intent and task type: non-text pairs are positive for Quantity questions, and Image-Table is positive for Locating questions.Text-containing pairs remain at or below zero for Quantity questions.Source: §7.1 · p. 11, Figure 11 · p. 12

7.2. Design Recommendations

  • Modality usefulness is not intrinsic: the value of retrieved evidence depends on the other modalities, the question, and the inference model.This variation motivates selective retrieval for DocVQA.Source: §7.2 · p. 11
  • Measuring both individual contribution and cooperation distinguishes complementary evidence from overlapping evidence that can raise retrieval cost without adding information.The analysis also identifies factors associated with redundancy, informing when multimodal retrieval may be beneficial.Source: §7.2 · pp. 11–12
  • GraphRAG can support modality-aware retrieval by linking each fact to its source modality and selectively prioritizing or suppressing evidence.The authors report integrating this linking into RAG-Anything without modifying other pipeline components and with modest overhead.Source: §7.2 · pp. 12–13
  • For DocVQA, Text and Table are usually the safest retrieval priorities, while Layout is the weakest contributor and should be deprioritized.Text should not be expected to cooperate strongly with other modalities; Image-Table shows the most consistent positive cooperation, particularly for Quantity and Locating questions.Source: §7.2 · p. 13

8. Scope and Generalisability

The analysis focuses on relative modality contributions and cooperation within RAG-Anything, not absolute retrieval performance or generalization across frameworks. Its retrieval design aims to isolate modality effects, while LLM-based question-intent labels remain a source of uncertainty, especially for smaller groups.

8. Scope and Generalisability

  • Results are scoped to RAG-Anything; generalization across frameworks and absolute retrieval performance are outside the analysis.The authors note that retrieval performance is sensitive to framework variations.Source: §8 · p. 13
  • Modality-specific chunks are filtered before cosine-similarity ranking, with the same procedure and top-20 retrieval used across configurations.For LongDocURL, the largest context window supported by each model is used to minimize truncation.Source: §8 · p. 13
  • At least one judge suggested a different intent label for 170 of 1,051 questions (16.2%), and mislabels may shift SHAPE scores in smaller intent groups.For 97 questions (9.2%), only one judge disagreed with the original label.Source: §8 · p. 13

9. Conclusion

Across five multimodal LLMs and two long-context DocVQA benchmarks, the study finds that text and tables often provide the strongest evidence, while adding modalities can introduce redundancy. Modality interactions vary with question intent and task type, supporting selective rather than uniform retrieval.

9. Conclusion

  • Tables and text often provide the strongest evidence, while additional modalities do not consistently improve performance and can introduce redundancy.The analysis covers five multimodal LLMs, two long-context DocVQA benchmarks, and a state-of-the-art multimodal GraphRAG pipeline.Source: §9 · p. 13
  • Modality interactions depend strongly on question intent and task type, with complementary effects appearing when modalities provide distinct forms of evidence.Source: §9 · p. 13
  • The findings support filtering modalities according to the downstream task rather than retrieving them uniformly.Source: §9 · p. 13
Loading 2609.35304v1…