PaperlayerView exact revision on arXiv

Source-linked AI summary

TopoGraphRAG-Bench: Evaluating Multimodal GraphRAG on Layout-Grounded Evidence Reasoning

Ruochi Li, Jianzhe Lin, Haoxuan Zhang, Haihua Chen, Junhua Ding, Edward Gehringer, Yang Zhang

arXiv:2610.09360v1cs.AIcs.CL

TL;DR

Complex document questions require recovering dependencies among text, figure, and table evidence, yet existing GraphRAG evaluations are largely text-centered and multimodal document benchmarks do not diagnose intended evidence-topology recovery. TOPOGRAPHRAG-BENCH constructs and counterfactually validates topology-grounded questions, and evaluation finds multimodal GraphRAG strongest overall while exposing continuing alignment and composition challenges.

  • Problem

    Existing GraphRAG evaluations are largely text-centered, while multimodal document RAG benchmarks do not directly assess recovery of intended evidence topology across layouts.

    Source: §1 · p. 2

  • Method

    The benchmark builds layout-grounded questions bottom-up across single-hop, bridge-chain, and multi-source synthesis topologies, then applies counterfactual validation for shortcut, modality, and evidence necessity.

    Source: §1 · p. 2

  • Results

    Multimodal GraphRAG systems perform best overall, but remain limited by modality-alignment failures and incomplete synthesis across heterogeneous evidence.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The benchmark tests whether systems recover and use intended evidence structure rather than rely on shortcut retrieval or single-source evidence.

    Source: §6 · p. 10

    The benchmark samples MMDocIR, so its document domains, layout styles, and evidence distributions are bounded by that source corpus.

    Source: D Limitations · p. 20

Abstract

from arXiv · show

Real-world documents distribute evidence across text, tables, figures, and captions within complex page layouts. Answering complex questions over such documents therefore requires more than retrieving relevant passages: systems must recover the evidence topology that connects heterogeneous evidence units. Existing GraphRAG evaluations remain largely text-centered, while multimodal document RAG benchmarks assess cross-modal retrieval and generation without directly evaluating recovery of the intended evidence topology. We introduce TOPOGRAPHRAG-BENCH, a layout-grounded benchmark for multimodal evidence reasoning in GraphRAG, comprising 2,024 questions over 201 long, visually rich documents. Questions are constructed bottom-up from text, figure, and table evidence units under three controlled topologies: single-hop retrieval, bridge-chain reasoning, and multi-source synthesis. To ensure that questions preserve their intended structure, we apply counterfactual validation for shortcut resistance, modality necessity, and evidence necessity. We evaluate text-only GraphRAG, page-level visual retrieval, and multimodal GraphRAG systems using retrieval, generation, and topology-aware reasoning metrics. Multimodal GraphRAG systems achieve the strongest overall performance, but still fail when visual-textual evidence alignment or multi-unit composition is incomplete. Text-only GraphRAG struggles when key dependencies are grounded in figures or tables, while page-level visual retrieval lacks the fine-grained structure needed for topology recovery. These findings motivate GraphRAG systems that move beyond text-derived entity relation graphs to explicitly model document layouts, cross-modal evidence alignment, and the reasoning roles of evidence units. Code and data are available at https://richardlrc.github.io/TopoGraphRAG-Bench/.

1 Introduction

Complex document questions require recovering how evidence units depend on one another, not merely retrieving isolated passages. TOPOGRAPHRAG-BENCH evaluates that structure recovery across text, figures, and tables, finding multimodal GraphRAG strongest overall while exposing limitations across all evaluated paradigms.

1 Introduction

  • Complex document retrieval must recover dependencies among evidence units, because questions may require an intermediate bridge or several units jointly supporting a conclusion.The target is evidence topology, not just isolated relevant evidence.Source: §1 · p. 2
  • Existing GraphRAG benchmarks are largely text-centered, while multimodal document RAG benchmarks do not diagnose recovery of intended evidence topology across layouts.This leaves open whether graph-structured retrieval can reason over layout-grounded multimodal evidence.Source: §1 · p. 2
  • TOPOGRAPHRAG-BENCH builds questions bottom-up from layout-grounded units, covering single-hop, bridge-chain, and multi-source synthesis with counterfactual topology validation.It contains 2,024 questions across 201 documents, including 773 cross-modal instances.Source: §1 · p. 2
  • Multimodal GraphRAG performs best overall, while text-only GraphRAG misses visual and tabular dependencies and page-level retrieval struggles with fine-grained topology.Multimodal systems remain limited by modality-alignment failures and incomplete synthesis across heterogeneous evidence.Source: §1 · p. 2

2 Related Work

Prior work studies graph-based retrieval, multimodal document retrieval, and multi-evidence reasoning, but multimodal evidence-topology recovery in GraphRAG remains underexplored. This benchmark connects layout-grounded evidence structure to separate evaluation of retrieval and composition.

GraphRAG methods and benchmarks

  • GraphRAG methods model chunks, entities, relations, communities, or summaries, but existing GraphRAG benchmarks remain largely text-centered.These evaluations cover retrieval and generation tasks but leave multimodal document evidence structures underexplored.Source: §2 · p. 3

Multimodal document RAG benchmarks

  • Multimodal document RAG benchmarks evaluate retrieval and generation across pages, layouts, figures, and tables, but are not designed to diagnose graph-based evidence-topology recovery.They include settings for cross-modal evidence chains, retrieval paradigms, realistic queries, and document retrieval.Source: §2 · p. 3

Multi-evidence reasoning and question construction

  • Multi-hop and multimodal QA benchmarks establish the need to connect multiple evidence sources across text, tables, images, charts, and long documents.The cited foundations include HotpotQA, 2WikiMultiHopQA, MuSiQue, MultiHop RAG, MultiModalQA, CHARGE, BRIDGE, and DocHop QA [27] [25] [16].Source: §2 · p. 3
  • This benchmark constructs questions from layout-grounded units while retaining page, layout, modality, and reasoning roles, enabling separate evaluation of evidence retrieval and composition.It covers entity-mediated bridge chains and synthesis over 3–6 complementary evidence points, with topology-specific evidence-necessity checks.Source: §2 · p. 3

3 Benchmark

TOPOGRAPHRAG-BENCH constructs questions from layout-grounded evidence units and validates that each requires its intended topology, modality, and evidence. The resulting strict benchmark contains 2,024 questions across 201 documents.

3.1 Construction

  • The benchmark samples 201 MMDocIR documents; text units retain original text, while figure and table units preserve images and add OCR and VLM descriptions.Each unit retains document, page, layout, and modality identifiers plus a semantic representation.Source: §3.1 · p. 3
  • The 201 documents cover nine of MMDocIR’s ten application domains, and 773 questions are cross-modal, with 1,353 requiring figures or tables.The evidence distribution also reports 1,596 cross-page questions.Source: §3.1 · pp. 3, 5
  • Questions are built over explicit evidence topologies rather than prompting over whole documents: isolated units support direct retrieval, linked units form bridge chains, and complementary sets support synthesis.Recurring normalized entities serve as anchors for linking evidence units across pages and modalities.Source: §3.1 · pp. 3–4
  • Single-hop questions use one text, figure, or table unit, and their answer must be recoverable without consulting another document layout.The gold evidence set is restricted to the source unit.Source: §3.1 · p. 4
  • Bridge-chain questions compose two atomic QA pairs through a shared entity, whose indirect mention links the first-hop evidence to a target attribute, value, trend, or relation.The bridge entity is omitted from the composed question’s surface wording.Source: §3.1 · pp. 4–5
  • Synthesis questions aggregate distinct facts from multiple evidence units into higher-level patterns and link each evidence point to its layout and modality.Entities must occur in at least five units, and retained cases have at least three distinct evidence facts.Source: §3.1 · p. 5

3.2 Quality Assurance

  • Counterfactual checks remove bridge-chain questions answerable from either hop alone and cross-modal questions answerable from text-only context.Synthesis questions undergo leave-one-out checks and are retained only when at least three evidence points are required.Source: §3.2 · p. 5
  • After automatic validation and manual quality assurance, the strict benchmark contains 2,024 questions across 201 documents, with about 15% single-hop, 50% bridge-chain, and 35% synthesis questions.Reviewers audited 303 questions and achieved Cohen’s κ = 0.84.Source: §3.2 · pp. 5–6

4 Task Definition

The benchmark treats document QA as recovery and use of structured evidence: one unit for single-hop questions, an ordered dependency path for bridge chains, and joint evidence for synthesis. It evaluates both retrieval coverage and answer generation against those structures.

4 Task Definition

  • Questions provide a gold answer, supporting evidence units, and a reasoning type; required evidence may be one unit, an ordered bridge path, or multiple units jointly supporting a conclusion.For synthesis, no single source is sufficient to recover the full answer.Source: §4 · p. 6

4.1 Retrieval Task and Metrics

  • Retrieval is evaluated as evidence-structure recovery, requiring coverage of the source unit, bridge and target evidence, or required synthesis points as appropriate.Topology-enriched context precision and recall references include bridge entities and traces or required synthesis evidence points.Source: §4.1 · p. 6

4.2 Generation Task and Metrics

  • Generation must produce a correct, grounded answer while covering the reasoning steps required by the question’s structure.A correct final answer can still omit a bridge hop or rely on only one source for synthesis; evaluation includes answer accuracy, faithfulness, response relevancy, and step coverage.Source: §4.2 · pp. 6–7
  • Table 2 reports overall retrieval and generation performance on 2,024 questions, with step coverage restricted to bridge-chain and synthesis questions.VisRAG is omitted from text-based retrieval evaluation, and its faithfulness is not reported for page-level visual context.Source: Table 2 · p. 7

5 Experiments

The experiments compare text-only GraphRAG, page-level visual retrieval, and multimodal GraphRAG across aggregate performance and three evidence topologies. Multimodal GraphRAG leads overall, but performance depends on recovering and composing heterogeneous evidence along the required paths.

5.1 Experimental Setup

  • The evaluation compares three evidence representations: text-derived graphs, page-image retrieval, and multimodal graph structures, using fixed backends within each system family.It reports aggregate retrieval and generation results, then breaks performance down by topology, ordered bridge path, and synthesis composition.Source: §5.1 · p. 7

5.2 Main Results

  • Multimodal GraphRAG systems achieve the strongest overall generation results: MegaRAG leads generation estimates, while RAG-Anything leads retrieval and has comparable answer accuracy.LightRAG is the strongest text-only baseline, but remains below multimodal GraphRAG systems on all generation metrics; retrieval and generation rankings differ.Source: §5.2 · p. 7
  • The reported gap between retrieval and generation rankings suggests that success depends not only on retrieving evidence, but also on aligning and composing it according to the intended topology.Page-level visual retrieval provides direct image access but remains substantially below multimodal GraphRAG in answer accuracy and step coverage.Source: §5.2 · pp. 7–8

5.3 Topology and Modality

  • On multimodal subsets, HippoRAG falls from 0.90 accuracy on single-hop text to 0.17 on Bridge-MM, while MegaRAG reaches 0.53 on Bridge-MM and 0.43 on Synthesis-MM.VisRAG reaches 0.65 on figure questions and 0.56 on table questions, but only 0.26 on Bridge-MM and 0.25 on Synthesis-MM.Source: Figure 3 · p. 8, §5.3 · p. 8

5.4 Bridge Paths

  • On ordered bridge paths, LightRAG reaches 57.6% on text→text but 12.0% on figure→table and 7.5% on table→figure; MegaRAG reaches 65.0% on table→figure.Multimodal systems are more robust across paths, though performance still varies by ordered modality sequence.Source: §5.4 · pp. 8–9, Figure 4 · p. 9
  • On visual-only bridge paths, MegaRAG achieves 83.0% step coverage, versus 44.9% for LightRAG and 49.4% for VisRAG.Text-only systems’ lower coverage indicates difficulty recovering or expressing required hop structure when bridges depend on figures or tables.Source: Figure 4 · p. 9, §5.4 · p. 9

5.5 Multi-Source Synthesis

  • Text-only synthesis is easier than cross-modal synthesis: LightRAG falls from 46.6% accuracy to 27.4%, and HippoRAG from 44.4% to 18.4%.Multimodal systems are more robust across compositions, yet both still trail their text-only synthesis performance.Source: Table 3 · p. 9, §5.5 · pp. 9–10

6 Conclusion

The conclusion presents TOPOGRAPHRAG-BENCH as an evaluation of evidence-topology recovery across layout-rich documents and identifies alignment and composition as continuing challenges. The appendix outlines benchmark limitations and reports configuration and efficiency information, including distinct cost profiles across systems.

6 Conclusion

  • The benchmark covers 2,024 QA pairs across 201 layout-rich documents and three topologies; multimodal GraphRAG performs best overall, but remains challenged by evidence alignment and composition.Text-only GraphRAG is limited when dependencies are grounded outside text, while page-level retrieval does not reliably recover fine-grained topology.Source: §6 · p. 10

Appendix Overview

  • The appendix lists further material on configurations and efficiency, evaluation metrics, experimental results, benchmark limitations, and data use and licensing.The stated limitation areas include benchmark scope, topology coverage, evaluated systems, and indexing cost.Source: Appendix Overview · p. 13

A Baseline Configuration and Efficiency

  • Systems use their official or framework-default indexing pipelines unless specified otherwise, and are indexed independently at the document level.Runtime, token usage, and graph statistics are reported per document using the 201-document benchmark denominator.Source: A Baseline Configuration and Efficiency · p. 13
  • Multimodal GraphRAG incurs additional indexing and token costs; RAG-Anything has the largest graph and highest token usage, while MegaRAG has higher retrieval cost.VisRAG has low indexing and retrieval cost but does not expose fine-grained graph statistics.Source: A Baseline Configuration and Efficiency · p. 13

B.1 Metric Definitions and Judging Protocol

The protocol evaluates retrieval, answer quality, and coverage of topology-specific reasoning content against references that encode both the gold answer and the required evidence structure.

B.1 Metric Definitions and Judging Protocol

  • Each question supplies a gold answer, gold evidence units, and a reasoning topology; systems return ranked contexts and a generated answer.The evaluation also uses each system’s native indexing and graph construction while standardizing document input, model backend, and retrieval settings.Source: §B.1 · p. 13, Table 4 · p. 14
  • The topology-enriched reference adds the ordered bridge trace or required synthesis evidence points to the answer-based reference.For bridge questions, these include the bridge entity and ordered trace; synthesis references include required evidence points.Source: §B.1 · p. 14
  • Context precision measures whether relevant contexts rank ahead of irrelevant ones, while context recall checks retrieved support for reference claims.Recall includes bridge and target evidence for bridge questions, and required evidence points plus the conclusion for synthesis questions.Source: §B.1 · pp. 14–15
  • Answer accuracy scores agreement with the gold answer, allowing partial credit when a multi-evidence response captures only part of the conclusion.Single-hop answers are judged primarily by semantic equivalence; bridge and synthesis answers are judged against the full gold conclusion.Source: §B.1 · p. 15
  • Faithfulness checks whether generated claims are supported by retrieved contexts, whereas response relevancy measures whether the answer content reflects the original question.Faithfulness is grounding in retrieved evidence rather than agreement with the gold answer; relevancy uses question variants generated from the response and embedding similarity.Source: §B.1 · p. 15
  • Step coverage evaluates required bridge hops or synthesis evidence points, complementing answer accuracy because correct answers can omit intermediate results.It is reported only for bridge-chain and synthesis questions; VisRAG’s page-level visual contexts do not support directly comparable text-based retrieval and faithfulness scores.Source: §B.1 · pp. 15–16

B.2 Reliability of Step Coverage Evaluation

Step-coverage rankings persist across model judges, and hop-level judgments show substantial agreement across judges and with a human spot-check.

C Additional Experimental Results

  • The reliability analysis covers 1,722 multi-step questions and excludes 302 single-hop questions from step-coverage evaluation.The appendix presents these analyses as additional fine-grained results complementing the main system-family comparison.Source: §B.2 · p. 16, C Additional Experimental Results · p. 17

B.2 Reliability of Step Coverage Evaluation

  • MegaRAG and RAG-Anything consistently rank first and second in mean step coverage across three model judges, ahead of text-only GraphRAG and page-level visual retrieval.The evaluator-family comparison uses fixed responses from all six systems.Source: §B.2 · p. 16, Table 6 · p. 16
  • Overall hop-level agreement ranges from 0.8512 to 0.8958, with Cohen’s κ from 0.7054 to 0.7923.Across topologies, agreement ranges from 0.8479 to 0.9028 and κ from 0.7002 to 0.8039.Source: §B.2 · p. 16, Table 7 · p. 17
  • On 100 manually annotated evaluation units, judge-human agreement ranges from 0.83 to 0.91, with Cohen’s κ from 0.6561 to 0.8190.The sample comprises 50 bridge-chain hops and 50 synthesis evidence points across all six systems.Source: §B.2 · p. 17, Table 8 · p. 17

C.1 Overall Metric Uncertainty

Retrieval leaders vary by reasoning topology, and the reported uncertainty intervals for overall metrics use per-question scores with step coverage limited to multi-step questions.

C.1 Overall Metric Uncertainty

  • Table 9 reports approximate 95% confidence intervals as mean ± 1.96 standard errors over non-missing per-question scores.Step-coverage intervals use 1,722 bridge-chain and synthesis questions, excluding the single-hop subset.Source: §C.1 · p. 17, Table 9 · p. 17

C.2 Retrieval Performance by Reasoning Topology

  • RAG-Anything has the strongest retrieval scores on bridge-chain and synthesis questions, while MegaRAG leads on single-hop questions.The retrieval-versus-generation contrast indicates that retrieval coverage alone does not determine final answer quality; evidence must also be composed according to the intended topology.Source: §C.2 · p. 17, Table 10 · p. 18

C.3 Complete Bridge-Path Results

The appendix details modality- and pattern-dependent reasoning difficulty and finds that multimodal GraphRAG’s advantage over the evaluated alternatives also appears on a separate corpus extension.

C.3 Complete Bridge-Path Results

  • Bridge-path answer accuracy varies by ordered modality path: figure→text and table→text are easier than figure→table or table→figure.Step coverage more directly reflects recovery of both bridge and target evidence roles than final-answer accuracy.Source: §C.3 · p. 18, Table 11 · p. 18, Table 12 · p. 18

C.4 Synthesis Pattern Breakdown

  • Synthesis difficulty depends on both evidence modality and relation type; comparison and contradiction patterns are often challenging, while explicit trends and causal patterns can be easier.The synthesis breakdown reports answer accuracy by synthesis pattern.Source: §C.4 · p. 18, Table 13 · p. 19

C.5 Cross-Corpus Generalization

  • On the 473-question UniDoc-Bench extension, RAG-Anything leads LightRAG and VisRAG in answer accuracy and step coverage.Answer accuracy is 0.5842 versus 0.5356 and 0.3800, respectively; step coverage is 0.7632 versus 0.5716 and 0.5021. The extension uses 60 financial and legal documents and preserves the main benchmark’s topology-specific validation criteria.Source: §C.5 · pp. 19–20, Table 15 · p. 20

D Limitations

The benchmark is bounded by its source corpus, three controlled evidence topologies, and the evaluated system and configuration choices; the authors also note substantial offline indexing costs.

D Limitations

  • The sampled MMDocIR corpus bounds the benchmark's document domains, layout styles, and evidence distributions.The authors identify broader document collections as a direction for future work.Source: D Limitations · p. 20
  • The benchmark covers only single-hop retrieval, bridge-chain reasoning, and multi-source synthesis, leaving other reasoning forms and longer dependency chains outside its scope.The authors suggest expanding the topology space to include more diverse and realistic patterns.Source: D Limitations · p. 20
  • The evaluated paradigms and controlled generation and embedding configurations do not cover all system designs or backbone sensitivity; offline graph construction can also incur substantial indexing costs.The authors propose studying additional designs, quality-cost tradeoffs, and incremental indexing for large-scale deployment.Source: D Limitations · p. 20

E License and Data Use

The benchmark uses MMDocIR source materials subject to that dataset's license, while its new annotations are released for academic and noncommercial research use.

E License and Data Use

  • The benchmark draws on 201 MMDocIR documents and associated source materials, which remain subject to the upstream dataset's license and usage terms.The authors do not claim ownership of the original documents, page images, OCR outputs, layout units, or MMDocIR annotations.Source: E License and Data Use · p. 20
  • The new questions, answers, topology labels, and related annotations are released for academic and noncommercial research use.Users must comply with upstream terms and cite both MMDocIR and this work when using or redistributing the benchmark.Source: E License and Data Use · p. 20

Related papers

Loading 2610.09360v1…