PaperlayerView exact revision on arXiv

Source-linked AI summary

When to use Graphs in RAG: A Comprehensive Analysis for Graph Retrieval-Augmented Generation

Zhishang Xiang, Chuanjie Wu, Qinggang Zhang, Shengyuan Chen, Zijin Hong, Xiao Huang, Jinsong Su

arXiv:2506.05690v3cs.CL

TL;DR

Existing benchmarks provide limited assessment of when GraphRAG benefits RAG, despite reported real-world underperformance and task-specific strengths. GraphRAG-Bench evaluates progressively difficult tasks across varied corpora and pipeline stages; reported comparisons distinguish RAG's strength on discrete facts from a creative-generation faithfulness result for RAPTOR.

  • Problem

    Existing benchmarks inadequately assess when graph structures benefit RAG because their tasks and corpora do not sufficiently represent reasoning complexity and domain hierarchies.

    Source: §1 · p. 2

  • Method

    GraphRAG-Bench combines varied-information-density corpora and increasingly difficult tasks with evaluation spanning graph construction, retrieval, and generation.

    Source: §1 · p. 3, §3 · p. 5

  • Results

    Across tasks, RAG excels at discrete-fact retrieval, while RAPTOR has highest faithfulness on creative generation and RAG covers more evidence.

    Source: §4.2 · p. 8, §4.1 · p. 8

  • Takeaways & Limitations

    The benchmark is intended to investigate when GraphRAG surpasses traditional RAG and offer guidelines for practical application.

    Source: §1 · p. 3

    The benchmark is limited to unimodal text contexts and omits multimodal data integration.

    Source: J LIMITATION · p. 33

Abstract

from arXiv · show

Graph retrieval-augmented generation (GraphRAG) has emerged as a powerful paradigm for enhancing large language models (LLMs) with external knowledge. It leverages graphs to model the hierarchical structure between specific concepts, enabling more coherent and effective knowledge retrieval for accurate reasoning.Despite its conceptual promise, recent studies report that GraphRAG frequently underperforms vanilla RAG on many real-world tasks. This raises a critical question: Is GraphRAG really effective, and in which scenarios do graph structures provide measurable benefits for RAG systems? To address this, we propose GraphRAG-Bench, a comprehensive benchmark designed to evaluate GraphRAG models onboth hierarchical knowledge retrieval and deep contextual reasoning. GraphRAG-Bench features a comprehensive dataset with tasks of increasing difficulty, coveringfact retrieval, complex reasoning, contextual summarization, and creative generation, and a systematic evaluation across the entire pipeline, from graph constructionand knowledge retrieval to final generation. Leveraging this novel benchmark, we systematically investigate the conditions when GraphRAG surpasses traditional RAG and the underlying reasons for its success, offering guidelines for its practical application. All related resources and analyses are collected for the community at https://github.com/GraphRAG-Bench/GraphRAG-Benchmark.

1 INTRODUCTION

RAG can lose contextual relationships when chunking complex, dispersed domain knowledge, while GraphRAG uses graph structure to retrieve connected information. Yet reported underperformance and benchmark gaps leave unclear when graphs provide measurable benefits; GraphRAG-Bench is proposed to investigate those conditions.

1 INTRODUCTION

  • Chunking unstructured corpora can sacrifice contextual information and compromise retrieval accuracy and contextual comprehension for complex reasoning.Documents may vary in accuracy and completeness, while domain knowledge is scattered across sources without clear hierarchical relationships.Source: §1 · pp. 1–2
  • Prior studies report GraphRAG underperforming traditional RAG on real-world tasks, raising questions about when graph structures offer measurable benefits.Reported comparisons include lower Natural Question accuracy and higher latency, alongside improved reasoning depth on HotpotQA multi-hop questions.Source: §1 · p. 2
  • Existing benchmarks inadequately assess graph structures because they underrepresent reasoning complexity and domain corpora with explicit hierarchies and varying information density.They often focus on retrieval or linear multi-hop questions rather than contextual synthesis.Source: §1 · p. 2
  • GraphRAG structures entities and their connections as a graph, retrieving interconnected subgraphs to support deeper contextual analysis than direct semantic retrieval.The paper contrasts RAG's strength in rapid access to discrete information with GraphRAG's emphasis on tasks involving interconnected data.Source: §1 · p. 2, Figure 1 · p. 3
  • GraphRAG-Bench evaluates tasks of increasing difficulty and varied corpus information density, with assessment spanning graph construction, retrieval, and generation.Its tasks cover fact retrieval, multi-hop reasoning, contextual summarization, and creative generation.Source: §1 · p. 3

2 PRELIMINARY STUDY

The preliminary study contrasts RAG's direct semantic retrieval with GraphRAG's graph traversal, then identifies shortcomings in existing benchmarks' task design, corpora, and pipeline evaluation.

2.1 RAG VS. GRAPHRAG

  • RAG retrieves by direct semantic similarity, whereas GraphRAG traverses linked entities and subgraphs to support contextual analysis and multi-step reasoning.The paper characterizes RAG as suited to rapid access to discrete information and GraphRAG as emphasizing deeper analysis of interconnected data.Source: §2.1 · p. 3

2.2 CURRENT RAG BENCHMARKS

  • Existing benchmarks emphasize retrieval difficulty but inadequately represent complex logical synthesis and reasoning across domain-specific hierarchies.Their multi-hop questions can reduce to extracting discrete facts rather than integrating contextual evidence.Source: §2.2 · p. 4
  • Benchmark corpora often have inconsistent quality, low information density, or sparse domain hierarchies, limiting assessment across different knowledge structures.The paper argues that varying densities—from tightly structured domain knowledge to loosely organized documents—are needed to examine scalability.Source: §2.2 · p. 4
  • Output-only metrics treat graph construction and retrieval as black boxes, so the study argues for evaluating graph quality, retrieved knowledge, and answer faithfulness across the pipeline.The proposed scope includes graph construction efficiency, graph quality, retrieval structure and relevance, and generation faithfulness.Source: §2.2 · pp. 4–5

3 GRAPHRAG-BENCH

GraphRAG-Bench is designed to assess GraphRAG through progressively harder tasks, varied corpora, and evaluation across graph construction, retrieval, and generation.

3 GRAPHRAG-BENCH

  • GraphRAG-Bench combines increasingly difficult tasks, real-world corpora with different information densities, and evaluation from graph construction through final generation.Its task set covers fact retrieval, multi-hop reasoning, contextual summarization, and creative generation.Source: §3 · p. 5

3.1 TASK FORMULATION

  • The task hierarchy scales retrieval difficulty and reasoning complexity, using lower-level tasks for retrieval and higher levels to assess reasoning depth and contextual comprehension.The benchmark is intended to move beyond simple fact retrieval and linear multi-hop reasoning.Source: §3.1 · p. 5

3.2 DATASET CONSTRUCTION

  • Dataset construction combines hierarchical medical guidelines with loosely organized pre-20th-century novels, then derives structured evidence and calibrates questions to its complexity.Questions range from isolated subgraph fragments to global topology-aware reasoning, with validation and refinement after initial construction.Source: §3.2 · pp. 5–6

3.3 EVALUATION METRICS

  • Evaluation metrics cover graph structure, retrieval relevance and completeness, and generation quality, including answer accuracy, faithfulness, and evidence coverage.Graph measures include node and edge counts, average degree, and average clustering coefficient.Source: §3.3 · p. 6

4 EXPERIMENT

The experiments compare GraphRAG and RAG on generation, retrieval, graph structure, and efficiency. Results vary with task complexity, while GraphRAG’s broader retrieval can come with redundancy and higher prompt costs.

4.1 GENERATION ACCURACY (Q1)

  • Basic RAG matches or outperforms GraphRAG on simple fact retrieval, while GraphRAG has a clear advantage in complex reasoning, contextual summarization, and creative generation.On novel creative-generation tasks, RAPTOR reaches 70.9% faithfulness, while RAG covers more evidence (40.0%).Source: §4.1 · pp. 7–8, Figure 4 · p. 7

4.2 RETRIEVAL PERFORMANCE (Q2)

  • RAG achieves 83.2% Evidence Recall on simple novel questions; for Level 2–3 novel questions, HippoRAG reaches 87.9–90.9% Evidence Recall and HippoRAG2 reaches 85.8–87.8% Context Relevance.Medical-dataset results are reported to reinforce the pattern that graph retrieval benefits as questions become more complex.Source: §4.2 · p. 8

4.3 GRAPH COMPLEXITY (Q3)

  • HippoRAG2’s index graphs average 2,310 edges and 523 nodes per 10k novel-corpus tokens, and 3,979 edges and 598 nodes per 10k medical-corpus tokens.The paper reports that these denser graphs improve connectivity and coverage and coincide with higher retrieval recall.Source: §4.3 · p. 9, Figure 5 · p. 9

4.4 EFFICIENCY (Q4)

  • GraphRAG increases prompt length over vanilla RAG; MS-GraphRAG(global) reaches 4 × 10^4 tokens, LightRAG about 10^4, and HippoRAG2 about 10^3.MS-GraphRAG(global) grows from 7,800 to 40,000 tokens as task difficulty rises; the paper says this accumulation can introduce redundancy and degrade context relevance.Source: §4.4 · p. 9

5 CONCLUSION

The conclusion frames GraphRAG’s real-world effectiveness as unresolved and argues that current benchmarks inadequately assess it. The paper responds with a benchmark for hierarchical retrieval and deep contextual reasoning.

5 CONCLUSION

  • The conclusion notes empirical reports that GraphRAG often fails to outperform vanilla RAG on many NLP tasks, leaving its real-world effectiveness in question.Source: §5 · pp. 9–10
  • The authors identify a benchmark gap: existing evaluations inadequately assess GraphRAG because they lack domain-specific corpora and sufficiently granular tasks.Source: §5 · p. 10
  • GraphRAG-Bench is proposed to evaluate hierarchical knowledge retrieval and deep contextual reasoning.Source: §5 · p. 10

ETHICS STATEMENT

The appendix describes the benchmark’s data and evaluation choices, complexity controls, and reproducibility resources, alongside privacy and bias considerations.

ETHICS STATEMENT

  • The benchmark uses publicly available internet data, filters personally identifiable information, and advises users to consider possible societal biases in source data.The authors state that data collection respects user privacy and source-platform terms, and that all evaluated models are open-source.Source: ETHICS STATEMENT · p. 10

A.1 CODE AND LEADERBOARD

  • The authors provide code and evaluation scripts on GitHub for reproducibility, and the leaderboard ranks systems by average generation performance (ACC).Source: §A.1 · p. 17, Figure 6 · p. 17

A.2 WHY IS IT IMPORTANT TO INCLUDE TASKS WITH VARYING COMPLEXITY?

  • Varying task complexity tests whether systems can synthesize interconnected concepts, rather than only locate scattered facts.The appendix argues that assessments omitting complexity risk overestimating utility for real applications.Source: §A.2 · pp. 17–18

A.3 HOW TO CONTROL THE COMPLEXITY OF EACH TASK?

  • GraphRAG-Bench controls question complexity through Knowledge Breadth, the number of required triples, and Reasoning Depth, the number of inference hops.For the Novel dataset, the authors build a knowledge graph, form logic chains from relevant triples, and adjust triple count and information span.Source: §A.3 · p. 18

A.4 WHY CONSTRUCT TWO DATASETS FROM BOTH NOVELS AND THE MEDICAL CORPUS?

  • Medical guidelines test navigation of explicit hierarchies, while novels test inference across implicit, context-dependent relationships.Source: §A.4 · p. 18

A.5 WHY NOT USE OTHER FORMATS FOR QUESTION CONSTRUCTION?

  • The authors prioritize retrieval and reasoning complexity over question format, arguing that QA, multiple-choice, and fact-checking can require the same underlying search and reasoning.Source: §A.5 · p. 18

A.6 WHAT IS THE ADVANTAGE OF USING STAGE-SPECIFIC EVALUATION METRICS?

  • Stage-specific metrics distinguish weaknesses in graph construction, retrieval, and reasoning that end-to-end answer metrics can obscure.Examples include graph completeness, retrieval relevance, and contextual coherence.Source: §A.6 · pp. 18–19

A.7 COMPARISON WITH EXISTING BENCHMARKS AND ANALYSIS PAPERS.

  • GraphRAG-Bench combines medical guidelines and novels, tasks of increasing complexity, and stage-specific metrics to investigate when GraphRAG surpasses traditional RAG.The authors contrast this with prior studies using homogeneous datasets that miss synthesis of hierarchical expertise and unstructured narratives.Source: §A.7 · p. 19

B TAKEAWAY FINDINGS

The paper’s recommendations emphasize retrieval precision, graph quality, and context control as design priorities for GraphRAG systems.

B TAKEAWAY FINDINGS

  • Prioritize retrieving the key information needed to answer a question while minimizing redundant details, keeping context focused for reasoning and efficiency.Source: B TAKEAWAY FINDINGS · p. 19
  • Build quality graphs rather than simply larger ones; tightly connected communities can encode implicit multi-hop knowledge and support faster traversal.Source: B TAKEAWAY FINDINGS · p. 19
  • Manage context growth with search boundaries, since GraphRAG can accumulate entities, relations, and raw text snippets, raising reasoning costs.Source: B TAKEAWAY FINDINGS · p. 19

C BENCHMARK CONSTRUCTION

GraphRAG-Bench constructs a benchmark with task hierarchies and structured knowledge to control question difficulty. Its process formalizes domain logic, grounds questions in evidence, and checks and refines the resulting questions and answers.

C BENCHMARK CONSTRUCTION

  • Six construction stages integrate domain-specific logical hierarchies and contextual dependencies to control question difficulty.Source: C BENCHMARK CONSTRUCTION · p. 20

C.1 CORPUS COLLECTION

  • The corpus combines NCCN medical guidelines with lesser-known pre-20th-century novels selected for length and narrative ambiguity.The two datasets balance domain-specific hierarchies with loosely organized, ambiguous texts.Source: §C.1 · p. 20

C.2 LOGIC MINING

  • GPT-4.1 converts text into ontologies that make vertical hierarchies and horizontal dependencies explicit as a ground-truth map of domain logic.This provides a basis for distinguishing factual retrieval from logical synthesis and calibrating question difficulty.Source: §C.2 · p. 20

C.3 EVIDENCE COLLECTION

  • Evidence is extracted as localized subgraphs and multi-hop chains, with difficulty tied to entity density, path length, and inferential distance.Retrieval questions use context-rich subgraphs, while complex reasoning tasks require traversal of interconnected chains.Source: §C.3 · p. 21

C.4 QUESTION GENERATION

  • Question types align with evidence structure: retrieval targets subgraphs, reasoning uses short chains, summarization combines subgraphs, and creation draws on global graph topology.Difficulty scales with contextual synthesis and measurable graph properties rather than hop count alone.Source: §C.4 · p. 21

C.5 CHECK & CORRECT

  • Questions are discarded when their evidence cannot be logically derived from the corpus, and answers are checked for logical inference from the evidence.Advanced models and human checking support final confirmation.Source: §C.5 · pp. 21, 23

C.6 REFINEMENT

  • GPT-4.1 refines overly direct questions by adding relevant background from the corpus segment supporting their evidence.The refinement aims to retain logical structure while improving question context and clarity.Source: §C.6 · p. 23

D EXTENDING GRAPHRAG-BENCH TO NEW DOMAINS

GraphRAG-Bench extends its evaluation through domain-specific corpus suggestions and by addressing gaps in benchmark question coverage and graph structure. Its medical and novel subsets have more connected graphs than those analyzed in prior benchmarks.

D EXTENDING GRAPHRAG-BENCH TO NEW DOMAINS

  • The initial benchmark uses medical data, while its construction method is proposed as extendable to fields such as law and finance.Source: D EXTENDING GRAPHRAG-BENCH TO NEW… · p. 23
  • For legal extensions, the authors suggest case-law corpora and combining ontologies with logic chains to represent regulations, interpretations, and legal reasoning.Source: D EXTENDING GRAPHRAG-BENCH TO NEW… · p. 23
  • For finance extensions, the authors suggest S&P 500 company filings from 2015–2024 and an ontology schema, noting that numerical data may need further processing.Source: D EXTENDING GRAPHRAG-BENCH TO NEW… · p. 23
  • Existing benchmark graphs are sparse and fragmented; UltraDomain averages 2.7 entities per connected component and only 9% of its entities have degree greater than 3.Across the benchmarks analyzed, about 40% of entities are non-isolated, and the average degree reaches at most 0.86.Source: D EXTENDING GRAPHRAG-BENCH TO NEW… · p. 24
  • The medical and novel subsets have higher average degrees (1.05 and 2.27), non-isolated entity proportions (0.48 and 0.66), and average component sizes (3.15 and 3.99).The authors describe these graphs as more coherent and connected, offering a foundation for controlled study of graph structure in retrieval-augmented generation.Source: D EXTENDING GRAPHRAG-BENCH TO NEW… · pp. 24–25
  • Existing benchmarks show uneven task coverage: UltraDomain is 97% Contextual Summarize, HotpotQA is 78.2% Fact Retrieval, and MultiHop-RAG lacks basic Fact Retrieval questions.Source: D EXTENDING GRAPHRAG-BENCH TO NEW… · p. 25
  • GraphRAG-Bench balances four question categories and adds Creative Generation, addressing existing benchmarks' limited coverage of question difficulty and generative creativity.Source: D EXTENDING GRAPHRAG-BENCH TO NEW… · p. 25

F EVALUATION METRICS

The evaluation framework measures retrieval completeness and relevance alongside generation quality. In open-source-model experiments, different lightweight GraphRAG frameworks lead on different tasks; the appendix also reports complete results for 11 frameworks.

F EVALUATION METRICS

  • Retrieval evaluation uses Context Relevance for query alignment and Evidence Recall for retrieval completeness.The authors describe effective retrieval as both comprehensive and low in irrelevant content.Source: F EVALUATION METRICS · p. 25
  • Generation evaluation includes lexical overlap, answer accuracy, faithfulness, and evidence coverage.Answer accuracy combines semantic alignment and factual precision; faithfulness checks claim-to-context support, while evidence coverage assesses inclusion of reference knowledge.Source: F EVALUATION METRICS · pp. 25–26

G.1 EXPERIMENTS ON MORE GRAPHRAG FRAMEWORKS

  • The appendix reports complete results for 11 GraphRAG frameworks, while the main paper presents only a subset.Source: §G.1 · p. 26

G.2 EXPERIMENTS ON OPEN-SOURCE MODEL

  • Qwen2.5-14B is used as an open-source generation backbone, but only a representative subset of GraphRAG models is tested with it.Source: §G.2 · p. 26
  • On Qwen2.5-14B, Fast-GraphRAG achieves the highest fact-retrieval accuracy (60.08%) and ROUGE-L (41.31%), while other frameworks lead on selected Medical tasks.LightRAG leads Medical Contextual Summarize accuracy (69.37%) and creative-generation FS (70.84%); HippoRAG2 has the best ROUGE-L for fact retrieval (33.92%) and complex reasoning (33.02%).Source: §G.2 · p. 26

G.3 EXPERIMENTS ON GRAPH CONSTRUCTION EFFICIENCY

The experiments compare graph construction costs, GraphRAG’s sensitivity to LLM size, and its performance stability as corpus size grows.

G.3 EXPERIMENTS ON GRAPH CONSTRUCTION EFFICIENCY

  • 32.01 seconds and 89,215 tokens: KGP has the lowest reported construction time and token usage, while LightRAG takes over 710 seconds to index.The evaluation used one Novel dataset book of approximately 56k tokens and recorded construction time and token usage during indexing.Source: §G.3 · pp. 26–27, Table 15 · p. 27

G.4 EXPERIMENTS ON THE IMPACT OF LLM SIZE

  • Avg 57.08 to 63.91: GraphRAG improves as Qwen2.5 scales from 3B to 14B, while Standard RAG changes from Avg 56.13 to 59.10.The authors report a performance inflection at 7B: GraphRAG Avg rises from 57.59 to 61.50.Source: §G.4 · p. 27, Table 16 · p. 27

G.5 EXPERIMENTS ON SCALABILITY WITH CORPUS SIZE

  • 58.64% to 43.20%: Standard RAG’s Complex Reasoning accuracy falls from Small to Large, while HippoRAG2 remains stable across corpus scales.The study attributes this stability to graph structural constraints filtering retrieval noise as the search space expands.Source: §G.5 · p. 28, Table 17 · p. 28

H.1 IMPLEMENTATION DETAILS OF REPRESENTATIVE RAG MODELS

This section describes the knowledge structures and retrieval frameworks used in the evaluation, their configurations, and limitations of traditional RAG and the benefits attributed to graphs.

H.1 IMPLEMENTATION DETAILS OF REPRESENTATIVE RAG MODELS

  • Knowledge graphs contain entities and relations, textual knowledge graphs add entity descriptions and types, and tree structures organize document content and summaries.The table summarizes indexing, retrieval, and generation implementation details across the evaluated RAG models.Source: §H.1 · p. 28, Table 18 · p. 28
  • The evaluation includes local and global Microsoft GraphRAG, LightRAG, HippoRAG, HippoRAG2, and Fast-GraphRAG; GraphRAG defaults were retained for native-performance evaluation.The authors also describe RAPTOR, STRUCTRAG, KGP, LazyGraphRAG, and KET-RAG, and note that time limits restricted the set of assessed methods.Source: §H.1 · pp. 28–29

H.2 CONFIGURATION OF GRAPHRAG MODELS

  • All systems used bge-large-en-v1.5 for retrieval and a generation temperature of 0.7, while GraphRAG frameworks kept their default configurations.The authors retained framework-specific indexing, retrieval, and generation settings to assess practical native performance.Source: §H.2 · p. 29

I.1 TRADITIONAL RAG AND THEIR LIMITATIONS

  • Traditional chunk-based RAG can disconnect related concepts, lose contextual relationships, and retrieve redundant content that strains fixed context windows.The section says these limits hinder multi-hop domain reasoning and relational reasoning across distributed evidence.Source: §I.1 · p. 31

I.2 GRAPHRAG AND ITS ADVANTAGES

  • GraphRAG methods described here use hierarchical search, dual-level retrieval, pruning, or task-specific structures to support contextual and multi-hop retrieval.The section also describes graph-based integration of heterogeneous data and auditable retrieval pathways.Source: §I.2 · p. 33

I.3 EXISTING BENCHMARKS AND ANALYSIS

The authors present GraphRAG-Bench as an evaluation of reasoning across varied corpora and task complexity, while identifying text-only scope as a limitation. They also discuss broader implications and disclose their use of an LLM in manuscript preparation.

I.3 EXISTING BENCHMARKS AND ANALYSIS

  • GraphRAG-Bench evaluates hybrid corpora, increasingly complex tasks, and stage-specific metrics to investigate when GraphRAG outperforms traditional RAG and where failures occur.The authors identify prior benchmark limitations in problem design and corpus composition, and note that earlier analyses focused mainly on architectural comparisons using homogeneous datasets.Source: §I.3 · p. 33

J LIMITATION

  • The benchmark covers unimodal text contexts and omits multimodal integration, including diagrams, tables, and sensor-generated temporal sequences.The authors state that future iterations will extend evaluation to multimodal data.Source: J LIMITATION · p. 33

K BROADER IMPACT

  • The authors frame holistic evaluation of outputs and reasoning processes as relevant to knowledge-intensive domains and traceability of reasoning.They give clinical decision support and policy analysis as application examples.Source: K BROADER IMPACT · p. 34

L THE USE OF LARGE LANGUAGE MODELS (LLMS)

Loading 2506.05690v3…