Source-linked AI summary
From Local to Global: A Graph RAG Approach to Query-Focused Summarization
Darren Edge, Ha Trinh, Newman Cheng, Joshua Bradley, Alex Chao, Apurva Mody, Steven Truitt, Dasha Metropolitansky, Robert Osazuwa Ness, Jonathan Larson
TL;DR
Vector RAG does not handle global sensemaking over large corpora well, while prior query-focused summarization methods do not scale to typical RAG-sized collections. GraphRAG builds a knowledge graph and hierarchical community summaries, then synthesizes query-specific answers from them. On approximately million-token datasets, it substantially improves answer comprehensiveness and diversity over vector RAG, while evaluation remains limited to two corpora.
Problem
Vector RAG fails on corpus-wide sensemaking questions, while prior query-focused summarization methods do not scale to the quantities of text indexed by typical RAG systems.
Method
GraphRAG uses an LLM to extract entities, relationships, and claims into a knowledge graph, generate hierarchical community summaries, and combine community answers into a global response.
Results
GraphRAG substantially improves comprehensiveness and diversity over vector RAG for global sensemaking questions on approximately million-token datasets.
Takeaways & Limitations
Root-level graph-community summaries provide a reusable index that is superior to vector RAG and competitive with graph-free global methods at a fraction of the token cost.
Takeaways & Limitations
Evaluation covers only two approximately million-token corpora, and broader domain generalization and fabrication-rate comparisons remain open.
Abstract
from arXiv · showhide
The use of retrieval-augmented generation (RAG) to retrieve relevant information from an external knowledge source enables large language models (LLMs) to answer questions over private and/or previously unseen document collections. However, RAG fails on global questions directed at an entire text corpus, such as "What are the main themes in the dataset?", since this is inherently a query-focused summarization (QFS) task, rather than an explicit retrieval task. Prior QFS methods, meanwhile, do not scale to the quantities of text indexed by typical RAG systems. To combine the strengths of these contrasting methods, we propose GraphRAG, a graph-based approach to question answering over private text corpora that scales with both the generality of user questions and the quantity of source text. Our approach uses an LLM to build a graph index in two stages: first, to derive an entity knowledge graph from the source documents, then to pregenerate community summaries for all groups of closely related entities. Given a question, each community summary is used to generate a partial response, before all partial responses are again summarized in a final response to the user. For a class of global sensemaking questions over datasets in the 1 million token range, we show that GraphRAG leads to substantial improvements over a conventional RAG baseline for both the comprehensiveness and diversity of generated answers.
1 Introduction
Vector RAG answers localized queries well but does not support sensemaking over an entire corpus. GraphRAG addresses this gap with an LLM-derived knowledge graph, hierarchical community summaries, and query-focused synthesis.
- RAG limitations: Vector RAG retrieves records relevant to a query and fits them within the LLM context window.It works well when the answer is localized within a small set of records.
- RAG limitations: Global sensemaking requires reasoning over interconnected concepts across an entire corpus, which conventional vector RAG cannot support.The challenge arises when corpus volume requires retrieval rather than direct summarization.
- GraphRAG approach: GraphRAG constructs a knowledge graph, hierarchically partitions related entities into communities, and generates community-level summaries.The summaries are produced bottom-up across the community hierarchy.
- Evaluation: GraphRAG evaluates global sensemaking with LLM-generated questions and LLM judging across two real-world datasets.The evaluation compares GraphRAG with vector RAG using predefined criteria.
- Evaluation: GraphRAG strongly outperforms vector RAG when using GPT-4 as the LLM.The paper presents this as evidence for GraphRAG’s global sensemaking ability.
2 Background
The paper positions GraphRAG as a knowledge-graph-based extension of RAG for global sensemaking, supported by adaptive evaluation questions and LLM-based comparative judging.
- RAG background: RAG retrieves query-relevant records from external sources and incorporates them into an LLM-generated response.It is useful when the source exceeds the model’s context window.
- RAG background: Vector RAG retrieves semantically similar chunks, whereas GraphRAG is designed to answer questions requiring global understanding.The contrast is between explicit fact retrieval and corpus-wide sensemaking.
- Related approaches: GraphRAG combines LLM-generated knowledge-graph extraction with summaries over large data sections and iterative global aggregation.This connects graph indexing with query-focused summarization and map-reduce-style synthesis.
- Evaluation: Adaptive benchmarking uses inferred users and use cases to generate diverse, corpus-specific global sensemaking queries.The procedure is tailored to the target domain rather than relying only on generic benchmarks.
- Evaluation: LLM-as-a-judge compares competing generations using criteria designed for global sensemaking when no gold-standard answer exists.Generic vector-RAG criteria are not sufficient for this evaluation target.
3 Methods
GraphRAG transforms chunked documents into an LLM-derived knowledge graph, organizes the graph into hierarchical communities, and summarizes those communities before generating query-specific global answers.
- Source processing: Documents are split into chunks before an LLM extracts information for downstream processing.Longer chunks reduce calls and cost but can reduce recall for information appearing early in a chunk.
- Knowledge-graph extraction: The LLM extracts entities, relationships, and claims from each chunk using domain-tailored prompts and few-shot examples.Extracted entities and relationships receive short descriptions, while claims capture factual statements about entities.
- Knowledge-graph extraction: Extracted elements are abstractive summaries that are aggregated into graph nodes, weighted edges, and claims.Repeated detections create multiple instances that are consolidated during graph construction.
- Community hierarchy: Leiden community detection recursively partitions the graph into a hierarchy of strongly connected communities.Each level provides a mutually exclusive, collectively exhaustive partition for divide-and-conquer summarization.
- Community summarization: Community summaries are generated bottom-up, substituting shorter lower-level summaries when higher-level content exceeds the context limit.Element summaries are prioritized by graph prominence at leaf level.
- Query answering: For a query, community summaries are chunked, mapped to parallel scored answers, and reduced into a final global answer.Intermediate answers with score 0 are filtered, and the most helpful answers are prioritized for the final context.
4 Analysis
The evaluation compares GraphRAG community-summary levels with source-text summarization and vector RAG on approximately million-token corpora under matched context and prompting conditions.
- Datasets: The evaluation uses podcast and news corpora containing approximately 1 million and 1.7 million tokens, respectively.The podcast corpus contains 1669 chunks, while the news corpus contains 3197 chunks.
- Experimental conditions: Six conditions compare GraphRAG at four community levels with text summarization and vector-RAG semantic search.C0 uses root summaries and C3 uses low-level summaries.
- Experimental conditions: All six conditions use the same context-window size and answer-generation prompts, differing only in how context contents are created.This controls the comparison across indexing and summarization strategies.
- Implementation: Graph indexing uses domain-tailored entity types and few-shot examples, with an 8k-token context window for summaries and answers.Podcast graph indexing took 281 minutes under the reported hardware and GPT-4 Turbo endpoint setup.
- Metrics: Comprehensiveness is measured by average factual claims per answer, while diversity is measured by the average number of claim clusters.The claim-based validation extracts and deduplicates factual claims before clustering.
5 Results
Across datasets, GraphRAG's global approaches outperformed vector RAG in comprehensiveness and diversity, while community summaries reduced context requirements. Claim-based measures broadly corroborated these findings, though differences among global methods were generally nonsignificant.
- Global approaches vs. vector RAG: 72–83% comprehensiveness win rates on Podcast and 72–80% on News favored global approaches over vector RAG (p<.001).Diversity win rates were 75–82% for Podcast (p<.001) and 62–71% for News (p<.01).
- Community summaries vs. source texts: Root-level community summaries used over 97% fewer context tokens than source-text summarization, while low-level summaries used 26–33% fewer.Root-level GraphRAG retained 72% and 62% win rates over vector RAG for comprehensiveness and diversity, respectively.
- Community summaries vs. source texts: 57% and 64% comprehensiveness win rates favored intermediate Podcast and low-level News community summaries over source texts, respectively.Diversity win rates were 57% for Podcast intermediate summaries and 60% for News low-level summaries.
- Claim-based validation: All global search conditions and source-text summarization produced more extracted claims than vector RAG for both datasets, with all differences significant (p<.05).These claim-based results aligned with the LLM-based win rates from Experiment 1.
- Claim-based validation: Claim-based diversity significantly exceeded vector RAG for all global conditions on Podcast, but only C0 consistently did so across News distance thresholds (p<.05).News differences were smaller than Podcast differences, matching the directional pattern from the LLM-based results.
- Cross-condition comparisons: No statistically significant differences appeared among global search conditions or between global search and source-text summarization for comprehensiveness or diversity.This limits evidence for distinctions among the global approaches themselves despite their advantage over vector RAG.
6 Discussion
The evaluation is limited to global sensemaking questions over two approximately 1-million-token corpora, while broader domain generalization and fabrication-rate comparisons remain open. The paper also identifies future extensions combining graph structure with local retrieval and hierarchical exploration.
- 6.1 Limitations of evaluation approach: Evaluation covers sensemaking questions specific to two corpora, each containing approximately 1 million tokens.The authors call for testing across domains with different use cases.
- 6.1 Limitations of evaluation approach: Fabrication-rate comparisons, such as with SelfCheckGPT, would strengthen the current analysis.
- 6.2 Future work: Hybrid RAG could combine embedding-based query matching with just-in-time community report generation and map-reduce summarization.
- 6.2 Future work: Generated answers can pose downstream sensemaking and decision-making risks when they do not accurately represent source data.The paper recommends clear disclosures of AI use and potential output errors.
7 Conclusion
GraphRAG combines knowledge-graph generation with query-focused summarization to support sensemaking over entire text corpora. Its root-level community summaries outperform vector RAG and remain competitive with other global methods at lower token cost.
- 7 Conclusion: GraphRAG combines knowledge graph generation and query-focused summarization to support human sensemaking over entire text corpora.
- 7 Conclusion: GraphRAG substantially improves answer comprehensiveness and diversity over a vector RAG baseline.It also compares favorably with a global graph-free map-reduce approach.
- 7 Conclusion: Root-level community summaries provide a data index superior to vector RAG and competitive with other global methods at a fraction of the token cost.This conclusion applies to situations requiring many global queries over the same dataset.
- 7 Conclusion: The default GraphRAG initialization pipeline uses prompts for graph extraction and claim extraction.
- 7 Conclusion: Graph extraction identifies entities and relationships as delimited tuples containing names, types, descriptions, and relationship information.
A.2 Self-Reflection
Self-reflection addresses reduced entity extraction from larger chunks by prompting the LLM to assess and recover missed entities. The approach supports larger chunks without a drop in quality or forced noise, while trading against additional processing.
- A.2 Self-Reflection: Self-reflection prompts the LLM to evaluate an extraction for correctness, clarity, or completeness before generating an improved response.
- A.2 Self-Reflection: Larger chunks reduce LLM calls but can yield fewer extracted entities; GPT-4 found almost twice as many HotPotQA entity references at 600 than 2400 tokens.
- A.2 Self-Reflection: The pipeline feeds extracted entities back to the LLM, which assesses whether entities were missed and performs additional gleaning when necessary.Self-reflection steps continue up to a specified maximum.
- A.2 Self-Reflection: The approach enables larger chunk sizes without a drop in extraction quality or the forced introduction of noise.Figure 3 examines entity references across chunk sizes and self-reflection iterations.
B Example Community Detection
The example visualizes hierarchical communities detected in the indexed MultiHop-RAG entity graph. Node size encodes degree, while colors distinguish communities at two clustering levels.
- B Example Community Detection: Leiden detects graph communities in the indexed MultiHop-RAG dataset at two hierarchical clustering levels.Level 0 is the maximum-modularity partition, while Level 1 reveals internal structure within root communities.
- B Example Community Detection: Circles represent entity nodes, with node size proportional to degree and node colors representing entity communities.OpenORD and Force Atlas 2 determine the node layout.
C Context Window Selection
The study examines how context-window size affects task outcomes and seeks the optimum size for the baseline condition across datasets, questions, and metrics.
- The study varies context-window size across combinations of datasets, questions, and metrics.The purpose is to examine its effects on the evaluated tasks.
- The effect of context-window size on a particular task is unclear, including for models with 128k-token contexts.The passage specifically discusses gpt-4-turbo and its large context size.
- The experiment seeks the optimum context size for the baseline condition (SS).
D Example Answer Comparison
The example comparison section presents a report-generation workflow that organizes entity communities, synthesizes analyst reports, and grounds claims in referenced data.
- Report structure: The example report organizes a community around its entities, relationships, and associated claims.Its required structure includes a title, summary, impact severity rating, rating explanation, and findings.
- Report structure: The workflow requires a comprehensive summary of the community’s structure, entity relationships, and significant associated information.
- Output format: The report format uses JSON with fields for title, summary, rating, rating explanation, and findings.
- Grounding rules: Supported points must preserve data references, limiting each reference to the top 5 record ids before adding “+more”.The examples show references attached to claims about entities and relationships.
- Synthesis rules: The final response removes irrelevant report content, merges the remaining information, and preserves the original meaning of modal verbs and data references.
G Statistical Analysis
The statistical analysis compares six conditions across four metrics, using pairwise scores averaged over repeated evaluation runs and non-parametric testing because the data were non-normal.
- Six conditions are compared on four metrics across 125 questions and two datasets.
- For each question and metric, winning conditions receive 100, losing conditions 0, and tied conditions 50.
- The comparison scores are averaged over five evaluation runs for each condition.
- Shapiro-Wilk tests indicated that the data did not follow a normal distribution, so the analysis uses non-parametric tests.The passage identifies Wilcoxon signed-rank tests as the subsequent testing approach.