Source-linked AI summary
WildGraphBench: Benchmarking GraphRAG with Wild-Source Corpora
Pengyu Wang, Benfeng Xu, Licheng Zhang, Shaohan Wang, Mingxuan Du, Chiwei Zhu, Zhendong Mao
TL;DR
Existing GraphRAG benchmarks under-test retrieval and reasoning over long, heterogeneous, uncurated corpora. WildGraphBench uses Wikipedia reference pages and citation-linked statements to evaluate three task types, finding advantages for multi-fact aggregation but persistently low summarization performance. The benchmark also notes that its gold statements and LLM-based evaluation may limit interpretation of absolute scores and fine-grained comparisons.
Problem
Existing GraphRAG benchmarks rely on short, curated passages and therefore provide limited evaluation of long-context retrieval over large, heterogeneous document collections.
Method
WildGraphBench samples Wikipedia articles across 12 topics, uses their external reference pages as retrieval corpora and citation-linked statements as ground truth, and constructs single-fact, multi-fact, and section-summary questions.
Results
GraphRAG offers limited gains on single-fact queries, significant advantages on multi-fact cross-document aggregation, and low summarization performance across methods.
Takeaways & Limitations
WildGraphBench indicates that graph-based retrieval is most useful for assembling scattered evidence across documents, while robust evidence acquisition and synthesis remain needed for wild-setting summaries.
Takeaways & Limitations
Wikipedia-derived gold statements may contain omissions or inaccuracies, and LLM-based judgment and statement matching may bias scores and fine-grained method comparisons.
Abstract
from arXiv · showhide
Graph-based Retrieval-Augmented Generation (GraphRAG) organizes external knowledge as a hierarchical graph, enabling efficient retrieval and aggregation of scattered evidence across multiple documents. However, many existing benchmarks for GraphRAG rely on short, curated passages as external knowledge, failing to adequately evaluate systems in realistic settings involving long contexts and large-scale heterogeneous documents. To bridge this gap, we introduce WildGraphBench, a benchmark designed to assess GraphRAG performance in the wild. We leverage Wikipedia's unique structure, where cohesive narratives are grounded in long and heterogeneous external reference documents, to construct a benchmark reflecting real-word scenarios. Specifically, we sample articles across 12 top-level topics, using their external references as the retrieval corpus and citation-linked statements as ground truth, resulting in 1,100 questions spanning three levels of complexity: single-fact QA, multi-fact QA, and section-level summarization. Experiments across multiple baselines reveal that current GraphRAG pipelines help on multi-fact aggregation when evidence comes from a moderate number of sources, but this aggregation paradigm may overemphasize high-level statements at the expense of fine-grained details, leading to weaker performance on summarization tasks. Project page:https://github.com/BstWPY/WildGraphBench.
1 Introduction
WildGraphBench addresses the limited realism of existing GraphRAG benchmarks by using long, heterogeneous Wikipedia reference corpora and citation-linked ground truth. It evaluates single-fact retrieval, multi-fact aggregation, and section-level summarization, finding stronger GraphRAG benefits for multi-fact tasks than broad summaries.
- Motivation: Existing GraphRAG benchmarks use short, curated passages, leaving long-context retrieval over heterogeneous, uncurated collections under-tested.Such settings can reduce evaluation to lookup-and-stitch rather than genuine multi-document aggregation.
- Benchmark Construction: WildGraphBench uses Wikipedia articles’ external reference pages as retrieval corpora and citation-linked statements as ground-truth facts.The benchmark samples articles from 12 top-level topics and retains the long, noisy reference pages.
- Task Design: 1,100+ questions span single-fact lookup, multi-fact evidence aggregation, and section-level summarization.These task types cover progressively broader requirements from precise retrieval to factual coverage.
- Findings: GraphRAG improves multi-fact aggregation but struggles with broad summary tasks.The reported experiments compare multiple methods across the benchmark’s three question types.
2 Related Work
Prior RAG and GraphRAG research combines retrieval with language-model generation, while graph methods add structured representations and graph operations for multi-document evidence assembly. Existing benchmarks cover multi-hop QA, longer domain corpora, and graph-based retrieval, but differ in evidence settings and corpus structure.
- Retrieval-Augmented Generation: RAG retrieves related text from an external corpus and generates an answer from that evidence, commonly using a retriever with an LLM reader.Prior work reports reduced hallucination compared with direct generation when evidence is needed.
- Graph Retrieval-Augmented Generation: GraphRAG builds graphs over chunks or documents and retrieves evidence through graph operations to capture dependencies that flat retrieval may miss.Representative systems use hierarchical summaries or entity–relation graphs alongside retrieval.
- Benchmarks: Existing benchmarks target multi-hop QA, longer domain corpora, or graph-based retrieval under different evidence settings.Examples include HotpotQA, 2WikiMultiHopQA, MultiHop-RAG, UltraDomain, and GraphRAG-Bench.
3 WildGraphBench
WildGraphBench constructs a Wikipedia-reference benchmark with three question types and statement-grounded evaluation, covering noisy evidence from long, heterogeneous sources. Its workflow extracts citation-linked gold statements, designs questions, and evaluates factual accuracy or statement-level coverage.
- Dataset construction: The benchmark collects noisy reference pages from articles spanning 12 high-level Wikipedia topics, retaining raw page text to simulate wild retrieval conditions.Articles are selected for dense and diverse citation structures, and archived pages are used when originals fail.
- Gold corpus construction: Citation-linked Wikipedia sentences are normalized into factual statements and aligned with crawled reference URLs to form the gold corpus.Sentences with missing referenced pages are dropped, and each leaf section stores statement, reference-URL, and reference-count information.
- Question design: WildGraphBench designs single-fact, multi-fact, and section-level summary questions from reference counts and leaf-section statement sets.Single-fact questions use triples with ref_count = 1, multi-fact questions use ref_count ≥2 with a multi-reference check, and summary questions target all valid statements in a leaf section.
- Question design: The dataset contains 1,197 questions distributed across three distinct categories.
- Evaluation: Single-fact and multi-fact answers receive binary statement-equivalence accuracy scores, while summaries are evaluated using statement-level precision, recall, and F1.Summary evaluation extracts predicted statements and matches them to gold statements through correct paraphrase equivalence; F1 measures the harmonic mean of precision and recall.
4 Experiments
Experiments show task-dependent GraphRAG behavior: graph methods help aggregate scattered evidence for multi-fact questions, while flat retrieval remains competitive for single facts and summaries. Graph connectivity and retrieval-budget analyses further show that wild corpora create dense, hub-centric structures, but summary performance depends on balancing coverage against noise.
- Experimental Setup: GraphRAG methods are evaluated against flat baselines including NaiveRAG and BM25, using gpt-4o-mini for construction and answering and gpt-5-mini for judging.Documents are chunked into 1200-token segments with 100-token overlap; top_k is 5 for fact questions and 10 for summaries.
- Main Results: 71.51 vs. 66.87, HippoRAG2 exceeds NaiveRAG on single-fact accuracy, but flat retrieval remains competitive when one salient chunk supports the answer.BM25 also surpasses several graph methods in some cases, so graph retrieval does not automatically improve straightforward lookup.
- Main Results: 47.64, Microsoft GraphRAG(global) achieves the best multi-fact accuracy, where structured traversal and global aggregation help combine scattered evidence.Several graph variants are comparable to or better than NaiveRAG and BM25 on this harder task.
- Main Results: Summary questions receive low statement-level scores across methods, while NaiveRAG achieves the highest recall and best F1 through broader evidence coverage.Graph pipelines can lose recall through imperfect extraction, sparsification, summarization, or limited traversal budgets, even when filtering improves precision.
- Graph Analysis: 3.11 average degree and 0.14 isolated-node proportion indicate that WildGraphBench has the densest connectivity under the shared LightRAG construction pipeline.Its hub-and-spoke structure links many pages through repeatedly referenced entities, requiring aggregation of partially overlapping evidence.
- Retrieval Budget: k = 8 is the optimal retrieval budget in the HippoRAG2 summary experiment: F1 rises with k, then declines when additional chunks add noise.Small budgets limit recall, whereas excessive budgets can overwhelm the generator with distractors and increase hallucinations or loss of focus.
5 Conclusions
WildGraphBench evaluates GraphRAG on heterogeneous reference-page corpora with increasingly demanding retrieval, aggregation, and summarization tasks. Graph retrieval helps most for cross-document multi-fact questions, whereas summarization remains difficult for all methods in this wild setting.
- Conclusion: WildGraphBench uses heterogeneous Wikipedia reference pages to test GraphRAG under wild-source conditions.The benchmark stresses retrieval, aggregation, and summarization in uncurated environments.
- Conclusion: Graph-based retrieval offers limited gains over strong flat baselines on simple single-fact queries but significant advantages on cross-document multi-fact aggregation.The results identify evidence assembly across multiple sources as the setting where graph retrieval is most beneficial.
- Conclusion: Summarization performance remains low across all methods, underscoring the need for more robust evidence acquisition and synthesis in real-world scenarios.Broad summaries remain difficult when evidence is noisy and distributed across long contexts.
Limitations
The benchmark’s gold statements and evaluation process introduce interpretation limits for absolute scores and fine-grained method comparisons.
- Limitations: Wikipedia-derived gold statements reflect editorial consensus rather than absolute truth and may inherit omissions or inaccuracies from articles and citations.This limits how directly benchmark scores represent complete or objectively correct knowledge.
- Limitations: LLM-based judgment and statement matching may introduce systematic biases, including preferences for particular phrasing or verbosity.These evaluation effects may not perfectly mirror unbiased human assessment.
Ethical considerations
WildGraphBench is intended for research-only benchmarking using third-party Wikipedia-linked content, with safety and licensing constraints. Because its web corpus may contain harmful or unreliable material, it evaluates robustness rather than absolute truth.
- WildGraphBench is intended for research-only benchmarking of retrieval robustness and multi-document evidence aggregation.
- Redistribution and derivative works must comply with the original sources’ licenses, terms, and access conditions.
- The long, heterogeneous web corpus may contain noise, bias, outdated claims, toxic language, or inadvertently exposed sensitive information.
- Users should apply appropriate safety practices, including toxicity and PII filtering when needed.
- WildGraphBench evaluates robustness rather than establishing absolute truth.
A Dataset Statistics by Domain
Table 5 reports the number of questions in WildGraphBench for each domain.
- Table 5 presents per-domain question counts for WildGraphBench.
B Results on every domain
The domain-level results are organized across twelve topical categories, covering culture, geography, health, history, human activities, mathematics, nature, people, philosophy, religion, society, and technology.
- Domain-level results include Culture, Geography, Health, and History.
- Additional results cover Human Activities, Mathematics, Nature, and People.
- The remaining domain categories are Philosophy, Religion, Society, and Technology.
C Prompts for Data Construction
WildGraphBench constructs single-fact, multi-fact, and summary questions from citation-linked Wikipedia statements and reference documents, then filters them for support and relevance.
- Question Generation: Single-fact: Single-fact questions use a Wikipedia sentence, cleaned factual statement, section path, article title, and cited reference URLs.
- Prompt Documentation: Figures 6–11 document the generation and filtering prompts for single-fact, multi-fact, and summary questions.
- Question Generation: Summary: Summary questions are topic-centered and specify gold statements that a good answer should cover.
- Summary Question Filtering: Summary questions undergo post-hoc verification using the article title, leaf section path, leaf topic, and multiple items.
- Question Filtering: Single-fact and multi-fact filtering check whether the provided references collectively support the question-answer pair.
- Question Generation: Multi-fact: Multi-fact question construction judges whether several references are jointly necessary to support the full factual content.