PaperlayerView exact revision on arXiv

Source-linked AI summary

WildGraphBench: Benchmarking GraphRAG with Wild-Source Corpora

Pengyu Wang, Benfeng Xu, Licheng Zhang, Shaohan Wang, Mingxuan Du, Chiwei Zhu, Zhendong Mao

arXiv:2602.02053v2cs.CL

TL;DR

GraphRAG benchmarks often rely on short, curated passages, leaving long-context, heterogeneous corpora under-tested. WildGraphBench builds a benchmark from Wikipedia references and citation-linked statements; its experiments show advantages for multi-fact aggregation while summarization remains difficult.

  • Problem

    Long-context retrieval over large, heterogeneous corpora remains under-tested in GraphRAG benchmarks, despite its relevance to real-world applications.

    Source: §1 · pp. 1–2

  • Method

    WildGraphBench uses Wikipedia reference pages as retrieval corpora and citation-linked article statements as ground truth for questions across 12 topics.

    Source: §1 · p. 2

  • Results

    Graph-based retrieval shows significant advantages on cross-document multi-fact aggregation, while summarization performance remains low across methods in this wild setting.

    Source: §5 · pp. 8–9

  • Takeaways & Limitations

    The results point to a need for more robust evidence acquisition and synthesis mechanisms in real-world scenarios.

    Source: §5 · p. 9

    Wikipedia-derived gold may inherit omissions or inaccuracies, while LLM-based evaluation may bias absolute scores and fine-grained method comparisons.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Graph-based Retrieval-Augmented Generation (GraphRAG) organizes external knowledge as a hierarchical graph, enabling efficient retrieval and aggregation of scattered evidence across multiple documents. However, many existing benchmarks for GraphRAG rely on short, curated passages as external knowledge, failing to adequately evaluate systems in realistic settings involving long contexts and large-scale heterogeneous documents. To bridge this gap, we introduce WildGraphBench, a benchmark designed to assess GraphRAG performance in the wild. We leverage Wikipedia's unique structure, where cohesive narratives are grounded in long and heterogeneous external reference documents, to construct a benchmark reflecting real-word scenarios. Specifically, we sample articles across 12 top-level topics, using their external references as the retrieval corpus and citation-linked statements as ground truth, resulting in 1,100 questions spanning three levels of complexity: single-fact QA, multi-fact QA, and section-level summarization. Experiments across multiple baselines reveal that current GraphRAG pipelines help on multi-fact aggregation when evidence comes from a moderate number of sources, but this aggregation paradigm may overemphasize high-level statements at the expense of fine-grained details, leading to weaker performance on summarization tasks. Project page:https://github.com/BstWPY/WildGraphBench.

1 Introduction

WildGraphBench targets a gap in GraphRAG evaluation by testing evidence retrieval and synthesis over long, heterogeneous sources rather than short, curated passages. It uses Wikipedia reference pages and citation-linked statements to create 1,100+ questions across single-fact, multi-fact, and section-level summary tasks.

  • Existing GraphRAG benchmarks rely on short, curated passages, leaving long-context retrieval over large heterogeneous corpora under-tested and enabling lookup-and-stitch answers instead of genuine aggregation.Even longer-document and GraphRAG-specific benchmarks retain cleaner boundaries or more structured, less heterogeneous corpora than wild sources.Source: §1 · pp. 1–2
  • WildGraphBench samples Wikipedia articles across 12 topics, using their external reference pages as retrieval corpora and citation-linked statements as ground-truth facts.The benchmark contains 1,100+ questions designed to test synthesis across scattered support from multiple sources.Source: §1 · p. 2
  • Its single-fact, multi-fact, and section-level summary questions span precise retrieval to broad factual coverage, with statement-grounded evaluation across all three types.Source: §1 · p. 2
  • Experiments show GraphRAG improves multi-fact aggregation but struggles with broad summary tasks.Source: §1 · p. 2

2 Related Work

Related work traces retrieval-augmented generation from retrieve-then-generate systems to graph-based evidence aggregation, then surveys RAG and GraphRAG benchmarks and identifies a gap for evaluation in wild, unstructured web settings.

Retrieval-Augmented Generation

  • RAG retrieves external text for an LLM to answer queries, and prior work finds retrieval can reduce hallucination when evidence is needed.Common systems pair a retriever such as BM25 with an LLM reader (Chen et al., 2017); retrieval-augmented generation is described by Lewis et al. (2021), and reduced hallucination by Shuster et al. (2021).Source: §2 · p. 3

Graph Retrieval-Augmented Generation

  • GraphRAG retrieves evidence through graph operations to capture dependencies that flat retrieval may miss.Microsoft GraphRAG (Edge et al., 2025) uses community detection and hierarchical summaries for local entity queries and global themes, while LightRAG (Guo et al., 2025) combines a dual-level entity–relation graph with vector retrieval for multi-stage evidence expansion.Source: §2 · p. 3
  • Other GraphRAG systems target efficiency, noise filtering, multi-hop paths, or scalability through lightweight indexing, Personalized PageRank, and linear-complexity propagation.These approaches include Fast-GraphRAG, HippoRAG2 (Gutiérrez et al., 2025), and LinearRAG (Zhuang et al., 2025); collectively, the methods demonstrate graph traversal for evidence aggregation compared with flat baselines.Source: §2 · p. 3

Benchmarks for RAG and GraphRAG

3 WildGraphBench

WildGraphBench builds a benchmark from citation-linked Wikipedia statements and their external reference pages, then evaluates question answering and summaries against gold factual statements. Its design spans single-fact, multi-fact, and section-level summary questions, with metrics tailored to factual correctness and coverage.

3 WildGraphBench

  • The benchmark workflow collects reference pages, extracts citation-linked statements, creates three question types, and evaluates answers against statements.The three phases produce a Wikipedia gold corpus, single-fact, multi-fact, and section-level summary questions, and statement-grounded evaluation.Source: §3 · p. 3

3.1 Data Collection

  • Articles with many references are sampled from 12 Wikipedia topics, and raw fetched page text—including boilerplate and noise—is retained to simulate wild retrieval.Archived pages are used when the original page fails, supporting data completeness.Source: §3.1 · p. 4

3.2 Statement Extraction

  • Citation-marked sentences in Wikipedia leaf sections are rewritten as clean factual statements, with local coreferences resolved and exact reference URLs retrieved.Each leaf section is identified by its section path.Source: §3.2 · p. 4

3.3 Wikipedia Gold Corpus Construction

  • Cited sentences missing any referenced page are dropped; the resulting leaf-section corpus stores each normalized statement, its reference URLs, and reference count as gold data.The corpus is treated as the authoritative source of gold statements for evaluation.Source: §3.3 · p. 4

3.4 Question Design

  • Single-fact questions use statements with one reference, multi-fact questions require at least two jointly necessary references, and summary questions target deduplicated statements from a leaf section.Summary prompts use the article title and section path, not section text, and test retrieval and summarization under noisy, long-context evidence conditions.Source: §3.4 · p. 5

3.5 Evaluation Metrics

  • Single- and multi-fact answers receive binary factual-equivalence accuracy scores, while summaries are scored by statement-level precision, recall, and their harmonic-mean F1.Summary matching accepts correct paraphrases, measuring factual coverage and hallucination rate while remaining robust to minor paraphrasing.Source: §3.5 · p. 5

4 Experiments

Across WildGraphBench, GraphRAG’s clearest advantage is multi-fact aggregation, while flat retrieval remains competitive on single-fact questions and NaiveRAG leads summary recall and F1. The experiments also examine graph connectivity, human performance, and retrieval-budget effects on summarization.

4.1 Experimental Settings

  • Evaluation compares flat NaiveRAG (Lewis et al., 2021) and BM25 with Fast-GraphRAG, Microsoft GraphRAG, LightRAG, LinearRAG, and HippoRAG2.Pre-chunking uses 1,200-token chunks with 100-token overlap; top_k is 5 for single- and multi-fact questions and 10 for summaries. gpt-4o-mini handles graph construction and answering, while gpt-5-mini judges.Source: §4.1 · p. 6

4.2 Main Results

  • 47.64 accuracy makes Microsoft GraphRAG(global) the best method on multi-fact questions, where graph variants can outperform flat baselines on evidence aggregation.The reported advantage is associated with combining evidence scattered across references; a flat top-k pipeline may miss complementary information.Source: Table 2 · p. 6, §4.2 · p. 6
  • NaiveRAG achieves the highest recall and best F1 on summary questions in our Dataset, while all methods have low statement-level scores.The passage attributes NaiveRAG’s advantage plausibly to broader evidence coverage; graph extraction, sparsification, summarization, and traversal constraints can limit recall under noisy long-context conditions.Source: Table 2 · p. 6, §4.2 · pp. 6–7
  • NaiveRAG achieves 66.87 single-fact accuracy on the Novel Dataset, exceeded among GraphRAG methods only by HippoRAG2 at 71.51.The passage notes that BM25 is also competitive and can surpass several graph methods on straightforward fact lookup.Source: §4.2 · p. 6

4.4 Human Performance

  • 15.30 F1 on the people subset makes human performance a strong upper-bound reference; annotators prioritize comprehensive fact coverage, sometimes at the cost of precision.Domain-knowledgeable annotators answered under the same evidence constraint as RAG systems.Source: Table 3 · p. 7, §4.4 · p. 8

4.3 Graph Analysis

  • Other benchmark graphs are less connected: curated UltraDomain, compact HotpotQA evidence, and locality or extraction challenges in GraphRAG-Bench are associated with sparser graphs.The passage describes narrative locality and coreference in the Novel subset, and document discontinuity plus specialized-term extraction ambiguity in the Medical subset.Source: §4.3 · pp. 7–8
  • 3.11 average degree and 0.14 isolated nodes give WildGraphBench the strongest connectivity under shared LightRAG construction; its max degree exceeds UltraDomain’s despite a smaller token budget.The authors interpret this structure as hub entities linked from many distinct pages, requiring aggregation of partially overlapping evidence.Source: Table 4 · p. 7, §4.3 · pp. 7–8

4.5 Ablation Study

  • HippoRAG2 summary F1 follows an inverted U as top-k rises from 2 to 12, peaking at k = 8 before declining.The passage attributes the decline beyond k = 8 to distractor noise, while too-small budgets limit evidence coverage and recall.Source: §4.5 · p. 8

5 Conclusions

WildGraphBench evaluates GraphRAG on heterogeneous Wikipedia reference corpora through single-fact QA, multi-fact QA, and summarization. Experiments show gains for cross-document aggregation, but low summarization performance across methods.

  • F1 for summary questions peaks at k = 8 as retrieval budget grows, then drops when k is too large.Source: Figure 5 · p. 8
  • Graph-based retrieval offers limited gains over strong flat baselines on single-fact queries but significant advantages on multi-fact questions requiring cross-document evidence aggregation.Source: §5 · pp. 8–9
  • WildGraphBench uses heterogeneous Wikipedia reference pages as its retrieval corpus and article-grounded statements to construct three progressively challenging tasks that stress-test retrieval, aggregation, and summarization.Source: §5 · p. 8
  • Summarization performance remains low across all methods in this wild setting, highlighting the need for more robust evidence acquisition and synthesis mechanisms.Source: §5 · p. 9

Limitations

The benchmark’s Wikipedia-derived gold statements may inherit omissions or inaccuracies, while LLM-based evaluation may introduce bias; both limit interpretation of absolute scores and fine-grained comparisons.

  • Wikipedia-derived gold statements reflect editorial consensus rather than absolute truth and may inherit omissions or inaccuracies from Wikipedia and its citations.Source: Limitations · p. 9
  • LLM-based judgment and statement matching may favor certain phrasing or verbosity and may not mirror unbiased human assessment.Source: Limitations · p. 9
  • These limitations warrant caution when interpreting absolute scores and fine-grained comparisons between methods.Source: Limitations · p. 9

Ethical considerations

WildGraphBench uses citation-linked Wikipedia statements as gold facts and cited pages as a noisy retrieval corpus, while respecting source terms and limiting its stated purpose to research benchmarking. Its heterogeneous web corpus may contain harmful or unreliable content, so the benchmark evaluates robustness rather than absolute truth.

Ethical considerations

  • WildGraphBench is intended for research-only benchmarking of retrieval robustness and multi-document evidence aggregation; redistribution or derivatives must follow source conditions.The dataset uses citation-linked Wikipedia statements as gold facts and cited pages as a noisy retrieval corpus, and discourages non-research use unless source conditions explicitly permit it.Source: Ethical considerations · p. 9
  • Long, heterogeneous web pages may contain noise, bias, outdated claims, toxic language, or inadvertently exposed sensitive information.The authors recommend standard safety practices such as toxicity or PII filtering when appropriate, and emphasize that WildGraphBench evaluates robustness rather than establishing absolute truth.Source: Ethical considerations · p. 9

A Dataset Statistics by Domain

Table 5 reports WildGraphBench question counts by domain.

A Dataset Statistics by Domain

  • Table 5 presents the per-domain question counts in WildGraphBench.Source: Table 5 · p. 11

B Results on every domain

The section presents domain-specific results in twelve tables, covering Culture, Geography, Health, History, Human Activities, Mathematics, Nature, People, Philosophy, Religion, Society, and Technology.

B Results on every domain

C Prompts for Data Construction

The appendix presents prompts for generating and filtering single-fact, multi-fact, and summary questions.

Related papers

Loading 2602.02053v2…