Source-linked AI summary
SelfGraphRAG: Bridging the Supervision Gap in Graph-Based RAG with Synthetic QA Generation
Ben Lagnese, Manas Gaur
TL;DR
Graph retrieval can model relationships in knowledge graphs but needs labeled QA data that newly constructed private graphs do not provide. SelfGraphRAG generates multi-hop and neighborhood QA pairs from graph structure to train a query-conditioned retriever, outperforming embedding-based baselines across evaluated benchmarks. The method is scoped to unlabeled corpora and has not been tested when training and query-time corpora diverge.
Problem
Graph retrieval requires labeled QA supervision aligned to the target graph, which is unavailable for automatically extracted knowledge graphs from private unlabeled corpora.
Method
SelfGraphRAG generates multi-hop and node-neighborhood QA pairs from the extracted graph and uses them to train a graph retrieval model for inference.
Results
SelfGraphRAG outperforms standard RAG and LightRAG across three benchmarks on precision, recall, and F1, including MultiHop-RAG F1 of 24.62 versus 2.60 for RAG and 0.98 for LightRAG.
Takeaways & Limitations
Synthetic QA generation from graph structure enables supervised graph retrieval without human annotation for unlabeled document corpora.
Takeaways & Limitations
SynthGen may bias the retriever toward graph-local reasoning, and the study does not test behavior when training and query-time corpora diverge.
Abstract
from arXiv · showhide
Retrieval-augmented generation (RAG) improves large language models by incorporating external knowledge without retraining, but existing methods often underuse the relational structure encoded in knowledge graphs. Graph-based RAG can capture entity relationships, yet supervised graph retrieval typically requires labeled question-answer data that may not be available for newly constructed graphs. We address this limitation with SelfGraphRAG, a framework that generates question-answer pairs directly from knowledge graph structure and uses them to train a query-conditioned graph retriever. The generated questions capture multi-hop paths and local neighborhoods, providing relational supervision without manual annotation. Experiments on multi-hop question answering and classification benchmarks show that SelfGraphRAG improves retrieval precision and downstream reasoning performance over embedding-based baselines. These results suggest that knowledge graph structure can provide useful supervision for training graph retrievers when labeled data are unavailable.
1 Introduction
RAG supplies external context to address limitations of parameterized LLM knowledge, but graph retrieval requires labeled QA data that private extracted graphs lack. SelfGraphRAG generates synthetic QA supervision from graph structure to train retrieval for multi-hop and neighborhood reasoning.
- RAG indexes external knowledge and supplies relevant context at inference time to improve accuracy and transparency.
- Graph retrieval models learn task-relevant subgraphs but require QA datasets labeled against the target graph.
- SelfGraphRAG generates multi-hop and node-summarization QA pairs from an extracted graph, then trains a graph retrieval model on them.
2 Related Work
Prior RAG systems use external retrieval, while graph-based variants exploit entity structure but commonly retain embedding-based matching. SelfGraphRAG instead uses synthetic graph-derived QA data to train a graph retriever without external supervision.
- Standard RAG retrieves document chunks through embedding similarity, which limits reasoning over entity relationships and corpus-wide synthesis.
- GraphRAG adds entity neighborhoods and community summaries, while LightRAG uses dual-level retrieval but still relies on embedding similarity for entity matching.
- Embedding-based retrieval can fail on multi-hop questions when relevant nodes are semantically distant from the query string.
- G-Retriever trains graph-transformer retrieval with labeled QA data, and SelfGraphRAG generates that training set from the extracted graph instead.
- SubgraphRAG and GNN-RAG require labeled QA data tied to the target graph, whereas SelfGraphRAG removes this shared dependency through synthetic QA generation.
- SelfGraphRAG extends synthetic supervision from graph extraction to downstream graph retrieval, following SynthKG’s self-supervised philosophy.
3 SelfGraphRAG
SelfGraphRAG targets unlabeled private document corpora by constructing a knowledge graph, generating synthetic QA supervision from graph structure, and training a query-conditioned graph retriever. Its pipeline combines multi-hop and neighborhood-based questions, while retaining limitations inherited from graph construction and unisolated design choices.
- Motivation: SelfGraphRAG addresses the absence of graph-grounded QA labels needed to train supervised graph retrievers on automatically constructed private knowledge graphs.The framework replaces unavailable manual annotation with synthetic supervision generated from the target graph.
- Doc2Graph: Doc2Graph converts documents into a knowledge graph whose nodes represent textual entity descriptions and whose directed labeled edges encode relations.The pipeline chunks documents before separately extracting and canonicalizing entity and relation mentions, reusing the GraphRAG indexing pipeline.
- GraphLM: GraphLM trains a query-conditioned retriever on the synthetic QA dataset while keeping the language model frozen, returning query-specific subgraphs for inference.Unlike independent embedding similarity, the retriever scores graph substructures jointly; deterministic verbalization makes retrieved-subgraph effects attributable to retrieval choices.
- Limitations: The method inherits a graph-construction limitation: independently canonicalized mentions across chunks may remain disconnected because no deduplication or coreference step is applied.The formulation also does not isolate the contributions of the two synthetic QA types or the G-Retriever architecture to downstream performance.
- SynthGen: SynthGen prompts an LLM to generate QA pairs from two-hop paths and fixed-size local neighborhoods, targeting complementary multi-hop retrieval and neighborhood evidence integration.The complete synthetic dataset is the union T = T_hop ∪ T_sum; two-hop enumeration is tractable because path length is bounded, while nodes with fewer than three in-neighbors are excluded from T_sum.
4 Experimental Setup
The evaluation uses three datasets chosen to test graph-based multi-hop retrieval across diverse question-answering and classification settings, and compares systems spanning distinct retrieval designs. Metrics are tailored to each task, including token-level QA measures and macro-averaged classification scores.
- Datasets: MoreHopQA requires reasoning across multiple documents and entity relationships, with yes/no, numerical, date, and named-entity answers.
- Datasets: MultiHop-RAG tests synthesis across multiple retrieved passages using constrained yes/no or named-entity answers.
- Datasets: PubMedQA evaluates whether graph retrieval generalizes to biomedical yes/no/maybe classification over research abstracts.
- Compared systems: The comparison spans flat chunk retrieval, community summarization, entity-level embedding retrieval, and trained graph retrieval.
- Evaluation metrics: MoreHopQA and MultiHop-RAG use token-level precision, recall, and F1, while PubMedQA reports per-class and macro-averaged classification metrics.
5 Results and Analysis
Across three benchmarks, SelfGraphRAG generally outperforms standard RAG and LightRAG, especially when retrieval must preserve multi-hop relations. Error analysis attributes gains to more precise subgraph selection, while also identifying metric and graph-extraction limitations.
- Overall results: SelfGraphRAG outperforms standard RAG and LightRAG across all three benchmarks on precision, recall, and F1.The comparison is reported across Tables 1–4.
- Overall results: 24.62 F1 on MultiHop-RAG exceeds RAG’s 2.60 and LightRAG’s 0.98, where multi-hop synthesis requires connecting entities across documents.SelfGraphRAG learns to follow two-hop relational paths despite limited lexical overlap.
- Retrieval precision and recall: 31.48 recall and 23.65 precision show that SelfGraphRAG retrieves subgraphs that are both more complete and more precise than LightRAG’s retrieval on MultiHop-RAG.LightRAG reaches 25.53 recall but only 0.50 precision, reflecting broad neighborhood retrieval and context noise.
- Domain-specific classification: 55.2% overall accuracy on PubMedQA exceeds RAG’s 51.6% and LightRAG’s 25.6%, with especially large gains on No and Maybe.SelfGraphRAG reaches 29.4 versus 0.0 accuracy on No and 36.6 versus 0.6 on Maybe against RAG.
- Evaluation caveats: Token-level F1 is the primary metric for the multi-hop datasets because exact-match accuracy penalizes paraphrases and partial answers across diverse answer types.PubMedQA’s class imbalance also makes aggregate comparisons sensitive to its Yes-heavy test distribution.
- Error analysis: SelfGraphRAG can preserve multi-step evidence chains, but it may still generate unsupported justifications after retrieving the correct answer.The method also inherits Doc2Graph’s lack of cross-chunk entity resolution, allowing extraction errors to propagate into training and inference.
6 Conclusion
SelfGraphRAG uses synthetic supervision from graph structure to train graph retrieval and outperforms embedding-based baselines on multi-hop reasoning and domain-specific classification. The paper identifies structured synthetic supervision as a broader route toward graph-based reasoning in LLM systems.
- SelfGraphRAG generates synthetic QA pairs from graph structure to enable supervised graph retrieval without human annotation.The framework targets automatically extracted knowledge graphs that lack labeled QA data.
- SelfGraphRAG outperforms embedding-based baselines on multi-hop reasoning and domain-specific classification.The reported comparison supports learned structural retrieval for complex queries.
- Future work includes improving graph construction fidelity, developing semantically aligned metrics, scaling to larger standard multi-hop benchmarks, and integrating retrieval more tightly with generation.
- Synthetic supervision derived from structured representations may help unlock graph-based reasoning in large language model systems.
7 Limitations
The limitations concern synthetic-data coverage and untested hyperparameters, entity-resolution gaps in graph construction, and evaluation without held-out document splits.
- Synthetic QA distributions may differ from real-world queries, biasing retrievers toward graph-local reasoning and limiting generalization to natural user queries.
- The two-hop and three-neighbor sampling hyperparameters were not ablated, so sensitivity to these choices remains untested.
- Doc2Graph performs no cross-chunk deduplication or coreference resolution, allowing duplicate entities to become disconnected nodes and relations to be lost.
- Training uses the same corpus graph as evaluation without a held-out document split, leaving corpus-divergence behavior untested.
8 Ethical Considerations
The ethical considerations cover privacy exposure when hosted models process private corpus content and compounded, unaudited bias across multiple LLM stages.
- Replacing locally hosted components with hosted or proprietary LLMs may transmit private document content and sensitive extracted entities beyond organizational boundaries.
- Bias can enter through entity and relation extraction, synthetic question generation, and answer phrasing across separate LLM calls.
- The work did not audit these biases, leaving their extent and downstream effects on retrieval or answer quality open.