Source-linked AI summary
Agent-Enhanced Heterogeneous Graph RAG for Academic Question Answering
Runsong Jia, Mengjia Wu, Ying Ding, Jie Lu, Yi Zhang
TL;DR
Academic QA over heterogeneous scholarly graphs is challenged by retrieval strategies that ignore query complexity, incomplete evidence control, and limited graph-grounded verification. The paper introduces an agentic RAG pipeline with query-aware planning, sufficiency-aware evidence expansion, and answer verification, and reports consistent improvements over LLM, graph-RAG, and agent-based baselines on OpenAlex and DBLP.
Problem
Existing graph-augmented RAG methods rarely adapt traversal to query complexity, control evidence sufficiency, or verify generated answers against graph facts.
Method
The method agenticizes RAG through query-aware retrieval planning, sufficiency-aware evidence control, and graph-grounded verification of entities, relations, attributes, and completeness.
Results
The method consistently improves performance over pure LLM, graph-augmented RAG, and agent-based retrieval baselines on heterogeneous graphs from OpenAlex and DBLP.
Takeaways & Limitations
Explicitly aligning retrieval and verification with heterogeneous scholarly graph structure improves answer selection and overall Academic QA performance.
Abstract
from arXiv · showhide
Academic question answering requires reasoning over heterogeneous scholarly graphs, where queries range from simple attribute lookups to multi-hop inference across author--paper--venue structures. Existing retrieval-augmented generation (RAG) systems struggle in this setting due to three limitations: (1) fixed retrieval strategies that do not adapt to varying query complexity, (2) the absence of sufficiency evaluation leading to incomplete or misaligned evidence, and (3) a lack of structured verification against graph facts. To address these issues, we propose an agentic heterogeneous graph RAG method that transforms the three core stages of the RAG pipeline into explicit agentic decision steps. A query-aware retrieval agent analyzes query type and selects an appropriate graph traversal strategy; a sufficiency-aware reranking agent assesses evidence completeness and adaptively expands the retrieved subgraph; and a graph-grounded verification agent checks entity, relation, and attribute correctness before finalizing the answer. Experiments on heterogeneous graphs constructed from OpenAlex and DBLP suggest that our method consistently outperforms strong LLM, graph-augmented RAG, and agent-based baselines.
1 Introduction
Academic QA over heterogeneous scholarly graphs remains limited by rigid retrieval, insufficient evidence control, and missing graph-grounded verification. The paper proposes an agentic pipeline that adapts retrieval, expands evidence when needed, and validates answers against graph structure.
- Existing graph-augmented RAG methods apply uniform traversal depth across simple attribute and multi-hop reasoning queries.
- They typically retrieve one-shot subgraphs without checking whether entities, constraints, or aggregation signals required by the query are covered.
- The proposed pipeline assigns retrieval planning, evidence control, and answer verification to dedicated agents that leverage heterogeneous scholarly graph structure.
- The verification agent checks entity existence, relation validity, attribute correctness, and answer completeness before returning the final answer.
- Experiments on OpenAlex and DBLP report consistent improvements from integrating query-aware planning, adaptive retrieval, and graph-grounded verification.
- The retrieval method classifies questions into four academic query types and adaptively guides retrieval and reranking according to query structure.
2 Related Works
The related work situates the paper at the intersection of heterogeneous scholarly graphs, graph-augmented RAG, agentic language models, and answer verification. Prior approaches provide structured representations, graph evidence, tool-based planning, or reliability assessment that motivate the proposed combination.
- Scholarly data linking authors, papers, venues, and citations naturally forms heterogeneous graphs used in recommendation, citation analysis, and structured QA.
- Heterogeneous Graph Transformer learns type-aware representations by aggregating information across different node and edge types.
- Graph- and knowledge-graph-augmented RAG systems incorporate traversal into retrieval so structured relations can serve as evidence for generation.
- G-Retriever retrieves subgraphs for textual graph understanding and integrates graph foundation models into RAG to improve retrieval over structured relations.
- Agentic LLM frameworks support tool invocation, action planning, and coordinated subtasks, while self-consistency and FActScore assess generated-answer reliability.
3 Methodology
The method decomposes academic question answering over heterogeneous scholarly graphs into query-aware retrieval planning, sufficiency-aware evidence control, and graph-grounded answer verification. These agents select traversal strategies, expand inadequate evidence, and check generated answers against graph facts.
- Academic Heterogeneous Graph: The scholarly graph contains Author, Paper, and Venue nodes connected by authorship and publication-venue relations.Authors store names and organizations; Papers store titles, abstracts, keywords, citation counts, and years; Venues store names and ranks.
- Query-Aware Retrieval Agent: Queries are classified as Attribute, Direct Relation, Aggregation, or Multi-hop, with different start-node and traversal-depth requirements.The categories correspond to 0-hop, 1-hop, 1–2-hop numerical reasoning, and 2+-hop relational chains, respectively.
- Query-Aware Retrieval Agent: The Retrieval Agent predicts query type, start-node type, hop budget, entity mentions, and a traversal plan to construct a structured retrieval plan.The plan supports node lookup, single-hop, multi-hop, and aggregation traversal strategies and is implemented with a prompted GPT-4-turbo classifier without supervised fine-tuning.
- Graph Retrieval and Sufficiency-Aware Reranking Agent: Entity lookup uses text-encoder embeddings to select a start node, after which traversal follows the planned hop budget and strategy.The initial subgraph is formed by constrained traversal from the selected node; experiments cap retrieval at two expansions to control latency and compactness.
- Graph Retrieval and Sufficiency-Aware Reranking Agent: The Reranking Agent evaluates entity or attribute coverage and semantic relevance, triggering controlled expansion when sufficiency falls below a threshold.Coverage measures matched query entity mentions, while relevance averages similarity between the query and textual descriptions of subgraph nodes.
- Answer Generation and Graph-Grounded Validation Agent: The Validation Agent checks generated answers for entity existence, relation validity, and attribute correctness against the retrieved subgraph.Answers below the consistency threshold are regenerated up to two times or accompanied by an uncertainty signal, helping detect unsupported or hallucinated content.
4 Experimental Setup
Experiments evaluate the method on heterogeneous academic graphs derived from OpenAlex and DBLP, using four query categories, three answer-quality metrics, and diverse LLM, graph-RAG, and agent-based baselines.
- Datasets: The evaluation uses OpenAlex and DBLP graphs with Authors, Papers, and Venues linked by writes and published_in relations.OpenAlex contains 76,569 nodes and 105,290 edges; DBLP contains 62,443 nodes and 79,697 edges.
- Datasets: Each dataset contributes 400 queries, evenly divided across attribute, direct-relation, aggregation, and multi-hop types.The query sets contain 100 queries per type with automatically extracted ground-truth answers.
- Implementation Details: The system uses GPT-4-turbo for agentic controllers and answer generation, sentence-transformers for entity lookup, and caps retrieval expansions and regenerations at two each.Accuracy, entity-level F1, and Hit@1 are reported as evaluation metrics.
- Baselines: Baselines span pure LLMs, graph-augmented RAG systems, and agent-based or complexity-aware retrieval methods.The comparisons include Qwen-2.5-7B, GPT-o3, Vanilla RAG, GraphPrompter, GraphRAG, KGRAG, GraphCoT, AdaptiveRAG, and Agent-G.
5 Results
The method achieves the strongest overall performance on OpenAlex and DBLP, while ablations show that retrieval planning, adaptive reranking, and verification each contribute to robustness.
- 76.68% accuracy on OpenAlex and 73.43% accuracy on DBLP mark the method’s strongest performance across both datasets.
- The method outperforms pure LLM, graph-augmented RAG, and agent-based baselines across the reported evaluation settings.
- Removing the Retrieval Agent causes the largest performance drop, highlighting query-type analysis and hop-budget planning.
- The Reranking Agent contributes substantially by adaptively expanding evidence when the initial subgraph is insufficient.
- The Verification Agent improves F1 and Hit@1 by checking entity, relation, and attribute correctness against the retrieved subgraph.
- The full agentic pipeline provides the most robust performance because each agent contributes meaningfully to the method.
6 Conclusion
The paper concludes that agentic heterogeneous graph RAG combines query-aware planning, sufficiency-aware evidence control, and graph-grounded verification for Academic QA, improving performance on OpenAlex and DBLP.
- The method incorporates query-aware retrieval planning, sufficiency-aware evidence control, and graph-grounded answer verification.
- Experiments on OpenAlex and DBLP demonstrate consistent improvements over pure LLM, graph-augmented RAG, and agent-based retrieval methods.