Source-linked AI summary
Graph Counselor: Adaptive Graph Exploration via Multi-Agent Synergy to Enhance LLM Reasoning
Junqi Gao, Xiang Zou, YIng Ai, Dong Li, Yichen Niu, Biqing Qi, Jianxing Liu
TL;DR
Existing GraphRAG methods can miss multi-level graph information and use preset reasoning patterns that do not adapt depth or correct semantic misalignment. Graph Counselor combines adaptive three-agent graph extraction with self-reflection and reverse reasoning. The paper reports improved graph-based reasoning accuracy and validation across six backbone LLMs.
Problem
Existing GraphRAG methods may rely on fixed single-agent extraction and reasoning patterns, limiting multi-level graph information capture and adaptive reasoning depth.
Source: §1 · p. 2Method
Graph Counselor combines Planning, Thought, and Execution agents in AGIEM with self-reflection, reverse reasoning, and multiple perspectives to refine graph reasoning.
Source: §1 · pp. 2–3Results
Takeaways & Limitations
The paper’s multi-round Plan, Thought, Execution, and Reflection paradigm was tested on six backbone LLMs.
Source: §4 · p. 9The authors observed that reflection-model size might influence effectiveness but did not analyze this relationship further.
Source: Limitations · p. 9
Abstract
from arXiv · showhide
Graph Retrieval Augmented Generation (GraphRAG) effectively enhances external knowledge integration capabilities by explicitly modeling knowledge relationships, thereby improving the factual accuracy and generation quality of Large Language Models (LLMs) in specialized domains. However, existing methods suffer from two inherent limitations: 1) Inefficient Information Aggregation: They rely on a single agent and fixed iterative patterns, making it difficult to adaptively capture multi-level textual, structural, and degree information within graph data. 2) Rigid Reasoning Mechanism: They employ preset reasoning schemes, which cannot dynamically adjust reasoning depth nor achieve precise semantic correction. To overcome these limitations, we propose Graph Counselor, an GraphRAG method based on multi-agent collaboration. This method uses the Adaptive Graph Information Extraction Module (AGIEM), where Planning, Thought, and Execution Agents work together to precisely model complex graph structures and dynamically adjust information extraction strategies, addressing the challenges of multi-level dependency modeling and adaptive reasoning depth. Additionally, the Self-Reflection with Multiple Perspectives (SR) module improves the accuracy and semantic consistency of reasoning results through self-reflection and backward reasoning mechanisms. Experiments demonstrate that Graph Counselor outperforms existing methods in multiple graph reasoning tasks, exhibiting higher reasoning accuracy and generalization ability. Our code is available at https://github.com/gjq100/Graph-Counselor.git.
1 Introduction
GraphRAG models knowledge relationships to support factual, multi-hop reasoning, but existing approaches can lose graph structure and lack adaptive information aggregation and reasoning depth. Graph Counselor addresses these challenges with three collaborating agents and self-reflection, and its experiments report improved reasoning accuracy and generalization.
1 Introduction
- Linearizing retrieved graph information as text or code can lose critical structural information and weaken reasoning performance.Source: §1 · p. 2
- Existing GraphRAG methods can use fixed single-agent extraction and reasoning patterns, limiting multi-level graph information capture and adjustment of reasoning depth to task complexity.The introduction also identifies semantic misalignment that can bias retrieval paths and introduce erroneous information.Source: §1 · p. 2
- Graph Counselor’s AGIEM assigns Planning, Thought, and Execution agents to structure reasoning, select needed graph knowledge, and adapt retrieval strategies across iterations.The roles are intended to preserve graph structure and interdependencies while improving reasoning precision.Source: §1 · p. 2
- SR evaluates reasoning paths and answers for consistency, then uses self-reflection, reverse reasoning, and multiple perspectives to refine context and reasoning outcomes.The introduction presents SR as a way to address misalignment between graph structure and semantic content.Source: §1 · pp. 2–3
- Experiments on various benchmarks show that Graph Counselor outperforms existing methods in reasoning accuracy and generalization.Source: §1 · p. 3
2 Graph Counselor
Graph Counselor combines hierarchical graph-information extraction with multi-perspective reflection in an iterative workflow. Its agents adapt graph queries and revise reasoning when reflection identifies issues.
2.1 Overview
- Graph Counselor combines AGIEM’s hierarchical graph extraction with SR’s memory-based reflection to support complex KGQA reasoning.AGIEM uses planning, reasoning, and execution agents; SR guides reflection across multiple perspectives.Source: §2.1 · p. 3
2.2 Adaptive Graph Information Extraction Module
- Planning identifies relevant information and reasoning paths, Thought specifies needed graph information or infers an answer, and Execution adaptively extracts graph structures.Planning and Thought clarify the graph information required for multi-step reasoning.Source: §2.2 · p. 3
- The Execution Agent can combine Retrieve, Feature, Neighbor, and Degree components in parallel or in series to extract complex graph information.Degree returns the count of neighbors connected by a specified relation.Source: §2.2 · p. 4
2.3 Self-Reflection with Multiple Perspectives
- SR uses divergent, multi-perspective reflection to detect and correct reasoning biases and keep graph structure aligned with query semantics.It analyzes omissions, redundancies, and inconsistencies, then adjusts the reasoning strategy for subsequent steps.Source: §2.3 · pp. 4–5
- SR’s reflection proceeds through recap and understanding, analysis and adjustment, then refinement and update.These stages revisit the current query and graph extraction, address misalignments, and refine subsequent reasoning.Source: §2.3 · pp. 4–5
2.4 LLM State Transition Mechanism and Workflow
- AGIEM iterates through planning, thought, and execution, while an outer reflection loop reruns it when a correctness flag is false and the reflection limit remains.Reflection results update the inner reasoning context; the loop ends when the result is judged true or the limit is reached.Source: §2.4 · p. 5
3 Experiments
Across six LLM backbones, Graph Counselor is evaluated on GRBENCH and WebQSP, with ablations testing its agents and reflection settings. The experiments report stronger reasoning performance and a performance–cost trade-off against Graph-CoT.
3.1 Experimental Setup
- On the 10-graph, five-domain GRBENCH dataset, experiments compare Graph Counselor with LLM-only, Text RAG, GraphRAG, and Graph-CoT baselines across six LLM backbones.GRBENCH contains 1,740 questions divided into simple, medium, and hard difficulty levels.Source: §3.1 · pp. 5–6
3.2 Overall Performance
- Graph Counselor improves Rouge-L by up to 24.2% over the state-of-the-art GraphRAG approach on GRBENCH; 2-hop retrieval is not always better than 1-hop.GraphRAG also outperforms Text RAG overall, though the advantage is less pronounced on Legal, where query context can help retrieve relevant text directly.Source: Table 1 · p. 6, §3.2 · pp. 6–7
3.3 Additional Comparative Experiment
- On WebQSP, Mistral-NeMo-2407 with Graph Counselor improves QS by 12.50%, LS by 11.00%, and RL by 8.99% over the second-best method, Graph-CoT.These results are reported as evidence of generality on an additional KGQA dataset.Source: Table 2 · p. 7, §3.3 · p. 7
3.4.1 How important is Plan and Complex Graph Information?
- On GRBENCH, removing Plan reduces medium- and high-difficulty accuracy by up to 6.1%, while limiting Complex to one component at a time reduces it by up to 3.6%.The authors attribute the Planning Agent's role to task decomposition and inference-path planning, and the Execution Agent's role to adapting graph-information extraction and integration.Source: §3.4.1 · p. 7, Figure 3 · p. 8
3.4.2 How important is Reflect?
- Removing SR causes an overall accuracy drop of up to 7.26%, while two reflection iterations yield the largest gain before improvements slow.The study uses two iterations in its other experiments as a performance–cost trade-off.Source: §3.4.2 · pp. 7–8
- Graph Counselor uses two reflection iterations in other experiments because the largest gain occurs at two, with slower improvement afterward.The paper also finds no statistically significant reflection-performance difference between large and small models in its sampled comparison.Source: §3.4.2 · p. 8
- In a comparison using 500 sampled reasoning outputs, reflection performance differences between large and small models were not statistically significant.The authors consequently report model size was not a decisive factor in these reflection tasks.Source: §3.4.2 · p. 8
3.5 Trade-off between time and performance
- A 9B Gemma-2 model with Graph Counselor exceeds 70B Graph-CoT by more than 10% on E-commerce, at 13.71% of Graph-CoT's reasoning cost.The authors note that Graph Counselor still requires more absolute reasoning time than Graph-CoT.Source: §3.5 · p. 9, Table 3 · p. 9
4 Conclusion
The conclusion presents Graph Counselor as a four-step interactive approach to adaptive graph extraction and self-reflection. Experiments validate its performance across six backbone LLMs, while future work could address efficiency, interpretability, dynamic graph updates, and multimodal representation.
4 Conclusion
- Graph Counselor organizes its multi-round interactive reasoning into Plan, Thought, Execution, and Reflection to support adaptive graph knowledge extraction.The conclusion frames the work around complex graph reasoning and misalignment between graph structures and semantic information.Source: §4 · p. 9
- Experiments test and validate Graph Counselor's performance on six backbone LLMs.Source: §4 · p. 9
- Future work could optimize interactive iteration efficiency and interpretability, and investigate dynamic graph updating and multimodal knowledge representation.The conclusion identifies these directions as potential ways to enhance reasoning generalization in open-domain scenarios.Source: §4 · p. 9
Limitations
The authors leave the possible effect of reflection-model size on Graph Counselor unresolved.
Limitations
- The effect of reflection-model size on Graph Counselor may matter, but the authors did not investigate it further because it was outside the paper's core objective.Source: Limitations · p. 9
A Graph Counselor Flow
The flow takes a query and graph through planning, thought, and execution, updating context as it iterates. It regularizes a completed answer and can invoke reflection when the answer is incorrect, subject to a reflection limit.
A Graph Counselor Flow
- The flow takes a query, graph, iteration limits, knowledge extractor, agents, and initial context as inputs, and returns a final answer.Source: A Graph Counselor Flow · p. 12
- Thought clarifies needed graph knowledge, and the flow updates context from planning, thought, execution, and observations when continuing its loop.Source: A Graph Counselor Flow · p. 12
- When a finish condition is met, the flow regularizes the answer and checks correctness; an incorrect answer can trigger reflective summarization and context updating, within the reflection limit.Source: A Graph Counselor Flow · p. 12
B Additional Performance
Table 4 reports model performance percentages using Rouge-L and LlamaScore.
B Additional Performance
- Table 4 compares model performance percentages using Rouge-L (RL) and LlamaScore (LS).The supplied caption identifies the metrics but provides no cell values or comparison outcomes.Source: Table 4 · p. 13
C Case Studies of Graph Counselor
The case studies illustrate reflection correcting reasoning errors and also document cases where graph-structure assumptions led to unsuccessful answers.
C Case Studies of Graph Counselor
- Reflection identifies reasoning errors and improves the reasoning process to produce correct answers in the Amazon and Biomedical examples.The examples show corrections after an intermediate reflection process.Source: C Case Studies of Graph Counselor · p. 14
- The revised Nokia-item reasoning excludes the original item and returns the correct count of 48.An earlier attempt counted 49, while the corrected trace filters out the original item before counting.Source: C Case Studies of Graph Counselor · pp. 14–15
- The Fluocinolone Acetonide example identifies atopic dermatitis as the ear-localized treated disease after checking disease and anatomy relations.The reflection-guided example reports the answer as atopic dermatitis.Source: C Case Studies of Graph Counselor · p. 16
- The Alice B. Cooper attempt fails because it assumes collaborators are direct author neighbors, while collaboration is mediated by shared paper nodes.The proposed corrected strategy counts co-authors across Alice B. Cooper’s paper neighbors.Source: C Case Studies of Graph Counselor · p. 17
D.1 Model Reasoning Prompts
The reasoning prompt structures graph question answering as alternating planning, thought, graph interaction, and feedback, ending with a finish action.
D.1 Model Reasoning Prompts
- The prompt defines Retrieve, Feature, Degree, and Neighbour graph functions, including their node, feature, relation, or keyword inputs and outputs.It also permits compound functions such as Feature[Retrieve[keyword], feature].Source: §D.1 · p. 17
- The prompt requires one Plan after an observation or when no plan exists, then one Thought after a plan and one Action after a thought.When ready to stop, the model uses Finish and does not produce another Observation.Source: §D.1 · pp. 17–18
- Each question is presented with a request for node main features, such as names, rather than node IDs.The prompt also reserves placeholders for the graph definition and examples.Source: §D.1 · pp. 17–18
D.2 Model Reflection Prompts
The reflection prompt guides models to review a prior trial, diagnose reasoning and graph-selection issues, then revise their understanding and strategy.
D.2 Model Reflection Prompts
- The reflection template includes a recap of the trial with fields for the question and graph structural information used.It is framed as reflection on a prior reasoning trial to improve future question answering.Source: §D.2 · p. 18
- The prompt asks the model to identify the question’s core goal, missed information, and potential misunderstandings.It also prompts analysis of selected graph information, missed insights, and redundancies.Source: §D.2 · p. 18
- The guided reflection asks whether problem understanding and graph-function choices align, what inconsistencies occurred, and how to adjust them.It separately requests an improved strategy for the next attempt.Source: §D.2 · p. 18
- The prompt requests an updated problem understanding and revised graph selection, allowing compound functions with no more than two functions at each step.The revised strategy should explain how it addresses prior issues.Source: §D.2 · p. 18
D.3 Model Evaluation Prompts
The prompt specifies an interleaved graph question-answering workflow and a separate final-answer check. It requires answers to name node features rather than IDs and to end with an explained correctness judgment.
D.3 Model Evaluation Prompts
- The evaluation agent checks whether the final answer matches retrieved graph data, meets the question requirements, avoids irrelevant or incorrect content, and follows logically from observations.It then provides a brief explanation and concludes with [yes] or [no].Source: §D.3 · p. 19
- The workflow interleaves planning, thought, graph interaction, and feedback, with the next step constrained by the preceding step and a Finish action ending the process.The prompt also says not to make an Observation and to provide only one next Plan when the last Observation has been given or there is no Plan.Source: §D.3 · p. 19
- Graph interaction offers Retrieve, Feature, Degree, and Neighbour functions, including compound calls that apply a feature lookup to a retrieved node.The graph definition and question are supplied as inputs to the task.Source: §D.3 · p. 19
- Answers must provide a node’s main feature, such as its name, rather than its ID.The prompt includes the question and scratchpad as input.Source: §D.3 · p. 19