Source-linked AI summary
KG-Agent: An Efficient Autonomous Agent Framework for Complex Reasoning over Knowledge Graph
Jinhao Jiang, Kun Zhou, Wayne Xin Zhao, Yang Song, Chen Zhu, Hengshu Zhu, Ji-Rong Wen
TL;DR
LLMs have limited capacity for complex reasoning from parametric knowledge alone, and existing KG-reasoning approaches can depend on predefined interaction plans or stronger closed-source models. KG-Agent uses synthesized code-based instructions and an autonomous tool-and-memory iteration process over KGs. With 10K samples, it mostly outperforms baselines based on full-data tuning or larger LLMs.
Problem
LLMs have limited capacity for complex, multi-hop and knowledge-intensive reasoning using parametric knowledge alone, while existing synergy approaches can use predefined plans and stronger closed-source models.
Source: §1 · p. 1Method
KG-Agent fine-tunes an LLM with synthesized code-based instructions and autonomously iterates tool selection and memory updates while reasoning over a KG.
Source: §1 · pp. 1–2, §4 · p. 3, §4.2.2 · p. 5Results
Takeaways & Limitations
KG-Agent targets autonomous decision-making during KG reasoning without human assistance, including with relatively small LLMs.
Source: §1 · p. 1The evaluation focuses on factual question answering over KGs and uses LLaMA2-7B; other comparable backbones, knowledge sources, and wider scenarios remain to be evaluated.
Source: Limitations · p. 9
Abstract
from arXiv · showhide
In this paper, we aim to improve the reasoning ability of large language models (LLMs) over knowledge graphs (KGs) to answer complex questions. Inspired by existing methods that design the interaction strategy between LLMs and KG, we propose an autonomous LLM-based agent framework, called KG-Agent, which enables a small LLM to actively make decisions until finishing the reasoning process over KGs. In KG-Agent, we integrate the LLM, multifunctional toolbox, KG-based executor, and knowledge memory, and develop an iteration mechanism that autonomously selects the tool then updates the memory for reasoning over KG. To guarantee the effectiveness, we leverage program language to formulate the multi-hop reasoning process over the KG, and synthesize a code-based instruction dataset to fine-tune the base LLM. Extensive experiments demonstrate that only using 10K samples for tuning LLaMA-7B can outperform state-of-the-art methods using larger LLMs or more data, on both in-domain and out-domain datasets. Our code and data will be publicly released.
1 Introduction
LLMs struggle with complex reasoning from parametric knowledge alone, while existing KG-agent approaches can rely on fixed interaction plans and stronger closed-source models. KG-Agent combines tools, code-based instruction tuning, and autonomous memory-updating decisions; experiments report gains with a smaller model and less training data.
1 Introduction
- Existing synergy-augmented methods often use predefined interaction plans and rely on stronger closed-source LLM APIs, which may not fit varied tasks or weaker models.Fixed plans can be ineffective when reasoning requirements vary.Source: §1 · p. 1
- KG-Agent combines a multifunctional toolbox, synthesized code-based instruction data, and autonomous tool selection with memory updates for KG reasoning.The toolbox supports discrete or advanced operations on KG data and intermediate results.Source: §1 · pp. 1–2
- The evaluation covers in-domain and out-of-domain KG-based and open-domain question-answering tasks.The reported comparisons include KGQA and ODQA.Source: §1 · p. 2
- 10K samples tune LLaMA-7B to outperform competitive baselines on in-domain datasets and achieve better zero-shot out-of-domain accuracy than full-data supervised models.Reported relative F1 improvements are 7.5% on CWQ and 2.7% on GrailQA; zero-shot relative accuracy improvements are 9.7% on WQ-Freebase and 8.5% on TQ-Wiki.Source: §1 · p. 2
2 Related Work
Prior work augments LLMs with KG information through retrieval or iterative interaction, while agent research includes both general-purpose and domain-specific systems. KG-Agent is presented as an autonomous approach to KG reasoning with a smaller LLM.
2 Related Work
- Retrieval-augmented methods serialize KG triples into prompts, while synergy-augmented methods interact with KGs iteratively; the latter can preserve structured search but often use predefined plans.Retrieval can lose KG structure and include redundant knowledge.Source: §2 · p. 2
- KG-Agent contrasts with fixed interaction methods by autonomously making decisions during KG reasoning without human assistance.This contrast addresses the limited flexibility attributed to predefined interaction mechanisms.Source: §2 · p. 2
- Many language-agent approaches rely on stronger closed-source APIs, whereas KG-Agent is presented as supporting complex KG reasoning with a relatively small 7B LLM.The related-work discussion names general-purpose agents and systems for web, multimodal, and real-life environments.Source: §2 · p. 2
3 Preliminary
The section defines a KG as a graph of factual triples and frames the task as using an available KG and toolbox to infer answer entities for a natural-language question.
3 Preliminary
- A knowledge graph consists of factual triples linking head and tail entities through relations, with entities assigned IDs or strings and types.The formal representation is G = {⟨e, r, e′⟩|e, e′ ∈ E, r ∈ R}.Source: §3 · p. 2
- The reasoning task assumes the KG contains the answer entities and that a toolbox is available to facilitate access to KG information.The agent is to infer the answer entities for a natural-language question using the toolbox and KG.Source: §3 · p. 3
- The formal objective is to use a question, toolbox, and KG to deduce the question’s final answer entities.The section also introduces neighboring relations as the incoming and outgoing relations associated with an entity set.Source: §3 · p. 3
4 Approach
KG-Agent combines tools for KG manipulation, code-based instruction tuning, and a memory-supported agent that autonomously selects tools to reason over knowledge graphs.
4.1 Toolbox for Knowledge Graph
- The toolbox supports KG reasoning through extraction, semantic, and logic tools for accessing, filtering, and operating on KG information.The tools include relation and entity retrieval, entity disambiguation, counting, set operations, and condition checks.Source: §4.1 · p. 3
4.2 KG-Agent Instruction Tuning
- KG-Agent’s instruction-tuning data is synthesized from KG reasoning programs generated using existing KGQA datasets.The programs are decomposed into steps, each formulated as instruction input and output.Source: §4.2 · p. 3, §4.2.2 · p. 4
4.2.1 KG Reasoning Program Generation
- KGQA datasets provide annotated SQL queries that are grounded on the KG, converted into reasoning chains and constraints, then decomposed into code snippets.The query graph is traversed from a mentioned entity using breadth-first search; the resulting chain can include constraints or numerical operations.Source: §4.2.1 · pp. 3–4
- The reasoning chain is translated into executable function calls that retrieve relations and entities in sequence, forming a KG reasoning program.Calls are generated to follow information from each entity to the next along the chain.Source: §4.2.1 · p. 4
4.2.2 KG Reasoning Instruction Synthesis
- Each program step becomes an instruction pair containing the question, toolbox, current KG information, and prior calls as input, with the next function call as output.After each call executes, the KG information and reasoning history are updated to construct the next pair in the complete trajectory.Source: §4.2.2 · pp. 4–5
4.3 Autonomous Reasoning over KG
- KG-Agent combines an instruction-tuned LLM, multifunctional toolbox, KG-based executor, and knowledge memory for autonomous reasoning over KGs.The planner selects tools and the executor runs their function calls.Source: §4.3 · p. 5
- KG-Agent iterates tool selection and memory updates until it reaches answer entities, with the LLM planner and KG executor coordinating the process.Memory retains the question, toolbox definition, current KG information, and reasoning history.Source: §4.3 · pp. 5–6
4.4 Comparison to Previous Work
- KG-Agent combines autonomous tool-and-memory interaction with instruction tuning of a smaller 7B open-source LLM, unlike methods with predefined workflows or reliance on closed-source APIs.Its tuning data is constructed from multiple KGs, including Wikidata and Freebase.Source: §4.4 · p. 6
5 Experiment
KG-Agent is evaluated across in-domain and out-of-domain question answering, with further tests of transfer and instruction-data choices. Results favor the framework across datasets, while the analyses show that performance gains depend on data amount and sampling balance.
5.1 Experimental Setup
- The evaluation covers four in-domain KGQA datasets—WebQSP, CWQ, GrailQA, and KQA Pro—and three out-of-domain datasets: WQ, NQ, and TQ.In-domain datasets use Freebase or Wikidata; out-of-domain datasets are used for out-of-domain evaluation.Source: §5.1 · p. 6
5.2 Main Results
- F1 improves by 1.7% on WebQSP, 7.5% on CWQ, and 2.7% on GrailQA; KG-Agent outperforms the other competitive baselines on all datasets.The paper attributes these results to mutual augmentation between datasets and reports consistent improvements across two KGs.Source: §5.2 · p. 7
- In zero-shot out-of-domain evaluation, KG-Agent achieves consistent improvement over fine-tuned pre-trained language models by learning to interact with the KG rather than memorize specific knowledge.The evaluation covers out-of-domain datasets constructed from Wikipedia and Freebase.Source: §5.2 · p. 7
5.3 Further Analysis
- On the three MetaQA subsets, KG-Agent consistently improves on competitive supervised fine-tuning baselines, supporting transfer to a movie-domain KG.The one-shot test uses a domain-specific movie knowledge graph.Source: Table 5 · p. 7, §5.3 · p. 8
- F1 and Hits@1 rise as instruction data increases from 2k to 64k, but gains are not remarkable from 16k to 64k.The authors relate the plateau to limited instruction-data variety and propose constructing more diverse samples.Source: §5.3 · p. 8, Figure 2 · p. 8
- Increasing one dataset’s sampling proportion improves performance on that dataset, but each tested variant lowers the average across all three datasets relative to the selected proportions.The authors consider their chosen proportions suitable for balancing more comprehensive and general abilities.Source: §5.3 · p. 8, Table 6 · p. 8
6 Conclusion
KG-Agent combines KG tools, an autonomous tool-selection and memory-update loop, and code-based instruction tuning to support complex KG reasoning with a 7B LLM.
6 Conclusion
- KG-Agent integrates an LLM, a multifunctional KG toolbox, a KG-based executor, and knowledge memory in an autonomous reasoning framework.The toolbox supports typical KG operations, and the iteration mechanism selects tools and updates memory.Source: §6 · p. 8
- The method synthesizes code-based instruction-tuning data from existing KGQA datasets and uses 10K samples to tune a 7B LLM.The paper reports that the resulting agent mostly outperforms state-of-the-art baselines using full-data tuning or larger LLMs.Source: §6 · p. 8
- Future work will extend the framework to other structured data, including databases and tables.Source: §6 · p. 8
Limitations
KG-Agent’s evaluation and scope are limited to LLaMA2-7B and factual question answering over knowledge graphs. The authors identify broader knowledge sources, task types, and response filtering as areas for future work.
Limitations
- Only LLaMA2-7B is evaluated, leaving performance with comparable-size backbones such as Mistral-7B or CodeLLaMA-7b untested.The authors note that LLaMA2-7B has strong capability after instruction tuning.Source: Limitations · p. 9
- The framework is evaluated only on factual question answering over KGs, with databases or tables and tasks such as data-to-text left for future evaluation.Source: Limitations · p. 9
- The authors say additional rule-based post-processing should filter illegal responses, despite tuning the LLM to answer from KG information and avoid risky or discriminatory responses.Source: Limitations · p. 9
A.1 Datasets
The evaluation uses KGQA datasets spanning Freebase and Wikidata, out-of-domain QA subsets linked to KG entities, and a movie-domain KG dataset. The datasets vary in hop depth, generalization focus, and reasoning requirements.
A.1 Datasets
- The study selects WebQSP, CWQ, and GrailQA on Freebase, KQA Pro on Wikidata, three KG-linked ODQA subsets, and MetaQA on a movie-domain KG.Source: §A.1 · p. 13
- ODQA evaluation filters out questions not linkable to a KG entity, producing WQ-Freebase, NQ-Wiki, and TQ-Wiki subsets.Source: §A.1 · p. 13
- WebQSP answers are within two hops, while CWQ extends WebQSP with additional entities or answer constraints and allows up to four hops.Source: §A.1 · p. 13
- GrailQA evaluates i.i.d., compositional, and zero-shot generalization, whereas KQA Pro requires capabilities including multi-hop, quantitative comparison, and set operations.Source: §A.1 · p. 13
- MetaQA contains movie-KG questions with answers up to three hops away and is divided into one-, two-, and three-hop subsets; training samples one case per question template.Source: §A.1 · p. 13
A.2 Evaluation Protocol
The evaluation protocol specifies answer-ranking and answer-coverage metrics for KGQA, plus exact match for ODQA. Baselines include graph reasoning, sequence-to-sequence query generation, and language-model approaches.
A.2 Evaluation Protocol
- Hits@1 and F1 are used for WebQSP and CWQ, F1 for GrailQA, Hits@1 for MetaQA, and EM for ODQA.Hits@1 measures the top-ranked answer, F1 answer coverage, and EM normalized exact answer match.Source: §A.2 · p. 13
- For unordered answer sets, Hits@1 is averaged over 100 repetitions that randomly select one answer per question.Source: §A.2 · p. 13
A.3 Baselines for Comparison
- KGQA comparisons include subgraph reasoning, seq2seq SPARQL generation, and LLM-based methods, while closed-book ODQA comparisons include fine-tuned models and zero-shot LLMs.Source: §A.3 · pp. 13–14
- ODQA baselines comprise fine-tuned T5 and BART models and zero-shot GPT-3.5 variants.Source: §A.3 · p. 14
A.4 Implementation Details
The implementation fine-tunes LLaMA2-7B on 10,000 sampled examples and evaluates it across in-domain, out-of-domain, and domain-specific settings. The toolbox appendix points to tables listing prompt templates and tool definitions.
A.4 Implementation Details
- Instruction tuning uses 10,000 examples sampled from WebQSP, KQA Pro, GrailQA, and CWQ in a 1:5:5:10 ratio, with LLaMA2-7B trained for three epochs.The setup uses an initial learning rate of 2e-5, weight decay 0.1, batch size 256, and maximum length 1500.Source: §A.4 · p. 14
- Evaluation covers in-domain test or development sets, zero-shot ODQA subsets, and one-shot fine-tuning on MetaQA; API baselines use February versions and six demonstrations for in-domain questions.Source: §A.4 · p. 14
B Summary of Toolbox
- Table 7 lists prompts used for ChatGPT, Davinci-003, and GPT-4, while Table 8 provides tool definitions and usage details.Source: Table 7 · p. 15, Table 8 · p. 16