Source-linked AI summary
LLMs for Knowledge Graph Construction and Reasoning: Recent Capabilities and Future Opportunities
Yuqi Zhu, Xiaohan Wang, Jing Chen, Shuofei Qiao, Yixin Ou, Yunzhi Yao, Shumin Deng, Huajun Chen, Ningyu Zhang
TL;DR
This paper investigates whether LLM performance in KG construction and reasoning reflects memorized pre-training knowledge or contextual learning and generalization. It evaluates LLMs across zero-shot and one-shot settings, introduces Virtual Knowledge Extraction and VINE, and proposes multi-agent AutoKG; GPT-4 extracts knowledge across domains but does not surpass fine-tuned models, while the study highlights robust contextual learning.
Problem
The paper examines whether LLMs’ extraction abilities arise from vast pre-training corpora or strong contextual learning and generalization capabilities.
Method
The study evaluates ChatGPT and GPT-4 in zero-shot and one-shot KG construction and reasoning tasks across eight benchmark datasets, introduces VINE for Virtual Knowledge Extraction, and proposes multi-agent AutoKG.
Results
GPT-4 extracts knowledge across diverse domains but does not surpass fine-tuned models, while Virtual Knowledge Extraction results highlight robust contextual learning.
Takeaways & Limitations
LLMs can assist KG construction and reasoning, with AutoKG using multiple agents to support these tasks through iterative interactions.
Takeaways & Limitations
AutoKG’s constructed knowledge graphs still require manual evaluation and validation, while token limits, inefficient human-machine interaction, and hallucinations remain challenges.
Abstract
from arXiv · showhide
This paper presents an exhaustive quantitative and qualitative evaluation of Large Language Models (LLMs) for Knowledge Graph (KG) construction and reasoning. We engage in experiments across eight diverse datasets, focusing on four representative tasks encompassing entity and relation extraction, event extraction, link prediction, and question-answering, thereby thoroughly exploring LLMs' performance in the domain of construction and inference. Empirically, our findings suggest that LLMs, represented by GPT-4, are more suited as inference assistants rather than few-shot information extractors. Specifically, while GPT-4 exhibits good performance in tasks related to KG construction, it excels further in reasoning tasks, surpassing fine-tuned models in certain cases. Moreover, our investigation extends to the potential generalization ability of LLMs for information extraction, leading to the proposition of a Virtual Knowledge Extraction task and the development of the corresponding VINE dataset. Based on these empirical findings, we further propose AutoKG, a multi-agent-based approach employing LLMs and external sources for KG construction and reasoning. We anticipate that this research can provide invaluable insights for future undertakings in the field of knowledge graphs. The code and datasets are in https://github.com/zjunlp/AutoKG.
1 Introduction
The paper evaluates LLMs for KG construction and reasoning, examines their generalization through Virtual Knowledge Extraction, and proposes AutoKG for multi-agent assistance. It frames LLMs as adaptable tools whose effectiveness depends on task demands, prompting, and external resources.
- Basic Evaluation: The study evaluates text-davinci-003, ChatGPT, and GPT-4 in zero-shot and one-shot settings against fully supervised state-of-the-art benchmarks.The evaluation covers KG construction and reasoning tasks across eight benchmark datasets.
- Recent Capabilities: Entity, relation, and event extraction support KG construction, while link prediction and question answering support KG reasoning and application.These tasks organize structured knowledge, uncover latent relationships, and enable inference from question context.
- Generalizability Analysis: Virtual Knowledge Extraction tests whether LLMs can generalize to unfamiliar knowledge rather than relying only on memorized pre-training information.The VINE dataset replaces entities and relations in Re-TACRED with unseen ones to create virtual knowledge scenarios.
- Generalizability Analysis: Experiments on VINE indicate that GPT-4 can acquire new knowledge from instructions and execute extraction tasks.This provides evidence for robust contextual learning in the studied setting.
- Future Opportunities: LLMs offer adaptability in novel or data-scarce settings but depend strongly on prompt engineering and remain constrained by knowledge cutoffs.These limitations motivate interactive access to external resources.
- Future Opportunities: AutoKG proposes multi-agent communication, external sources, and iterative dialogues to assist autonomous KG construction and reasoning.The framework reduces the human role while using multiple agents with distinct roles.
2 Recent Capabilities of LLMs for KG Construction and Reasoning
Across KG construction and reasoning tasks, GPT-4 generally improves over ChatGPT and shows stronger reasoning than construction performance, while remaining below fine-tuned models for extraction. Results vary with prompting, task complexity, domain specialization, and evaluation design.
- Entity and Relation Extraction: GPT-4’s zero-shot extraction performance significantly improves across DuIE2.0, Re-TACRED, and SciERC, including 31.03 versus ChatGPT’s 10.3 on DuIE2.0.On Re-TACRED, GPT-4 correctly identifies “org:alternate names” where ChatGPT fails.
- Event Extraction: GPT-4 outperforms ChatGPT in zero-shot event extraction by identifying multiple event types from complex context, but one-shot prompting slightly reduces GPT-4’s performance.Under one-shot evaluation, GPT-4 produces more erroneous responses when it misses the correct event types.
- Link Prediction: One-shot demonstrations improve GPT-4’s link prediction results on FB15k-237 and ATOMIC2020 by supporting accurate tail-entity prediction.A demonstration enables GPT-4 to identify Primetime Emmy Award after zero-shot prediction fails.
- Question Answering: On FreebaseQA, ChatGPT and GPT-4 match each other and surpass fully supervised SOTA by 16%, while GPT-4 exceeds ChatGPT by 11.1 points on MetaQA.MetaQA remains below supervised SOTA, with multi-answer questions and input-token constraints identified as possible factors.
- Cross-task Comparison: Across construction and reasoning experiments, LLMs show stronger reasoning capabilities than construction capabilities, although GPT-4 does not generally surpass fine-tuned extraction models.The study compares performance disparities between LLMs and current SOTA methods across diverse domains and datasets.
- Evaluation Considerations: Performance depends on dataset quality, instruction quality, and evaluation methods, because noise, complex contexts, prompt design, and synonym handling can affect results.The authors also note that randomly sampled subsets and interactive-interface evaluation may introduce variability.
3 Future Opportunities: Automatic KG Construction and Reasoning
AutoKG proposes multi-agent collaboration, external knowledge sources, and human-machine interaction for KG construction and reasoning. The approach offers practical benefits but still faces token, interaction, hallucination, and validation challenges.
- Automatic KG Construction and Reasoning: AutoKG assigns multiple communicative agents distinct roles to collaboratively construct and reason over knowledge graphs.The framework reduces the human role and enables agents to interact with external sources.
- Automatic KG Construction and Reasoning: The demonstration uses a task-specifier, KG assistant, KG user, and web searcher to coordinate role-based KG development.The KG assistant consults the web searcher before continuing with the user’s command.
- Potential Benefits: AutoKG combines domain and internet knowledge to simplify specialized KG creation, improve factual accuracy, and increase transparency.The paper presents these benefits as potential outcomes of combining artificial intelligence and human expertise.
- Challenges: Constructed graphs still require manual evaluation and validation despite AutoKG assistance.This remains an explicit limitation of the current system.
- Challenges: Three challenges remain: API token limits, inefficient human-machine interaction, and hallucinated information requiring output scrutiny.The paper mentions expert review, standard-answer comparison, and semi-automatic algorithms as possible checking mechanisms.
4 Conclusion and Future Work
The paper evaluates LLMs for KG construction and reasoning, examines their contextual learning through Virtual Knowledge Extraction, and proposes multi-agent AutoKG. Future work includes broader model coverage and multimodal reasoning.
- Conclusion and Future Work: The study investigates whether LLM extraction reflects pre-training knowledge or contextual learning, using Virtual Knowledge Extraction and a novel dataset.The reported results highlight robust contextual learning.
- Conclusion and Future Work: The paper proposes AutoKG, which uses multiple agents to perform KG construction and reasoning tasks.The proposal is presented as an innovative method based on the study’s findings.
- Conclusion and Future Work: Future work will extend the study to other LLMs and additional KG tasks, including multimodal reasoning.
Declarations
The declarations identify the work’s funding sources and provide the repository location for its code and datasets.
- Declarations: The work received support from Chinese national, provincial, university, and industry research programs.The listed funders include the National Natural Science Foundation of China, Zhejiang University, Zhejiang Provincial Natural Science Foundation, Tencent AI Lab, and others.
- Declarations: The paper provides its code and datasets through the AutoKG GitHub repository.The repository URL is https://github.com/zjunlp/AutoKG.
A.1 Large Language Models
Large language models are trained on substantial textual data and have become important in contemporary NLP research. Models such as GPT-3, ChatGPT, and GPT-4 show strong performance across multiple NLP tasks.
- Large Language Models: LLMs are pre-trained on substantial amounts of textual data and are significant components of contemporary NLP research.
- Large Language Models: GPT-3, ChatGPT, and GPT-4 demonstrate exceptional performance across tasks including translation, summarization, and question answering.
A.2 ChatGPT & GPT-4
ChatGPT and GPT-4 are general-purpose OpenAI language models studied for capabilities across information extraction, reasoning, and related NLP tasks. Prior work reports broad versatility for ChatGPT, while GPT-4 research remains comparatively limited.
- ChatGPT uses RLHF during fine-tuning to improve alignment with human preferences and values.
- GPT-4 is described as a large-scale multimodal model that processes image and text inputs.
- ChatGPT reportedly performs strongly in zero-shot learning across tasks but faces challenges with low-resource languages.
- ChatGPT has been studied for information extraction, reasoning, question answering, summarization, and machine translation.
- Investigations of GPT-4 have covered medical examinations, cognitive psychology, academic exams, and radiology-report translation.
- Existing LLM-based KG research includes named entity recognition, relation extraction, and link prediction.
Appendix B Datasets
The paper evaluates KG construction and reasoning using datasets spanning extraction, link prediction, and question answering. These benchmarks vary substantially in domain, scale, relation types, and inference structure.
- Entity, Relation and Event Extraction: DuIE2.0 contains more than 210,000 Chinese sentences and 48 predefined relationship categories.
- Entity, Relation and Event Extraction: SciERC consists of scientific abstracts annotated with seven relations.
- Entity, Relation and Event Extraction: Re-TACRED includes over 91,000 sentences spanning 40 relations.
- Entity, Relation and Event Extraction: MAVEN is a general-domain event extraction benchmark with 4,480 documents and 168 event types.
- Link Prediction: ATOMIC 2020 contains 1.33 million inferential knowledge tuples about entities and events, while FreebaseQA and MetaQA support open-domain and multi-hop question answering.
Appendix C Data Collection of VINE
VINE is constructed from participant responses to New York Times competitions that asked contributors to invent words addressing gaps in English. The resulting dataset is designed with controlled coverage of novel relations and entities.
- The data sources include the January 2022 February Vocabulary Challenge and the February 2023 Student Vocabulary Challenge.
- Both competitions promote distinctive new words intended to address gaps in the English language.
- The constructed dataset contains 1,400 sentences, 39 novel relations, and 786 unique entities.
- Each relation type has at least 10 associated samples to support subsequent experiments.
- Additional Re-TACRED training sentences were selected when test-set relation types had fewer than 10 instances.
Appendix D Prompts for Evaluation
The appendix documents prompts used to evaluate relation extraction, event detection, link prediction, question answering, and Virtual Knowledge Extraction. It includes examples of zero-shot and one-shot prompting alongside the VINE examples.
- Appendix D Prompts for Evaluation: Appendix D lists the prompts used for each evaluated task.
- Appendix D Prompts for Evaluation: Table D1 provides examples of zero-shot and one-shot prompts for relation extraction.
- Appendix D Prompts for Evaluation: Table D2 provides examples of zero-shot and one-shot prompts for event detection, link prediction, and question answering.
- Appendix E Prompts for Virtual Knowledge Extraction: Table E3 contains examples of Virtual Knowledge Extraction.