Source-linked AI summary
Exploring Large Language Models for Knowledge Graph Completion
Liang Yao, Jiazhen Peng, Chengsheng Mao, Yuan Luo
TL;DR
Knowledge graphs are often incomplete, motivating methods to assess the plausibility of missing triples. This paper frames completion as text-sequence prediction and instruction-tunes open large language models, reporting state-of-the-art results in triple classification and relation prediction.
Problem
Knowledge graphs remain incomplete, creating the task of evaluating the plausibility of triples that are not present.
Source: §I · p. 1Method
The method treats entities, relations, and triples as text sequences and instruction-tunes open large language models to predict triple plausibility or candidate entities and relations.
Source: §I · p. 1Results
The method achieves state-of-the-art results in triple classification and relation prediction, and fine-tuned LLaMA-7B and ChatGLM-6B can outperform ChatGPT and GPT-4.
Source: §I · p. 1Takeaways & Limitations
The authors conclude that instruction tuning helps large language models extract knowledge stored in their parameters more efficiently for knowledge graph prediction.
Source: B. Results · p. 4
Abstract
from arXiv · showhide
Knowledge graphs play a vital role in numerous artificial intelligence tasks, yet they frequently face the issue of incompleteness. In this study, we explore utilizing Large Language Models (LLM) for knowledge graph completion. We consider triples in knowledge graphs as text sequences and introduce an innovative framework called Knowledge Graph LLM (KG-LLM) to model these triples. Our technique employs entity and relation descriptions of a triple as prompts and utilizes the response for predictions. Experiments on various benchmark knowledge graphs demonstrate that our method attains state-of-the-art performance in tasks such as triple classification and relation prediction. We also find that fine-tuning relatively smaller models (e.g., LLaMA-7B, ChatGLM-6B) outperforms recent ChatGPT and GPT-4.
I. INTRODUCTION
Knowledge graphs support many AI tasks, but their incompleteness motivates evaluating whether unobserved triples are plausible. This study models triples as text and reports strong results from instruction-tuned large language models on knowledge graph completion tasks.
I. INTRODUCTION
- Knowledge graph completion evaluates the plausibility of triples missing from knowledge graphs that support tasks such as semantic search, recommendation, and question answering.A triple is a head entity, relation, and tail entity.Source: §I · p. 1
- Many knowledge graph embedding models rely on observed triple structure, making them vulnerable to sparsity; other studies incorporate textual information to enhance representations.KG-BERT uses a pretrained language model, while several later extensions focus on efficiency and performance but use relatively small models.Source: §I · p. 1
- The proposed approach treats entities, relations, and triples as text sequences and instruction-tunes open LLaMA and ChatGLM models to predict triple plausibility or a candidate entity or relation.The authors describe this as a systematic investigation of large language models for knowledge graph completion.Source: §I · p. 1
- Across several benchmarks, the method achieves state-of-the-art results in triple classification and relation prediction; fine-tuning smaller models such as LLaMA-7B and ChatGLM-6B can outperform recent ChatGPT and GPT-4.Source: §I · p. 1
A. Knowledge Graph Completion
Earlier knowledge graph completion approaches include structural scoring models and methods that incorporate external information, including textual descriptions. This work distinguishes itself by using large language models and their cited emergent abilities for completion.
A. Knowledge Graph Completion
- Structural-only completion methods include translational-distance and semantic-matching models, while external information such as entity types, logical rules, and textual descriptions can enhance performance.The related-work discussion also notes convolutional neural network approaches.Source: A. Knowledge Graph Completion · p. 1
- Textual-description methods range from entity-name embeddings and shared entity-word spaces to CNN encoders and pretrained-language-model approaches such as KG-BERT.Later work extended KG-BERT cross-encoders to bi-encoders for performance and inference efficiency; KGT5 and KG-S2S also use sequence-to-sequence completion with relatively small pretrained models.Source: A. Knowledge Graph Completion · p. 2
- The proposed method uses large language models with in-context learning, instruction following, and step-by-step reasoning abilities that the authors say are absent from small models and helpful for completion.Source: A. Knowledge Graph Completion · p. 2
B. LLMs with KG Completion
Prior studies evaluated ChatGPT and GPT-4 on knowledge-graph link prediction, and this study builds on that work with broader task coverage and instruction tuning. It also distinguishes its scope from a later approach focused only on link prediction.
B. LLMs with KG Completion
- The study builds on three related evaluations of ChatGPT and GPT-4 for link prediction, while providing more comprehensive completion results and instruction tuning across three tasks.Source: B. LLMs with KG Completion · p. 2
- A later study incorporated knowledge graph embeddings such as TransE during instruction fine-tuning but focused only on link prediction, unlike this study’s exploration of triple classification and relation classification as well.Source: B. LLMs with KG Completion · p. 2
A. Knowledge Graph Completion Tasks
The section presents three knowledge graph completion tasks and describes how they can be framed as prompt questions for a language model.
A. Knowledge Graph Completion Tasks
- Triple classification asks whether a given head-relation-tail triple is correct or incorrect, framed as a true-or-false question.The example prompt asks whether Steve Jobs founded Apple Inc.Source: A. Knowledge Graph Completion Tasks · p. 2
- Relation prediction supplies a head and tail entity and asks the model to choose their relationship from candidate relations.The example asks for the relation between Steve Jobs and Apple Inc.Source: A. Knowledge Graph Completion Tasks · p. 2
- Entity prediction supplies a head and relation to predict the tail, or a tail and relation to predict the head.The examples use “Steve Jobs founded” to ask for the tail and a question about “founded Apple Inc.” to ask for the head.Source: A. Knowledge Graph Completion Tasks · p. 2
B. Instruction Tuning LLM with KG (KG-LLM)
KG-LLM aligns open LLMs with knowledge-graph triples through instruction tuning on factual question-answer prompts.
B. Instruction Tuning LLM with KG (KG-LLM)
- KG-LLM fine-tunes ChatGLM-6B and LLaMA models on prompts and responses from knowledge-graph training triples.ChatGLM-6B uses P-tuning v2; LLaMA versions 1 and 2 use LoRA.Source: B. Instruction Tuning LLM with KG… · p. 2
- For entity prediction, training and test instructions can include five sampled neighboring entities, excluding the target.The neighbors are supplied as context before the prompt to complete the fact.Source: B. Instruction Tuning LLM with KG… · p. 2
- Table I summarizes statistics for the knowledge-graph datasets used in the study.Source: Table I · p. 2
A. Datasets and Settings
The experiments use four benchmark knowledge graphs and compare KG-LLM with embedding, language-model, and commercial LLM baselines.
A. Datasets and Settings
- Experiments use WN11, FB13, WN18RR, and YAGO3-10, with FB13-100 and YAGO3-10-100 subsets for GPT-4 evaluation.Each subset contains 100 randomly selected test examples, due to GPT-4 access limits.Source: A. Datasets and Settings · p. 2
- Baselines span knowledge-graph embedding methods, text-enhanced and pretrained-language-model approaches, ChatGPT, and GPT-4.Source: A. Datasets and Settings · p. 2
- The study evaluates triple-classification accuracy, entity-prediction Hits@1, and relation-prediction Hits@1 using the corresponding result tables.The supplied captions identify Tables II, IV, and V as reporting these measures.Source: Table II · p. 3, Table IV · p. 3, Table V · p. 3
B. Results
KG-LLM performs strongly across triple classification and prediction tasks; the reported comparisons also examine response quality and possible reasons for its performance.
B. Results
- KG-LLaMA2-13B achieves the highest triple-classification accuracy on WN11 and FB13, while KG-LLaMA scores above ChatGPT and equal to GPT-4 on FB13-100.Source: B. Results · p. 4
- KG-LLaMA-7B achieves the best relation-prediction Hits@1 on YAGO3-10, exceeding GPT-4, while KG-ChatGLM-6B also improves substantially.Instruction tuning and incorporating structural information improve entity-prediction results; the passage says structural information improves them by a large margin.Source: B. Results · p. 4
- Compared with original LLMs, instruction-tuned models give answers more like training triples and show greater awareness of facts.Source: B. Results · p. 4
- The authors attribute KG-LLM’s performance to LLMs’ broader general knowledge and instruction tuning’s alignment of pretrained weights with triple descriptions.Source: B. Results · p. 4
V. CONCLUSIONS AND FUTURE WORK
The paper concludes that KG-LLM achieves state-of-the-art results in several knowledge-graph completion tasks and identifies prompt engineering and entity prediction as future work.
V. CONCLUSIONS AND FUTURE WORK
- KG-LLM achieves state-of-the-art performance in triple classification and relation prediction.Source: §V · p. 4
- Future work will explore more effective prompt-engineering techniques for LLMs.Source: §V · p. 4
- For entity prediction, the authors plan to select top-n candidate entities with structural methods such as TransE, then rank them with LLMs.Source: §V · p. 4