PaperlayerView exact revision on arXiv

Source-linked AI summary

Making Large Language Models Perform Better in Knowledge Graph Completion

Yichi Zhang, Zhuo Chen, Lingbing Guo, Yajing Xu, Wen Zhang, Huajun Chen

arXiv:2310.06671v2cs.CL

TL;DR

Knowledge graphs omit unobserved triples, and LLM-based completion needs to incorporate graph structure for structure-aware reasoning. The paper adapts in-context learning and instruction tuning with textual structural information, then introduces KoPA, which maps pretrained structural embeddings into prefix tokens; KoPA outperforms baselines on triple classification.

  • Problem

    Knowledge graphs contain unobserved or missing triples, while LLM-based completion needs to incorporate graph structure for structure-aware reasoning.

    Source: §1 · p. 1, §3.1 · p. 3

  • Method

    The paper adapts in-context learning and instruction tuning with textual structural information, and proposes KoPA to project pretrained structural embeddings into virtual knowledge tokens prefixed to prompts.

    Source: §1 · p. 2

  • Results

    KoPA outperforms 16 baselines in accuracy and F1 across three datasets; on CoDeX-S, it improves accuracy by 1.81% and F1 by 1.85%.

    Source: §5.2 · p. 6, §6 · p. 9

  • Takeaways & Limitations

    KoPA achieves better results than text-based structure-aware instruction tuning while using shorter prompts.

    Source: §5.2 · p. 7

    Text-based structural prompts can add redundant information, and longer contexts can reduce model capability and increase computational consumption.

    Source: §3.2.2 · p. 4

Abstract

from arXiv · show

Large language model (LLM) based knowledge graph completion (KGC) aims to predict the missing triples in the KGs with LLMs. However, research about LLM-based KGC fails to sufficiently harness LLMs' inference proficiencies, overlooking critical structural information integral to KGs. In this paper, we explore methods to incorporate structural information into the LLMs, with the overarching goal of facilitating structure-aware reasoning. We first discuss on the existing LLM paradigms like in-context learning and instruction tuning, proposing basic structural information injection approaches. Then we propose a Knowledge Prefix Adapter (KoPA) to fulfill this stated goal. The KoPA uses a structural pre-training phase to comprehend the intricate entities and relations within KGs, representing them as structural embeddings. Then KoPA communicates such cross-modal structural information understanding to the LLMs through a knowledge prefix adapter which projects the structural embeddings into the textual space and obtains virtual knowledge tokens positioned as a prefix of the input prompt. We conduct comprehensive experiments and provide incisive analysis concerning how the introduction of cross-modal structural information would be better for LLM's factual knowledge reasoning ability. Our code and data are available at https://github.com/zjukg/KoPA .

1 INTRODUCTION

LLM-based KGC often treats triples as text inputs, leaving KG structure unused despite its relevance to structure-aware reasoning. The paper explores textual adaptations of existing LLM paradigms and introduces KoPA, which transfers structural embeddings into LLM prompts as virtual knowledge tokens.

1 INTRODUCTION

  • LLM-based KGC methods can overlook subgraph structure, relational patterns, and entity relationships when predicting individual triples as text.The paper argues that incorporating this non-textual structure can support LLM understanding of KGs.Source: §1 · pp. 1–2
  • Structure-aware ICL and IT add KG information through textual prompts, but a semantic divide limits how fully they capture complex KG structure.The prompts use textual descriptions of entities and relations as complementary background information.Source: §1 · p. 2
  • KoPA pre-trains structural embeddings, then projects them into textual space as virtual knowledge tokens prefixed to the input prompt.The tokens provide supplementary input-triple information during instruction tuning.Source: §1 · p. 2
  • The paper investigates structure-aware ICL and IT, proposes KoPA, and evaluates the methods on three public benchmarks with baseline comparisons.Its evaluation also explores transfer ability and knowledge retention.Source: §1 · p. 2

2 RELATED WORKS

Prior KGC work includes embedding-based and language-model approaches, while LLM research has begun exploring knowledge-graph tasks. The paper positions its work at the intersection of graph structure and LLMs, extending multimodal approaches to KGC.

2.1 Knowledge Graph Completion

  • Embedding-based KGC models triple plausibility from KG structure, while PLM-based methods cast completion as text learning that can miss complex graph structure.The passage describes KG-BERT as binary text classification and later PLM methods as adding tasks or changing triple encoding.Source: §2.1 · p. 2

2.2 LLMs for KG research

  • LLMs can support KG tasks, but earlier LLM-for-KGC work lacked systematic exploration of using KG structure for structure-aware reasoning.The passage identifies KGLLaMA as an initial vanilla instruction-tuning approach.Source: §2.2 · p. 2

2.3 Incorporate Non-textual Modality Information into LLMs

  • Multimodal LLMs commonly encode non-textual inputs as virtual text tokens aligned with words through instruction tuning.The examples named include images, audio, and video.Source: §2.3 · pp. 2–3
  • Graph data is another modality, with prior work applying graph encoders or converting graph structure for LLM tasks; this paper focuses on complex KG structure for KGC.The cited examples include drug-interaction prediction and graph classification or node classification.Source: §2.3 · p. 3

3 BASIC SETTINGS FOR LLM-BASED KGC

The section frames triple classification as judging a KG triple from text while emphasizing that entity and relation structure matters for reliable, structure-aware reasoning. It extends training-free and instruction-tuning paradigms with KG information, while noting that textual neighborhood prompts can become redundant and costly as they grow.

3.1 Notations and Preliminaries

  • Triple classification asks the LLM to answer true or false from an instruction, optional demonstrations, and a textual triple whose KG-defined entities and relations require structural context.The KG is represented as entities, relations, triples, and descriptions; the triple prompt concatenates the head, relation, and tail descriptions.Source: §3.1 · p. 3

3.2.1 Training-free reasoning approaches.

  • Zero-shot reasoning predicts from the instruction and test-triple descriptions alone, without adding KG information to the input.Adding KG information would violate the stated zero-shot setting.Source: §3.2.1 · p. 3
  • Structure-aware ICL samples labeled positive and negative triples from the test triple’s local structure, with equal counts for balanced demonstrations.The selected triples share the test head or tail; positive triples are labeled true and negative triples false.Source: §3.2.1 · p. 3

3.2.2 Instruction tuning approaches.

  • KoPA projects pre-trained structural embeddings for the input triple into the LLM’s textual space as prefix tokens before instruction tuning.In a decoder-only LLM, following text can attend to these tokens through unidirectional attention.Source: Figure 2 · p. 4
  • Vanilla instruction tuning fine-tunes the LLM on an instruction, input triple, and predicted answer using next-word prediction.Negative sampling supplies negative training examples because the training KG contains only positive triples.Source: §3.2.2 · pp. 3–4
  • Structure-aware instruction tuning adds textual descriptions of sampled head and tail neighborhoods to the demonstration prompt.The enhanced sequence places these descriptions alongside the instruction, triple, and answer.Source: §3.2.2 · p. 4
  • Textual neighborhood prompts can add redundant information, while indefinitely lengthening context risks reduced model capability and higher computational consumption.The authors also report difficulty identifying which KG structural information is decisive for triple discrimination.Source: §3.2.2 · p. 4

4 METHODLOGY

KoPA incorporates KG structure into LLM-based completion by pre-training structural embeddings and projecting them into virtual knowledge-token prefixes. Its fixed three-token prefix offers a shorter alternative to neighborhood-text prompts whose length grows with the number of triples.

4 METHODLOGY

  • KoPA pre-trains entity and relation embeddings, then uses an adapter to project them into the LLM’s textual token space as virtual knowledge tokens.The structural embeddings are learned with a self-supervised negative-sampling objective to capture KG structure and relational patterns.Source: §4 · p. 4, §4.1 · pp. 4–5

4.1 Structural Embedding Pre-training

  • The pre-training objective scores triples using a plausibility function and negative samples to optimize structural embeddings.The embeddings encode entities, relations, and KG structure before being adapted to the LLM representation space.Source: §4.1 · pp. 4–5

4.2 Knowledge Prefix Adapter

  • The adapter transforms the pre-trained triple embeddings into prefix tokens that precede the instruction and triple prompt during fine-tuning and inference.The LLM’s unidirectional attention lets subsequent text tokens attend to the structural prefix.Source: §4.2 · p. 5

4.3 Complexity Analysis

  • The adapter prepends three virtual tokens for each head, relation, and tail, while neighborhood-text prompt length grows with the number of triples k.The embeddings are frozen during LLM fine-tuning, while the adapter learns their mapping into textual representations.Source: §4.3 · p. 5, §4.2 · p. 5

5 EXPERIMENTS

The experiments evaluate triple classification on UMLS, CoDeX-S, and FB15K-237N against embedding-, PLM-, and LLM-based baselines. They use balanced validation and test sets and compare training-free and fine-tuned LLM approaches.

5.1 Experimental Settings

  • The study evaluates triple classification on UMLS, CoDeX-S, and FB15K-237N using accuracy, precision, recall, and F1-score.Validation and test samples are balanced between positive and negative labels.Source: §5.1.1 · p. 5, §5.1.4 · p. 6

5.1.1 Datasets.

  • The benchmark comparison includes embedding-based, PLM-based, and LLM-based KGC methods, with LLM baselines spanning training-free and fine-tuning approaches.Training-free methods include zero-shot reasoning and in-context learning; fine-tuning methods include vanilla and structure-aware instruction tuning.Source: §5.1.2 · p. 5

5.1.2 Baseline Methods.

  • Embedding-based baselines use learned structural embeddings, while KG-BERT and PKGC represent the PLM-based comparison and KGLLaMA the LLM-based comparison.The authors reproduce baseline results and implement KoPA; PKGC results are reported only for FB15K-237N because reproducing it requires relation-specific manual prompts.Source: §5.1.2 · p. 5, §5.1.3 · p. 6

5.1.3 Implementation and Detail Settings.

  • KoPA uses Alpaca-7B with RotatE embeddings, while fine-tuned methods use LoRA; the reported LLM fine-tuning takes one to eight hours across datasets.The experiments also test GPT-3.5-turbo for zero-shot reasoning and sample four neighborhood triples for structure-aware instruction tuning.Source: §5.1.3 · p. 6

5.2 Main Results

KoPA outperforms 16 existing baselines on accuracy and F1 across all three datasets. Results also compare the use of structural embeddings with LLM approaches that rely on text prompts or lack fine-tuning.

5.2 Main Results

  • 1.81% accuracy and 1.85% F1 improvement on CoDeX-S; KoPA outperforms 16 existing baselines on accuracy and F1 across all three datasets.The reported main experiment is triple classification.Source: §5.2 · p. 6
  • KoPA is compared with the original embedding-based RotatE method, especially on larger and more challenging datasets such as CoDeX-S and FB15K-237N.Source: §5.2 · p. 7
  • Training-free LLM methods have limited gains from ICL demonstrations and often produce extreme recall values alongside relatively low F1 scores.The reported comparison includes zero-shot GPT-3.5-turbo with 175B parameters.Source: §5.2 · p. 7
  • Structure-aware instruction tuning with neighborhood information remains limited compared with KoPA, which achieves better results with shorter prompts.The authors suggest that structural embeddings carry richer semantic information than text-based auxiliary prompts.Source: §5.2 · p. 7

5.3 Transferability Exploration

The transferability experiment tests KoPA on triples involving entities withheld from training. KoPA performs better on unseen triples and degrades less as the inductive rate rises.

5.3 Transferability Exploration

  • The transferability experiment tests whether the knowledge prefix adapter transfers structural embeddings into textual token representations that provide auxiliary information for LLM decoding.The LLM is fine-tuned on remaining seen triples and tested on both seen and unseen triples.Source: §5.3 · p. 7
  • The experiment withholds a defined proportion of entities and removes training triples involving them, then evaluates on seen, unseen, and all test triples.The inductive rate is the ratio of unseen entities during training; the LLM does not see their textual descriptions.Source: §5.3 · p. 7
  • KoPA outperforms the other methods on unseen triples and shows less performance degradation as the inductive rate increases.The authors interpret this as evidence that the adapter's mapping from structural embeddings to textual representations transfers to entities unseen during training.Source: §5.3 · p. 7

5.4 Ablation Study

The ablation study examines structural embeddings and virtual-token placement in KoPA. Removing or randomizing the embeddings, or moving the tokens away from the prefix, reduces performance.

5.4 Ablation Study

  • Removing structural embeddings or replacing them with random initialized embeddings leads to performance decline.The study also finds that gains depend on the original embedding's strength in triple classification: TransE and RotatE outperform DistMult and ComplEx.Source: §5.4 · pp. 7–8
  • KoPA is compatible with different structural embeddings, but its performance gain depends on whether the embedding was powerful in triple classification.TransE and RotatE outperform DistMult and ComplEx in the reported triple-classification results.Source: §5.4 · pp. 7–8
  • Placing virtual knowledge tokens in the middle or at the end of the input sequence decreases performance relative to prefix placement.The authors attribute the prefix advantage to decoder-only LLMs' unidirectional self-attention, which lets all text attend to the prefix tokens.Source: §5.4 · p. 8

5.5 Case Study

The case study examines overlap among models' correct predictions and gives a triple-classification example. KoPA has correct predictions that several other models miss.

5.5 Case Study

  • KoPA has a significant portion of correct predictions that do not overlap with several other models' correct predictions.The Venn diagram's intersections represent predictions shared by different models on the same data.Source: §5.5 · p. 8
  • In the micro example, neighborhood facts about Coming to America do not help the structure-aware fine-tuned LLM judge the target triple, while KoPA's structural embeddings contain more information than structural information expressed as text.The example triple is (John Landis, film director film, Coming to America).Source: §5.5 · p. 8

5.6 Common Ability Retention

The authors assess general abilities before and after fine-tuning with MMLU. They report declines in most instances, alongside improved STEM proficiency on UMLS.

5.6 Common Ability Retention

  • The evaluation uses MMLU to assess LLM abilities across domains including Humanities, Social Sciences, and STEM, before and after fine-tuning.The overall evaluation results are presented across different datasets.Source: §5.6 · p. 8
  • After KoPA training, generalized abilities decreased in most instances, while STEM proficiency improved on the UMLS dataset.The authors attribute the UMLS result to its medical knowledge, which overlaps with medicine, biology, and chemistry.Source: §5.6 · p. 8
  • The paper identifies MMLU subjects with higher scores after UMLS training as highly relevant to domains covered by the UMLS knowledge graph.Table 5 lists the specific domains in which scores increased.Source: §5.6 · p. 8, Table 5 · p. 8

6 CONCLUSION

The paper extends ICL and IT with text-based structural information and proposes KoPA to incorporate pre-trained structural embeddings into LLMs. Triple-classification experiments show KoPA outperforming the structure-aware methods; a more unified framework and downstream KG adaptation remain future work.

6 CONCLUSION

  • Triple-classification experiments demonstrate outperforming results for KoPA among the structure-aware methods.The paper identifies a more unified framework for all KGC tasks as future work.Source: §6 · p. 9
  • The paper extends ICL and IT into structure-aware methods that incorporate KG structural information through text.Source: §6 · p. 9
  • KoPA incorporates pre-trained structural embeddings into LLMs through a knowledge prefix adapter.Source: §6 · p. 9
  • Future work will explore a more unified framework for all KGC tasks and adapting KGs to downstream LLM applications.The stated aim for downstream applications is to make LLMs knowledgeable, reliable, and human-friendly.Source: §6 · p. 9
Loading 2310.06671v2…