Source-linked AI summary
From Discrimination to Generation: Knowledge Graph Completion with Generative Transformer
Xin Xie, Ningyu Zhang, Zhoubo Li, Shumin Deng, Hui Chen, Feiyu Xiong, Mosha Chen, Huajun Chen
TL;DR
Knowledge graph completion seeks to predict missing triples, while prior discrimination methods can require costly scoring of all possible triples and depend on negative sampling. GenKGC treats link prediction as sequence generation with a pretrained language model, relation-guided demonstrations, and hierarchical decoding. The authors report comparable results with reduced inference and training cost, especially in inference time.
Problem
Discrimination-based knowledge graph completion can require costly scoring of all possible triples and suffer from negative-sampling instability.
Source: §1 · p. 1Method
GenKGC represents entities and relations as text, generates target entities with a pretrained sequence-to-sequence model, and adds relation-guided demonstrations and entity-aware hierarchical decoding.
Source: §1 · p. 1, §2.1 · p. 2, §2.2 · p. 2Results
GenKGC reaches comparable results while reducing inference and training cost in link prediction with pretrained models, with effectiveness especially in inference time.
Source: §5 · p. 4Takeaways & Limitations
The authors release OpenBG500, a large-scale knowledge graph dataset, for research purposes.
Source: §1 · p. 2
Abstract
from arXiv · showhide
Knowledge graph completion aims to address the problem of extending a KG with missing triples. In this paper, we provide an approach GenKGC, which converts knowledge graph completion to sequence-to-sequence generation task with the pre-trained language model. We further introduce relation-guided demonstration and entity-aware hierarchical decoding for better representation learning and fast inference. Experimental results on three datasets show that our approach can obtain better or comparable performance than baselines and achieve faster inference speed compared with previous methods with pre-trained language models. We also release a new large-scale Chinese knowledge graph dataset AliopenKG500 for research purpose. Code and datasets are available in https://github.com/zjunlp/PromptKG/tree/main/GenKGC.
1 INTRODUCTION
Knowledge graph completion predicts missing triples, but earlier discrimination methods can require costly scoring of all possible triples and suffer from negative-sampling instability. GenKGC reframes link prediction as sequence generation, adding relation-guided demonstrations and entity-aware hierarchical decoding to reduce inference time while maintaining performance.
1 INTRODUCTION
- Earlier discrimination approaches can require costly scoring of all possible triples, suffer from negative-sampling instability, and use memory-intensive dense embeddings.The paper contrasts these approaches with a search for a new technical solution to knowledge graph completion.Source: §1 · p. 1
- GenKGC converts link prediction into sequence-to-sequence generation, using a pretrained language model to generate target entities from entity and relation inputs.The approach adds same-relation demonstrations and entity-aware hierarchical decoding.Source: §1 · pp. 1–2
- The authors report that GenKGC reduces inference time while maintaining performance, and release OpenBG500 as a large-scale knowledge graph dataset for research.The paper also reports results on two existing datasets.Source: §1 · p. 2
2 METHOD
GenKGC formulates link prediction as sequence-to-sequence generation: entity and relation text forms the input, and the model generates a target entity. It augments inputs with relation-guided examples and uses hierarchical decoding to constrain entity generation.
2.1 Link Prediction as Seq2Seq Generation
- Given a head entity and relation, GenKGC uses their descriptions as input text and trains BART to generate the missing tail entity.Entities and relations are represented as token sequences rather than unique embeddings.Source: §2.1 · p. 2
2.2 Relation-guided Demonstration
- Relation-guided demonstrations add training triples sharing the input relation, providing examples for the encoder in a long-tailed knowledge graph setting.The paper notes that FB15k-237 has only 37 instances of the relation film/type_of_appearance.Source: §2.2 · p. 2
2.3 Entity-aware Hierarchical Decoding
- Entity-aware hierarchical decoding uses beam search to obtain top-k entities and constrains generation with entity categories and a prefix tree.The method is motivated by the cost of enumerating and sorting all entities; category selection uses the lowest-frequency category in the training set.Source: §2.3 · pp. 2–3
3 EXPERIMENT
GenKGC is evaluated on three link-prediction datasets against embedding and textual-encoding baselines, with comparisons covering performance, inference, and memory. A case study examines how entity-aware decoding affects generated answers.
3 EXPERIMENT
- Experiments use BART-base, compare graph-embedding and textual-encoding baselines, and select test-set hyperparameters using the dev set.The datasets are FB15k-237, WN18RR, and OpenBG500; entity descriptions come from Wikipedia introductions, word definitions, and e-commerce pages, respectively.Source: §3 · p. 3
- GenKGC performs better than KG-BERT across all three datasets while maintaining high inference speed.For OpenBG500, TransE uses 260M parameters to store entities and relations, compared with 110M for BERT or BART.Source: §3 · p. 3
3.1 Case Study
- Table 4 compares the first five generated entities and their probabilities before and after GenKGC reranking.Without entity-aware decoding, normal beam search may stop at an answer that is correct but not precise enough; the authors attribute this to pretrained-language-model token bias.Source: Table 4 · p. 4, §3.1 · p. 4
- Entity-aware hierarchical decoding constrains generation and mitigates bias from high-frequency tokens.The authors argue that this bias can steer the model toward certain answers.Source: §3.1 · p. 4
4 RELATED WORK
Related work covers knowledge-graph embedding methods and pretrained-language-model approaches to completion. It distinguishes vector-based scoring from methods that encode triples or generate structured outputs.
Knowledge Graph Embedding Models
- Knowledge-graph embedding methods represent entities and relations with vectors; translation-based methods model relations as mappings between entities [17].The passage gives TransE, ConE [20], and TotatE [9] as examples, and describes semantic-matching models as using multilinear or bilinear products.Source: §4 · p. 4
Pre-trained Language Models for KGC
- Recent work applies transformer-based pretrained language models to knowledge-graph completion [15] [16] [19].Source: §4 · p. 4
- KG-BERT represents a triple as a sequence and formulates completion as binary sequence classification [14].Another approach uses a transformer encoder-decoder to take plain text as input and output a structured triple [7].Source: §4 · p. 4
5 CONCLUSION
GenKGC reaches comparable link-prediction results while reducing inference and training costs with pretrained models.
5 CONCLUSION
- GenKGC reaches comparable results while reducing inference and training cost in link prediction with pretrained models.Source: §5 · p. 4