PaperlayerView exact revision on arXiv

Source-linked AI summary

Sequence-to-Sequence Knowledge Graph Completion and Question Answering

Apoorv Saxena, Adrian Kochsiek, Rainer Gemulla

arXiv:2203.10321v1cs.CLcs.LG

TL;DR

Large knowledge graphs make entity-specific KGE models expensive and complicate their use in downstream question answering. KGT5 replaces triple scoring with sequence-to-sequence encoding and autoregressive decoding, then finetunes the same model for KGQA. It uses up to 98% fewer parameters and outperforms baselines across multiple large-scale incomplete-KGQA benchmarks.

  • Problem

    KGE models create entity-specific embeddings that become large on real-world graphs and often require multi-stage integration for downstream KGQA.

  • Method

    KGT5 poses link prediction and question answering as sequence-to-sequence tasks using one encoder-decoder Transformer, with link-prediction pretraining before QA finetuning.

  • Results

    98% fewer parameters and baseline-outperforming performance on multiple large-scale incomplete-KGQA benchmarks are reported for KGT5.

  • Takeaways & Limitations

    A single sequence-to-sequence model provides a scalable and versatile approach to KG completion and question answering over incomplete KGs.

  • Takeaways & Limitations

    KGT5 is not recommended as a standalone KGQA method because it may poorly memorize facts and performs poorly in full-KG KGQA; it also requires textual entity representations.

Abstract

from arXiv · show

Knowledge graph embedding (KGE) models represent each entity and relation of a knowledge graph (KG) with low-dimensional embedding vectors. These methods have recently been applied to KG link prediction and question answering over incomplete KGs (KGQA). KGEs typically create an embedding for each entity in the graph, which results in large model sizes on real-world graphs with millions of entities. For downstream tasks these atomic entity representations often need to be integrated into a multi stage pipeline, limiting their utility. We show that an off-the-shelf encoder-decoder Transformer model can serve as a scalable and versatile KGE model obtaining state-of-the-art results for KG link prediction and incomplete KG question answering. We achieve this by posing KG link prediction as a sequence-to-sequence task and exchange the triple scoring approach taken by prior KGE methods with autoregressive decoding. Such a simple but powerful method reduces the model size up to 98% compared to conventional KGE models while keeping inference time tractable. After finetuning this model on the task of KGQA over incomplete KGs, our approach outperforms baselines on multiple large-scale datasets without extensive hyperparameter tuning.

1 Introduction

Real-world KGs are incomplete and large, creating tension between scalable, high-quality representations and versatile, simple downstream use. KGT5 addresses this by treating link prediction and question answering as sequence-to-sequence tasks with one encoder-decoder Transformer.

  • Real-world KGs are incomplete, motivating knowledge graph completion from existing facts.
  • 90M entities in Wikidata illustrate why model size and inference time should not scale with the number of entities.
  • Traditional KGE models scale linearly with entity count and offer limited versatility because they assign unique embeddings to entities and relations.
  • KG-BERT is potentially versatile but remains non-scalable because its cross-encoder requires costly evaluation.An evaluation run on a KG with 40k entities was estimated to take 3 days.
  • KGT5 poses link prediction as sequence-to-sequence learning, then finetunes the same model for question answering with link-prediction regularization.The approach uses an off-the-shelf encoder-decoder Transformer and avoids task- or dataset-specific hyperparameter tuning.
  • 98% fewer parameters and performance that outperforms conventional KGEs on a dataset with 90M entities summarize KGT5's reported scalability and quality.For KGQA, it performs similarly to or better than more complex methods on multiple large-scale benchmarks.

2 Background & Related Work

KG link prediction predicts missing triples, while KGQA answers natural-language questions using a KG. Existing approaches face incompleteness, scalability, versatility, or pipeline-complexity limitations.

  • Link prediction answers missing-triple queries such as (s, p, ?) and (?, p, o), typically using KGE models.
  • Conventional KGEs assign embeddings to entities and relations and score candidate triples with model-specific functions.
  • KGQA ranges from single-fact retrieval to multi-fact reasoning over a KG.
  • Atomic KGE models achieve strong link prediction but scale linearly with entity count and require separate KGE and QA modules.
  • Semantic parsing can fail on incomplete KGs because one missing link can make a symbolic query fail.
  • Prior incomplete-KGQA methods are limited by query-type coverage, structural assumptions, entity-count constraints, or dataset-dependent components.

3 The KGT5 Model

KGT5 converts KG link prediction and question answering into sequence-to-sequence tasks handled by one encoder-decoder Transformer. It uses textual verbalization, autoregressive decoding, and link-prediction-pretrained QA finetuning.

  • 3 The KGT5 Model: KGT5 uses a T5-small-architecture encoder-decoder Transformer trained from scratch on link prediction and adapted to KGQA.During QA training, link prediction regularizes the model.
  • 3.1 Textual Representations & Verbalization: Link prediction queries require one-to-one textual representations of entities and relations plus a verbalization scheme.For Wikidata-based KGs, canonical mentions are disambiguated with descriptions and unique identifiers.
  • 3.1 Textual Representations & Verbalization: A query such as (barack obama, born in, ?) becomes “predict tail: barack obama | born in,” with the answer mention as output.
  • 3.2 Training KGT5 for Link Prediction: Training creates two verbalized inputs per triple, targets the corresponding entity mentions, and optimizes teacher-forced cross-entropy loss.
  • 3.3 Inference: Unlike conventional KGE scoring, KGT5 samples a fixed number of decoder sequences, maps them to entity IDs, and scores them by log probability.This avoids scoring every entity in the KG while approximating top-m predictions.
  • 3.4 KGQA Training and Inference: For KGQA, KGT5 is pretrained on background-KG link prediction, finetuned with a predict-answer prefix, and regularized using equal QA and link-prediction batches.Inference uses beam search followed by neighbourhood-based reranking.
  • 3.3 Inference: Decoded sequences are not guaranteed to be entity mentions, although increasing the number of samples was sufficient in experiments.

4 Experimental Study

The experimental study evaluates KGT5 on large-scale link prediction and incomplete-KG question answering benchmarks, alongside conventional KGE and KGQA baselines. Results indicate strong performance across tasks, while dataset construction and KG splits affect comparability.

  • Key findings: KGT5 reaches or outperforms current state-of-the-art on large-KG link prediction while reducing model size by up to 98%.The reported finding concerns link prediction on large knowledge graphs.
  • Key findings: KGT5 obtains better results than current state-of-the-art methods across multiple incomplete-KG question answering datasets.The evaluated KGQA benchmarks include MetaQA, WebQuestionsSP, and ComplexWebQuestions.
  • Datasets: The study evaluates link prediction on Wikidata5M, WikiKG90Mv2, FB15k-237, WN18RR, and YAGO3-10, and evaluates KGQA on MetaQA, WQSP, and CWQ.Wikidata5M and WikiKG90Mv2 are described as two of the largest publicly available benchmark KGs.
  • Experimental protocol: All KGs randomly lose 50% of their edges to simulate incompleteness, and the authors release their KG split to improve reproducibility.The authors note that differing stochastic splits make comparisons difficult and report ground-truth query accuracy as a split-hardness estimate.
  • Baselines: Link prediction comparisons include standard and text-based KGE models, while KGQA comparisons include PullNet, EmQL, EmbedKGQA, LEGO, and PathPred.The comparison set varies by dataset, with NBFNet used for the small benchmark KGs and PathPred additionally used for MetaQA.

4.3 Experimental Setup

The setup uses a small Transformer configuration with fixed training and inference procedures rather than dataset-specific hyperparameter tuning. Evaluation reports strong link-prediction results, alongside limitations on small graphs and incomplete test-set comparability.

  • Model and training: KGT5 uses the T5-small architecture with ∼60M parameters, randomly initialized weights, and a BPE tokenizer trained on verbalized KGs.The tokenizer is trained with SentencePiece.
  • Model and training: The experiments use AdaFactor, learning-rate warmup, batch size 320, 10% dropout, and a fixed 0.001 learning rate for QA finetuning.QA finetuning halves the link-prediction batch size, and no dataset-specific hyperparameter tuning is performed.
  • Inference: Inference uses sampling size = 500 for link prediction and beam size = 4 for KGQA, with neighbourhood-based reranking for eligible QA datasets.Reranking uses the topic entity’s 1-, 2-, or 3-hop neighbourhood when topic-entity annotations are available.
  • Link prediction results: KGT5 achieves the highest validation MRR on WikiKG90Mv2 with 98% fewer parameters than the next-best leaderboard model.The baselines are TransE, ComplEx, and variants, including methods using text embeddings.
  • Evaluation caveat: Hidden test-set scores were unavailable for WikiKG90Mv2 because the experiments used entity mentions supplied with the dataset.The authors therefore report validation results in the described comparison.
  • Link prediction results: On Wikidata5M, KGT5 outperforms all low-parameter models on every reported metric, while differing from the large ComplEx model by −0.008 MRR and +0.012 hits@1.The comparison to large ComplEx reports a drop in MRR and a gain in hits@1.
  • Link prediction analysis: KGT5 performs best on queries with few or no training answers, but performance drops when several entities can be correct; limited sampling may omit known positives.The authors attribute this pattern possibly to the difficulty of sampling and ranking low-probability sequences.
  • Limitations: On KGs with ≤150k entities, KGT5 sometimes trails baselines, possibly because Transformer models struggle when data are scarce.This is identified as a limitation for smaller datasets.

4.5 QA over Incomplete KGs with KGT5

KGT5 outperformed baselines on two Freebase-based KGQA datasets, with the largest gains on the hardest dataset. On MetaQA, performance varied by hop count and question entity types, while reranking and an ensemble improved results.

  • KGT5 outperformed all baselines on ComplexWebQuestions and WebQuestionsSP, with the largest gains on ComplexWebQuestions.ComplexWebQuestions was the hardest dataset in terms of complexity and KG size.
  • On MetaQA 1-hop and 3-hop questions, KGT5 was equal or better than all baselines, but it performed significantly worse on 2-hop questions.The 2-hop performance was also worse than ground-truth querying.
  • KGT5 struggled most when the head and answer entities had the same type, whereas it exceeded ground-truth querying when their types differed.Examples include actor → movie → actor versus director → movie → language questions.
  • The KGT5–PathPred ensemble outperformed every single model across all MetaQA datasets, often by large margins.It uses PathPred when that method returns answers and KGT5 otherwise.
  • Neighbourhood reranking produced small but consistent KGQA gains on all datasets.The reported metric was hits@1 in the 50% KG setting.

4.6 Relation to Knowledge Probing

The paper distinguishes KGT5’s KG-specific pretraining from generic language-model pretraining and examines its limitations in full-KG question answering. KGT5 performs strongly on incomplete KGs but is less effective at memorizing known facts and is not claimed as a general-purpose language model.

  • Relation to Knowledge Probing: The paper does not claim that KGT5 is as general-purpose as large language models or that it contains generic world knowledge.Consequently, it does not evaluate knowledge probing datasets such as T-REx or Google-RE.
  • Relation to Knowledge Probing: KGT5 vastly outperformed T5-small on KGQA, indicating that link prediction pretraining was responsible for its strong performance.T5-small has the same architecture but was pretrained on generic language-modeling and other tasks.
  • Relation to Knowledge Probing: KGT5 only marginally improved from full-KG pretraining over 50% KG pretraining and lagged behind EmbedKGQA and CBR-KGQA in full-KG QA.This pattern suggests stronger generalization to unseen facts than memorization of facts seen during pretraining.
  • Relation to Knowledge Probing: KGT5’s train MRR was significantly worse than ComplEx’s despite comparable test MRR, consistent with weaker memorization.KGT5 has only 60M parameters, which the authors identify as one possible explanation.

5 Conclusion and Future Work

The paper concludes that KG link prediction and question answering can be handled as seq2seq tasks by one encoder-decoder Transformer. KGT5 reduces parameters while remaining competitive and suggests link prediction as useful pretraining for knowledge-intensive QA.

  • 5 Conclusion and Future Work: KG link prediction and question answering can be treated as seq2seq tasks using a single encoder-decoder Transformer.KGT5 is trained on link prediction and then finetuned for QA.
  • 5 Conclusion and Future Work: KGT5 achieved competitive large-KG completion performance with up to 98% fewer parameters than conventional methods.The conclusion also reports that it outperformed baselines on multiple large-scale incomplete-KG QA benchmarks.
  • 5 Conclusion and Future Work: Link prediction training could be more beneficial than language-model pretraining for knowledge-intensive tasks such as KGQA.This conclusion follows the paper’s comparison of KGT5 and T5-small pretraining.
  • Future Work: Future work should investigate link prediction as an additional pretraining objective and whether larger Transformers store more relational information.The paper identifies model size and relational information storage as open questions.

A Textual representations of entities and relations

The experiments construct one-to-one textual representations for entities and relations, using canonical names and added descriptions, aliases, or unique identifiers to disambiguate duplicates.

  • Wikidata entities and relations use canonical page-title mentions, with one-line descriptions or unique identifiers appended when names collide.This produces a one-to-one mapping between entities and textual representations.
  • Freebase QA entities use English canonical names from identifier triples, with aliases or description fragments added for disambiguation.The relevant Freebase relations are /type/object/name, /common/topic/alias, and /common/topic/description.

B Teacher forcing

During teacher forcing, the model predicts each next token from the input and preceding target tokens, using cross-entropy against the true next-token distribution.

  • At each decoding step, the model produces a probability distribution over possible next tokens.
  • Cross-entropy penalizes differences between the predicted distribution and a target distribution assigning probability 1 to the true next token.
  • Teacher forcing supplies the target token as the next decoding input during training.
  • For a multi-token entity, the model’s output distribution is evaluated across the tokenized target sequence.

C Sampling strategy for link prediction

KGT5 generates link-prediction candidates by autoregressive sampling and scores complete sequences by token log probabilities. Sampling improves candidate generation relative to wider beam search, while performance varies across graph sizes and settings.

  • The model samples tokens from its output distribution and autoregressively decodes until a stop token to generate multiple predictions.
  • A sequence’s score is the sum of the log probabilities assigned to its tokens.
  • KGT5 is generally worse than NBFNet and ComplEx on FB15k-237 and YAGO3-10, while WN18RR performance is somewhat better.
  • Increasing sample size steadily increases MRR on Wikidata5M, whereas beam size ≥5 and beam search have the opposite effect.
  • Beam search is deterministic and can produce a requested number of predictions, but wider beams can sharply reduce generative-model performance.

D Path Predictor on MetaQA

PathPred maps MetaQA question templates to frequent relation paths and traverses the knowledge graph from each topic entity to obtain answers. Its ensemble with KGT5 provides a fallback when graph traversal returns no answers because of incompleteness.

  • MetaQA contains many questions but little linguistic variation because it is artificially generated from templates.
  • PathPred uses distant supervision to map a natural-language question to a relation path without using ground-truth queries.
  • For each question template, PathPred selects the most frequent k-hop relation path connecting the topic entity to an answer.
  • At test time, the predicted relation path traverses the knowledge graph from the topic entity to produce answers.
  • The KGT5 + PathPred ensemble applies PathPred first and invokes KGT5 when traversal yields an empty answer set.
  • MetaQA evaluation tables report Hits@1 for 1-hop, 2-hop, and 3-hop validation settings with 50% KG splits.
Loading 2203.10321v1…