PaperlayerView exact revision on arXiv

Source-linked AI summary

InGram: Inductive Knowledge Graph Embedding via Relation Graphs

Jaejun Lee, Chanyoung Chung, Joyce Jiyoung Whang

arXiv:2305.19987v3cs.LGcs.AI

TL;DR

Most inductive knowledge graph completion methods allow new entities but require relations to be seen during training, despite inference settings that include new relations. INGRAM constructs a relation graph and generates new relation and entity embeddings by aggregating neighbors; it outperforms 14 methods across varied inductive scenarios, especially at high new-relation ratios.

  • Problem

    Most inductive knowledge graph completion methods support new entities but assume relations were observed during training.

    Source: §1 · p. 1

  • Method

    INGRAM constructs a relation graph and generates embeddings for new relations and entities by aggregating their neighbors.

    Source: §1 · p. 1, §4 · pp. 3–4

  • Results

    INGRAM significantly outperforms 14 methods across varied inductive scenarios, with a substantial advantage when the ratio of new relations is high.

    Source: §1 · p. 2

  • Takeaways & Limitations

    INGRAM can infer from knowledge-graph structure without extra information about entities or relations or the aid of rich language models.

    Source: §1 · p. 2

    When known relations dominate, INGRAM is biased toward generalizing to new relations and does not yet incorporate known-relation-specific patterns.

    Source: §7 · p. 9

Abstract

from arXiv · show

Inductive knowledge graph completion has been considered as the task of predicting missing triplets between new entities that are not observed during training. While most inductive knowledge graph completion methods assume that all entities can be new, they do not allow new relations to appear at inference time. This restriction prohibits the existing methods from appropriately handling real-world knowledge graphs where new entities accompany new relations. In this paper, we propose an INductive knowledge GRAph eMbedding method, InGram, that can generate embeddings of new relations as well as new entities at inference time. Given a knowledge graph, we define a relation graph as a weighted graph consisting of relations and the affinity weights between them. Based on the relation graph and the original knowledge graph, InGram learns how to aggregate neighboring embeddings to generate relation and entity embeddings using an attention mechanism. Experimental results show that InGram outperforms 14 different state-of-the-art methods on varied inductive learning scenarios.

1. Introduction

Inductive knowledge graph completion addresses missing triplets involving new entities, but most existing methods still require relations to be known during training. InGram supports settings with new relations by aggregating neighbors to generate embeddings, and the paper reports gains over 14 methods across varied inductive scenarios.

1. Introduction

  • Knowledge graphs represent facts as head–relation–tail triplets, and classical embedding methods assume all entities and relations are observed during training.Such methods predict missing triplets by selecting plausible combinations of observed entities and relations.Source: §1 · p. 1
  • Most inductive methods allow new entities but require relations to be observed during training, limiting relation inference to the transductive setting.Figure 1(a) illustrates this existing setting for relations.Source: §1 · p. 1, Figure 1 · p. 1
  • INGRAM handles semi-inductive settings with both known and new relations and inductive settings where all relations are new.It generates embeddings for new relations and entities appearing only at inference time.Source: §1 · p. 1
  • INGRAM builds a relation graph and learns attention-based neighbor aggregation to generate relation and entity embeddings for inference.Training optimizes aggregation to maximize the plausibility scores of training triplets.Source: §1 · p. 2, Figure 2 · p. 2
  • INGRAM significantly outperforms 14 knowledge graph completion methods across inductive scenarios, with a substantial advantage when the ratio of new relations is high.The paper evaluates varied ratios of new relations, including settings where all relations are new.Source: §1 · p. 2

2. Related Work

Related work addresses inductive completion through rule mining, subgraph reasoning, evolving-graph methods, and language-based representations, but these approaches have differing constraints on new relations, entities, or available information. INGRAM instead uses a weighted relation graph and returns entity and relation embeddings.

2. Related Work

  • Rule-mining and subgraph-reasoning methods generally assume relations are known; GraphANGEL handles new relations but assumes all entities are known.The cited approaches include Yang et al. (2017), Sadeghian et al. (2019), Teru et al. (2020), and Jin et al. (2022).Source: §2 · p. 2
  • 15 minutes versus 52 hours on NL-100: INGRAM processed the dataset faster than RMPI and achieved much better link-prediction performance.INGRAM uses a global weighted relation graph, whereas RMPI extracts a local subgraph for each candidate entity and uses an unweighted relation view.Source: §2 · p. 2
  • Unlike RMPI, INGRAM returns entity and relation embedding vectors that can also be used in downstream tasks.RMPI is described as designed for knowledge graph completion without computing embedding vectors.Source: §2 · p. 3
  • Methods for evolving graphs generally assume each triplet contains one known entity and one new entity, unlike INGRAM’s setting where all entities are new.The cited approaches include Hamaguchi et al. (2017), Wang et al. (2019), Dai et al. (2021), and Cui et al. (2022).Source: §2 · p. 3
  • Text-based approaches use descriptions or language models, which might not always be available; INGRAM uses knowledge-graph structure alone.The passage cites BERT-based representations and Zha et al. (2022) as examples.Source: §2 · p. 3

3. Problem Definition and Setting

The setting separates training and inference graphs, with disjoint entity sets and inference relations that need not have appeared during training. It specifies how facts are split for training, validation, and testing.

3. Problem Definition and Setting

  • The training graph partitions its triplets into observed facts Ftr and prediction targets Ttr, while the inference graph partitions its facts into Finf, Tval, and Ttest.The inference split uses a 3:1:1 ratio for facts, validation triplets, and test triplets.Source: §3 · p. 3
  • Training and inference entity sets are disjoint, but inference relations need not be a subset of training relations.This allows new relations to appear in the inference graph.Source: §3 · p. 3
  • The model trains on the training graph, uses the inference graph to compute embeddings and tune hyperparameters against Tval, and evaluates on Ttest.The use of Finf, Tval, and Ttest follows the cited prior setup.Source: §3 · p. 3

4. Defining Relation Graphs

The relation graph is a weighted graph whose nodes are relations and whose edge weights measure affinity. INGRAM derives those affinities from shared-entity structure, with degree normalization and a neighborhood intended for embedding aggregation.

4. Defining Relation Graphs

  • A knowledge graph consists of entity–relation–entity triplets, and the construction adds a reverse relation for each triplet.The graph notation uses V for entities, R for relations, and F for triplets.Source: §4 · p. 3
  • Relation affinity reflects how many entities two relations share and how frequently they share the same entity.The graph represents each relation as a node and each affinity as an edge weight.Source: §4 · pp. 3–4
  • Head- and tail-frequency matrices record how often each entity appears in each relation’s head or tail position.The entity degree is defined from these frequencies.Source: §4 · p. 4
  • Degree normalization makes the affinity weights contributed by each entity sum to one.The passage applies this normalization to both head- and tail-based affinity matrices.Source: §4 · p. 4
  • The relation graph’s adjacency matrix adds head- and tail-based affinities, and its entries indicate affinity between relation pairs.The graph is intended to provide reasonable neighborhoods for constructing relation embeddings, not a perfect set of semantically similar pairs.Source: §4 · p. 4

5. INGRAM: Inductive Knowledge Graph Embedding Model

InGram combines relation-level aggregation, entity-level aggregation, and relation–entity interaction modeling to build embeddings. Its training regime supports generating embeddings on an inference graph for missing-triplet prediction.

5. INGRAM: Inductive Knowledge Graph Embedding Model

  • InGram comprises relation-level aggregation, entity-level aggregation, and modeling of relation–entity interactions.Source: §5 · p. 4

5.1. Updating Relation Representation Vectors via Relation-Graph-Based Aggregation

  • Relation vectors are updated by attention aggregation over graph neighbors, with a learned term encoding globally binned affinity between relation pairs.Source: §5.1 · p. 4

5.2. Entity Representation by Entity-level Aggregation

  • Entity updates aggregate neighboring and self representations together with adjacent relation representations, and the formulation extends GATv2 by incorporating relation vectors at every step.Source: §5.2 · p. 5

5.3. Modeling Relation-Entity Interactions

  • A DistMult variant scores triplet plausibility, while training uses corrupted negative triplets and a margin-based ranking loss.Source: §5.3 · p. 5

5.4. Training Regime

  • Training randomly re-splits edges 3:1 each epoch, requiring the training portion to include a minimum spanning tree and cover all relations; feature vectors are reinitialized each epoch.Source: §5.4 · p. 6

5.5. Embedding of New Relations and Entities

  • At inference, InGram computes entity and relation embeddings on the new graph, then predicts a missing entity by selecting the candidate with the highest triplet score.Source: §5.5 · p. 6

6. Experimental Results

The evaluation spans 12 datasets across three benchmarks, with all inference entities new and the share of new relations varied. INGRAM leads in most settings, especially fully new-relation cases; the section also examines known-relation inference, ablations, and learned affinity weights.

6.1. Datasets and Baseline Methods

  • The study uses 12 datasets from NELL-995, Wikidata68K, and FB15K237, varying new-relation share from 25% to 100%, and compares 14 methods.All inference entities are new; settings with 100% new relations are inductive for relations, while the others mix new and known relations.Source: §6.1 · p. 6

6.2. Inductive Link Prediction

  • INGRAM significantly outperforms all baselines on NL-100, WK-100, and FB-100, where entities and relations are new.These are the most challenging datasets, and the reported gap over the best baseline is considerable across all metrics.Source: Table 1 · p. 7, §6.2 · p. 7
  • INGRAM significantly outperforms baselines across all NL and WK datasets and FB-75; its performance gap grows as the share of new relations increases.Source: §6.2 · p. 8
  • INGRAM performs better than baselines on 10 of 12 datasets, but NBFNet and RED-GNN outperform it on FB-25 and FB-50.The authors associate these exceptions partly with simple rules between known relations that rule-oriented methods can capture.Source: §6.2 · p. 8
  • On NL-50, INGRAM outperforms the best baseline across all metrics for both known-relation and new-relation triplets.All methods perform better on known-relation than new-relation triplets, with substantial gaps for baseline methods.Source: §6.2 · p. 8, Table 2 · p. 8

6.3. Inductive Link Prediction with Known Relations

  • On NL-0 and NELL-995-v1, INGRAM outperforms all baselines across metrics when all relations are known and all entities are new.The authors report reasonable performance in this setting despite not learning relation-specific patterns, alongside generalization to settings with new relations.Source: Table 3 · p. 8, §6.3 · p. 8

6.4. Ablation Studies of INGRAM

  • Removing any tested module noticeably degrades performance; the full model performs best with all modules together.The ablations cover aggregation choices, dynamic re-splitting, relation updates, and affinity weights.Source: Table 4 · p. 9, §6.4 · p. 9

6.5. Qualitative Analysis of INGRAM

  • For B=10, learned c_s(i,j) values generally decline as s(i,j) rises, matching the expected affinity ordering despite exceptions.Among B values 1, 5, and 10, INGRAM performs best with B=10, which the authors say shows the advantage of differentiating bins.Source: §6.5 · p. 9, Figure 4 · p. 9

7. Conclusion and Future Work

InGram targets settings where new entities accompany new relations and generates embeddings for both using only knowledge-graph structure. Its generalization focus leaves known-relation-specific patterns and robustness to noisy information as areas for future work.

Conclusion and Future Work

  • InGram generates embeddings for new relations and entities at inference time using only the structure of a given knowledge graph.It requires no extra entity or relation information or rich language models.Source: §7 · p. 9
  • InGram emphasizes generalization for new relations rather than patterns specific to known relations.The authors identify incorporating known-relation-specific patterns when known relations dominate as future work.Source: §7 · p. 9
  • The authors also identify robustness to potentially noisy knowledge-graph information as future work.They additionally plan theoretical analysis and extensions to hyper-relational facts and bi-level or hierarchical structures.Source: §7 · p. 9

A. Generating Datasets for Inductive Knowledge Graph Completion

The dataset procedure constructs disjoint training and inference entity sets, samples local neighborhoods, and controls the relations and triplets included in each graph. It also fixes inference evaluation splits and delays reverse-edge augmentation to avoid leakage.

A. Generating Datasets for Inductive Knowledge Graph Completion

  • The procedure samples training and inference entities with their two-hop neighbors, capping each hop at 50 neighbors per entity.It begins with the giant connected component and retains triplets and entities within the resulting connected components.Source: A. Generating Datasets for Inductive… · p. 13
  • Relations are randomly split between training and inference, while inference triplets mix training and inference relations according to ptri.Inference entities are sampled after removing training entities, and ntr, ninf, prel, and ptri control the construction.Source: A. Generating Datasets for Inductive… · p. 13
  • Algorithm 2 describes dataset generation, with dataset statistics and generation hyperparameters reported in Tables 5 and 6.The supplied table captions identify those tables’ topics.Source: A. Generating Datasets for Inductive… · p. 13
  • Inference triplets are divided into pairwise-disjoint Finf, Tval, and Ttest sets at a fixed 3:1:1 ratio shared across models.The split and use of training triplets are model-dependent.Source: A. Generating Datasets for Inductive… · p. 13
  • Reverse relations and triplets are added only after data splits, preventing leakage between training, validation, and test sets.The same ordering is used for the training triplet split.Source: A. Generating Datasets for Inductive… · p. 13

B. Details about the Baseline Methods

Baseline comparisons use original implementations with minimal changes, with some candidate-generation and feature-input adjustments for fairness. Several methods are omitted because their tasks, implementations, assumptions, or reproducibility do not fit the evaluation setting.

B. Details about the Baseline Methods

  • Baseline implementations generally use their authors’ original code and default settings, with minimal modifications and hardware varying by implementation.The experiments use GeForce RTX 2080 Ti, RTX 3090, or RTX A6000 GPUs.Source: B. Details about the Baseline Methods · p. 13
  • Candidate sets were adjusted for NeuralLP, DRUM, BLP, and RAILD to support fair inference comparisons.NeuralLP and DRUM exclude training entities; BLP and RAILD consider all inference entities as candidates.Source: B. Details about the Baseline Methods · p. 13
  • BLP, QBLP, and RAILD receive BERT-based pre-trained vectors where dataset text descriptions or names are available.NL has no descriptions, FB has entity descriptions only, and WK has descriptions for entities and relations.Source: B. Details about the Baseline Methods · p. 13
  • Baseline tuning varies by model, including learning rates, regularization, layer counts, dropout, and relation-context settings.The listed search settings cover BLP, QBLP, GraIL, CoMPILE, SNRI, RMPI, RED-GNN, CompGCN, and NodePiece.Source: B. Details about the Baseline Methods · pp. 13–14
  • Several baselines were omitted because their code or task did not support link prediction, inference candidates, disjoint entity sets, or reproducible results.The exclusions include PathCon, TACT, ConGLR, CBGNN, MEAN, LAN, and GraphANGEL.Source: B. Details about the Baseline Methods · p. 14

C. Hyperparameters of INGRAM

InGram is tuned over a range of embedding, layer, neighborhood, margin, binning, and learning-rate settings. The reported best performance uses B = 10, and separate tables document dataset and dataset-generation settings.

C. Hyperparameters of INGRAM

  • B = 10 gives InGram’s best observed performance, supporting the effectiveness of its binning strategy.The tuning search also varies d′, bd′, L, bL, K, bK, γ, and learning rate.Source: C. Hyperparameters of INGRAM · p. 14
  • InGram validation is performed every 200 epochs during 10,000 training epochs.The passage lists the ranges tested for embedding dimensions, layers, neighborhood settings, γ, B, and learning rate.Source: C. Hyperparameters of INGRAM · p. 14
  • Tables 5 and 6 are identified as reporting dataset statistics and dataset-generation hyperparameters, respectively.The supplied captions specify these topics but do not provide table values.Source: Table 5 · p. 14, Table 6 · p. 14

Related papers

Loading 2305.19987v3…