Source-linked AI summary
Logic Attention Based Neighborhood Aggregation for Inductive Knowledge Graph Embedding
Peifeng Wang, Jialong Han, Chenliang Li, Rong Pan
TL;DR
Existing KG embedding methods generally require entities to be observed during training, creating a challenge for emerging entities and motivating inductive use of existing neighborhoods. The paper proposes Logic Attention Network, which uses permutation-invariant, relation- and neighbor-aware attention with logic rules and neural attention. Experiments on two KG completion tasks show that LAN significantly outperforms baseline aggregators, within the paper’s evaluated setting.
Problem
Existing KG embedding methods require all entities during training, while emerging entities make retraining from scratch impractical for timely missing-fact inference.
Method
LAN embeds emerging entities by permutation-invariant weighted aggregation, estimating weights with coarse relation-level logic rules and fine neighbor-level neural attention.
Results
LAN significantly outperforms baseline models on two typical KG completion tasks.
Takeaways & Limitations
Effective inductive KG aggregation should account for unordered neighbors, neighborhood redundancy, and the query relation.
Takeaways & Limitations
The evaluation focuses on predicting missing relations between unseen entities and existing entities.
Abstract
from arXiv · showhide
Knowledge graph embedding aims at modeling entities and relations with low-dimensional vectors. Most previous methods require that all entities should be seen during training, which is unpractical for real-world knowledge graphs with new entities emerging on a daily basis. Recent efforts on this issue suggest training a neighborhood aggregator in conjunction with the conventional entity and relation embeddings, which may help embed new entities inductively via their existing neighbors. However, their neighborhood aggregators neglect the unordered and unequal natures of an entity's neighbors. To this end, we summarize the desired properties that may lead to effective neighborhood aggregators. We also introduce a novel aggregator, namely, Logic Attention Network (LAN), which addresses the properties by aggregating neighbors with both rules- and network-based attention weights. By comparing with conventional aggregators on two knowledge graph completion tasks, we experimentally validate LAN's superiority in terms of the desired properties.
Introduction
Inductive KG embedding addresses emerging entities that appear after training by using their existing neighbors, but requires KG-specific aggregation that respects neighborhood structure. LAN combines permutation-invariant aggregation with relation- and neighbor-level attention informed by redundancy and query relations.
- Motivation: Conventional KG embedding models require all entities during training, making retraining impractical as new entities emerge.DBpedia reportedly saw 200 new entities daily between late 2015 and early 2016.
- Motivation: Inductive neighborhood aggregation can embed unseen entities by aggregating embeddings of their existing neighbors.This approach was developed for homogeneous graphs and motivates its extension to multi-relational KGs.
- Desired Properties: An effective KG aggregator should be permutation invariant and redundancy aware, while using the query relation to focus on relevant neighborhood facts.Neighbors are unordered, and dependent facts can provide informative redundancy.
- Logic Attention Network: LAN aggregates transformed neighbor embeddings with attention weights estimated from relation-level logic rules and fine-grained neural attention.Both mechanisms account for neighborhood redundancy and the query relation in a coarse-to-fine manner.
- Evaluation: The paper compares LAN with conventional aggregators on two KG completion tasks and reports superiority with respect to the proposed properties.The stated contributions include three desired properties, LAN, and extensive comparisons.
Related Works
Related work addresses emerging entities through text, images, or neighborhood aggregation, but prior neighborhood pooling can overlook differences among neighbors. The paper positions LAN as a KG-specific alternative to these approaches.
- Transductive Embedding: Conventional KG embedding approaches are transductive because they require every entity to be seen during training.This limitation motivates methods for handling emerging entities.
- Text- and Image-Based Models: Some inductive KG models use entity description text or images, but their ability to infer implicit or new facts beyond those inputs is unclear.The passage identifies this as an unresolved issue for text- and image-based embeddings.
- Neighborhood Aggregation: Hamaguchi et al. apply a graph neural network that embeds a new entity by aggregating its known neighbors, but simple pooling neglects differences among neighbors.Other cited graph methods target node classification with the entire graph rather than this inductive KG setting.
- Homogeneous-Graph Methods: Homogeneous-graph studies also use node attributes or neighborhood aggregation to represent unseen nodes, but their setting differs from multi-relational KGs.The paper identifies Hamilton, Ying, and Leskovec as especially related for inductive neighborhood aggregation.
Preliminaries
The paper formulates inductive KG embedding around a neighborhood aggregator that converts an entity’s relational neighborhood into an embedding usable for predicting unknown facts. It adopts an encoder-decoder framework, transforming neighbor embeddings before aggregation and scoring triplets with a decoder.
- A knowledge graph is a set of triplet facts over entity and relation sets, with reverse-relation triplets added for each fact.
- An entity’s neighborhood contains related entities paired with their involved relations, with separate projections for neighboring entities and relations.
- The aggregator embeds an entity from its neighborhood and uses the resulting subject and object embeddings to assess unknown triplet plausibility, including for emerging entities.
- Encoder-decoder framework: The encoder-decoder framework trains the aggregator through triplet scoring: the encoder produces entity embeddings, while the decoder measures triplet plausibility and provides feedback.The decoder mainly uses a TransE-based scoring function and also evaluates alternative scoring functions.
- Encoder: LAN’s encoder transforms each neighbor embedding with a relation-specific function before aggregating the transformed vectors into the target entity embedding.The transformation uses a unit relation vector and avoids matrix products for lower computational complexity.
- Conventional aggregators: Mean-, sum-, and max-pooling are permutation-invariant but treat neighbors equally, ignoring neighborhood redundancy and query relations; RNN aggregation instead requires random neighbor ordering.
Logic Attention Network
LAN is a KG-specific neighborhood aggregator designed to address the limitations of traditional aggregation by using permutation-invariant, relation-aware attention over neighbors. It combines logic-rule and neural-network mechanisms and trains with positive and corrupted negative triplets, plus an input-embedding subtask.
- LAN addresses the desired properties that traditional neighborhood aggregators do not preserve.
- LAN preserves permutation invariance while using neighborhood and query-relation information to estimate attention weights.The aggregation is weighted over transformed neighbor embeddings, with each weight specified for a neighbor and query relation.
- The logic-rule mechanism assigns higher weight to relations statistically relevant to the query and lower weight to relations implied by other neighborhood relations.This mechanism addresses query-relation awareness and neighborhood redundancy awareness.
- The neural-network mechanism uses transformed neighbor embeddings and query-relation parameters to identify important individual neighbors.It complements relation-level logic rules with finer-grained information about specific neighbor entities.
- LAN combines the two weighting mechanisms through double-view attention to measure neighbor importance.The two mechanisms answer which relation types are useful and which specific neighbors contain important information.
- Training uses knowledge-graph triplets as positives, corrupted subject-or-object triplets as negatives, and an auxiliary input-embedding ranking subtask.The overall objective combines the output-embedding and input-embedding losses.
Experimental Configurations
The experiments evaluate LAN against MEAN and LSTM on inductive knowledge-graph completion settings containing entities unseen during training. The evaluation uses triplet classification and link prediction datasets constructed or selected to provide existing neighbors for unseen entities.
- LAN is evaluated on link prediction and triplet classification against MEAN and LSTM aggregators.MEAN represents pooling functions, while LSTM is selected for its expressive capability.
- The datasets require test sets containing new entities unseen during training.Triplet classification uses datasets based on WordNet11, while link-prediction datasets are constructed from FB15K.
- The link-prediction construction samples R = {5%, 10%, 15%, 20%, 25%} of original testing triplets for inductive test sets.Subject and Object strategies separately add entities appearing in the corresponding test-set positions to the candidate unseen set.
- Entities without neighbors in the original training set are filtered out, and test triplets involving two unseen or two existing entities are removed.
- The original training data are split into a new training set and an auxiliary set supplying existing neighbors for unseen entities.The resulting 2 × 5 = 10 datasets use the Subject and Object strategies.
Experiments on Triplet Classification
The experiments compare inductive completion performance across triplet classification and link prediction settings. LAN consistently performs best, while LSTM performs worst or more poorly than MEAN, supporting the importance of permutation-invariant aggregation.
- Triplet classification labels validation and test triplets as true or false, while training triplets are all true.
- Link prediction evaluates relations between unseen entities and existing entities rather than relations between two unseen entities.The construction omits a Both strategy because the task predicts unseen entities’ missing relations with existing entities.
- LAN outperforms the other baselines significantly on link prediction, especially on Hit@k metrics.The improvement on MR may be less considerable because MR is more sensitive to lower ranking positions.
- LAN achieves the best results on all triplet-classification datasets, while LSTM performs worse than MEAN.The comparison is reported in Table 2, whose metric is evaluation accuracy.
Experiments on Link Prediction
LAN consistently outperforms conventional aggregators on inductive link prediction, while ablations and robustness studies examine why it works and where it generalizes. Its attention weights emphasize query-relevant and informative neighbors, and its performance degrades less severely as knowledge graphs become sparser.
- The link-prediction evaluation ranks ground-truth entities among candidates using MR, MRR, and Hits@k after filtering fake negatives.Subject-R and Object-R hide the object or subject of each testing triplet, respectively, then rank candidate entities.
- LAN significantly outperforms baseline aggregators on Subject-10 and Object-10, especially on Hit@k and MRR metrics.Its improvement on MR is less considerable because MR is more sensitive to lower ranking positions.
- Necessity of Query Relation Awareness: Logic Rules Only outperforms MEAN and Query-Attention, while combining logic rules with neural attention enables LAN to outperform all other models.These results support using logic rules to assign meaningful neighbor weights rather than relying solely on neural networks.
- Generalization to Other Scoring Functions: Across different scoring functions, LAN consistently outperforms MEAN by a large margin on all evaluation metrics.TransE produces the best results for both MEAN and LAN.
- Influence of the Proportion of Unseen Entities: As the proportion of unseen entities increases, performance declines for all models, but LAN decreases less drastically than MEAN and LSTM.This indicates greater robustness on sparse knowledge graphs.
- Neighbor Weight Analysis: LAN assigns higher weights to neighbors whose relations are relevant to the query and to entities that provide more informative evidence.For example, relation-specific neighbors receive higher weights for origin, while Aristotle, Metaphysics, and Aesthetics receive high weights for profession.
Conclusion
The paper addresses inductive KG embedding by proposing LAN, which assigns permutation-invariant, query-aware weights to neighbors using logic rules and neural attention. Experiments show that LAN significantly outperforms baseline models on two KG completion tasks.
- LAN addresses inductive KG embedding by assigning different weights to an entity’s neighbors in a permutation-invariant manner.Its weights consider both neighbor redundancy and the query relation.
- LAN estimates attention weights with logic rules at a coarse relation level and a neural attention network at a fine neighbor level.
- LAN significantly outperforms baseline models on two typical KG completion tasks.