Source-linked AI summary
Learning Attention-based Embeddings for Relation Prediction in Knowledge Graphs
Deepak Nathani, Jatin Chauhan, Charu Sharma, Manohar Kaul
TL;DR
Missing relations make knowledge-base completion necessary, while existing embeddings treat triples independently and miss latent neighborhood information. The paper extends graph attention to entity and relation features across multi-hop neighborhoods and relation clusters, reporting substantial improvements over state-of-the-art methods, including a 104% Hits@1 gain on FB15K-237.
Problem
Knowledge graphs contain missing relations, and existing translational and CNN-based embeddings process triples independently rather than capturing latent neighborhood information.
Method
The model uses generalized graph attention over entity and relation features in multi-hop neighborhoods, incorporating relation composition and semantically similar relation clusters.
Results
The model reports substantial improvements over state-of-the-art relation-prediction methods, including a 104% improvement in Hits@1 on FB15K-237.
Takeaways & Limitations
Attention-based embeddings provide a relation-prediction approach that captures neighborhood, multi-hop, and clustered-relation information in knowledge graphs.
Takeaways & Limitations
The authors identify hierarchical graphs and higher-order relations such as motifs as future extensions of the method.
Abstract
from arXiv · showhide
The recent proliferation of knowledge graphs (KGs) coupled with incomplete or partial information, in the form of missing relations (links) between entities, has fueled a lot of research on knowledge base completion (also known as relation prediction). Several recent works suggest that convolutional neural network (CNN) based models generate richer and more expressive feature embeddings and hence also perform well on relation prediction. However, we observe that these KG embeddings treat triples independently and thus fail to cover the complex and hidden information that is inherently implicit in the local neighborhood surrounding a triple. To this effect, our paper proposes a novel attention based feature embedding that captures both entity and relation features in any given entity's neighborhood. Additionally, we also encapsulate relation clusters and multihop relations in our model. Our empirical study offers insights into the efficacy of our attention based model and we show marked performance gains in comparison to state of the art methods on all datasets.
1 Introduction
Knowledge graphs often contain missing relations, motivating relation prediction; existing triple-independent embeddings miss latent neighborhood information. The paper proposes attention-based embeddings over entity and relation features, multi-hop neighborhoods, and relation clusters, reporting substantial gains over prior methods.
- Knowledge base completion predicts whether a given triple is valid when knowledge graphs contain missing relations.
- Translational and CNN-based models process triples independently, overlooking semantically rich and latent relations near each entity.
- The proposed generalized graph attention model attends to both entity and relation features in a given entity’s multi-hop neighborhood.
- The model captures multi-hop relations, diverse entity roles, and knowledge from semantically similar relation clusters.
- It assigns attention to neighborhood nodes, propagates attention iteratively, and adds composed auxiliary edges to preserve information flow across distant entities.The architecture uses the generalized graph attention model as an encoder and ConvKB as a decoder.
- 104% improvement over the state-of-the-art method is reported for Hits@1 on Freebase FB15K-237.
2 Related Work
Knowledge-graph relation-prediction methods include compositional, translational, CNN-based, and graph-based models. The paper positions its approach as holistically modeling multi-hop and semantically similar relations rather than treating triples independently or separating entity and relation features.
- Relation-prediction embeddings are broadly categorized as compositional, translational, CNN-based, and graph-based models.
- Compositional models capture rich interactions but can require many parameters, while HOLE offers more efficient circular-correlation representations.
- Translational models such as TransE, DISTMULT, and ComplEx are faster and easier to train but produce less expressive embeddings.
- CNN-based models use convolutional feature extraction for link prediction but consider each triple independently.
- R-GCN applies convolution to each entity’s neighborhood with equal weights and does not outperform CNN-based models.
- The proposed graph attention model holistically captures multi-hop and semantically similar relations in an entity’s n-hop neighborhood.
3 Our Approach
The model extends graph attention for knowledge graphs by jointly using entity and relation features, multi-hop neighborhoods, and auxiliary relations. Its architecture combines attention-weighted triple representations, multihead processing, residual initial entity information, and translational scoring.
- Graph Attention for Knowledge Graphs: The model extends GATs for knowledge graphs by incorporating relation features alongside entity features in attention.Unlike standard GATs, which focus on node features, the proposed layer represents each associated triple using the embeddings of its source entity, relation, and neighboring entity.
- Graph Attention for Knowledge Graphs: Each graph-attention layer takes entity and relation embedding matrices as input and produces updated entity and relation representations.Entity and relation embeddings are maintained as separate matrices, enabling the layer to process both feature types.
- Attention Mechanism: Attention weights prioritize triple representations, with relative attention coefficients normalized over the relations connecting neighboring entities.The entity update sums triple representations weighted by their attention values, and multihead attention concatenates independent attention outputs before final-layer averaging.
- Decoder and Embedding Preservation: The architecture adds transformed initial entity embeddings to final attentional embeddings and uses a translational L1 scoring function for relation prediction.The residual addition preserves initial entity information, while scoring uses dtij = ∥hi + gk − hj∥1.
- Multi-hop Relations: The model introduces auxiliary relations between n-hop neighbors to propagate information from distant entities and represent directed paths.An auxiliary relation embedding is formed by summing the embeddings of relations along the path, while stacked layers iteratively accumulate neighborhood information.
4 Experiments and Results
The experiments evaluate relation prediction across five benchmark datasets and analyze performance, attention learning, graph density, and component ablations. The model outperforms state-of-the-art methods on multiple metrics, while relation information and multi-hop paths materially affect results.
- Datasets and evaluation: Five benchmark datasets—WN18RR, FB15k-237, NELL-995, UMLS, and Alyawarra Kinship—are used to evaluate relation prediction.The evaluation includes datasets with known inverse-relation concerns, particularly WN18 and FB15K.
- Datasets and evaluation: Relation prediction ranks the correct triple among N − 1 corrupted triples generated by replacing the head or tail entity.Invalid triples are sampled equally from head-replacement and tail-replacement sets to evaluate both prediction directions.
- Prediction results: The proposed method significantly outperforms state-of-the-art results on five metrics for FB15k-237 and two metrics for WN18RR.Results are reported in Tables 2 and 3 for the benchmark test sets.
- Attention analysis: As training progresses, attention shifts from random allocation toward direct neighbors while retaining information from distant neighbors.After convergence, the model gathers multi-hop and clustered relation information from the node’s n-hop neighborhood.
- Graph-density analysis: Mean PageRank and relative MRR improvement show a strong correlation of r = 0.808, although WN18RR’s sparse hierarchical structure challenges the method.The analysis associates higher PageRank values with corresponding increases in MRR relative to DistMult.
- Ablation study: The model’s performance exceeds both ablated variants that remove path generalization or relation information, with a significant drop on NELL-995.Removing relation information has a particularly large impact, suggesting relation embeddings play a pivotal role in relation prediction.
5 Conclusion and Future Work
The paper presents attention-based graph embeddings for relation prediction and reports improvement over state-of-the-art models. It also identifies future extensions to hierarchical graphs and higher-order relations.
- Conclusion: Figure 7 compares mean rank across epochs for the full model and models ablated by removing path generalization or relation features.The caption identifies the full model and the −PG and −Relations ablations.
- Conclusion: The proposed graph attention-based embeddings target relation prediction and capture entity and relation features in multi-hop neighborhoods.The approach is presented as a novel method for relation prediction on knowledge graphs.
- Future Work: The authors intend to extend the method to hierarchical graphs and higher-order relations such as motifs.These extensions are stated as future work.