Source-linked AI summary
Hypernetwork Knowledge Graph Embeddings
Ivana Balažević, Carl Allen, Timothy M. Hospedales
TL;DR
Knowledge graphs are incomplete, making missing-link inference an important problem, while ConvE’s effective 2D convolution over embeddings is unintuitive. HypER generates relation-specific filters with a hypernetwork, achieves state-of-the-art results across multiple datasets, and frames convolution as sparse tensor factorization. The paper concludes that convolution offers a trade-off between expressiveness and parameter count, while noting extensions for applying filters to both entity embeddings.
Problem
Knowledge graphs contain missing links, and ConvE’s effective convolution over reshaped 1D entity and relation embeddings is unintuitive.
Method
HypER uses a hypernetwork to generate relation-specific 1D convolutional filters that process subject entity embeddings and share information across relations.
Results
HypER achieves state-of-the-art results across almost all metrics on multiple link-prediction datasets and is closely related to tensor factorization models.
Takeaways & Limitations
Convolution provides a good trade-off between nonlinear expressiveness and the number of parameters to learn through sparsity and parameter tying.
Takeaways & Limitations
Future work includes applying convolutional filters to both subject and object embeddings and studying label smoothing and feature-map interpretability.
Abstract
from arXiv · showhide
Knowledge graphs are graphical representations of large databases of facts, which typically suffer from incompleteness. Inferring missing relations (links) between entities (nodes) is the task of link prediction. A recent state-of-the-art approach to link prediction, ConvE, implements a convolutional neural network to extract features from concatenated subject and relation vectors. Whilst results are impressive, the method is unintuitive and poorly understood. We propose a hypernetwork architecture that generates simplified relation-specific convolutional filters that (i) outperforms ConvE and all previous approaches across standard datasets; and (ii) can be framed as tensor factorization and thus set within a well established family of factorization models for link prediction. We thus demonstrate that convolution simply offers a convenient computational means of introducing sparsity and parameter tying to find an effective trade-off between non-linear expressiveness and the number of parameters to learn.
1 Introduction
Knowledge graphs are incomplete fact databases, motivating link prediction, but existing approaches face a trade-off between expressiveness and overfitting. HypER addresses this with relation-specific convolutional filters and achieves state-of-the-art performance while connecting convolution to tensor factorization.
- Knowledge graphs contain triples of entities and relations, but many entity links are missing, motivating link prediction.
- Factorization-based link prediction models are shallow and linear, while added fully connected nonlinearities often lead to overfitting.
- HypER uses a hypernetwork to generate relation-specific convolutional filters for processing subject entity embeddings.This avoids ConvE’s reshaping and concatenation of subject and relation embeddings.
- HypER achieves state-of-the-art performance across standard link-prediction datasets.The evaluation includes FB15k-237, WN18RR, FB15k, WN18, and YAGO3-10.
- The benefit of convolution over fully connected layers comes from restricting interacting dimensions, while HypER remains within tensor factorization models.Convolution provides a trade-off between expressiveness and the number of learned parameters.
2 Related Work
Earlier link-prediction methods include matrix and tensor factorizations, translational models, and graph convolutions. HypER removes ConvE’s reshaping and concatenation, using relation-specific filters that combine convolutional sparsity with factorization-style interactions.
- RESCAL, DistMult, and ComplEx model link prediction through increasingly constrained matrix or tensor factorizations.TransE instead represents relations as translations between entity vectors.
- ConvE applies convolution to reshaped and concatenated subject and relation embeddings before scoring object entities.
- HypER avoids reshaping and concatenation by convolving subject embeddings with relation-specific filters generated by a hypernetwork.This combines entity and relation embeddings nonlinearly while reducing parameters.
- A 1D HypER convolution is equivalent to multiplication by a highly sparse tensor with tied weights, linking convolutional and factorization approaches.
3 Link Prediction
Link prediction learns scores for entity-relation-entity triples, converting them into probabilities for prospective facts. The section summarizes scoring functions, relation-parameter dimensions, and model space complexity.
- Link prediction learns a scoring function that assigns each triple a real-valued score indicating whether it represents a true fact.
- Table 1 compares scoring functions, relation-parameter dimensionality, and space complexity across state-of-the-art models.Entity and relation embedding dimensions are denoted by de and dr.
- Models map entity pairs to distributed embeddings and assign scores using relation-specific functions.
- The logistic sigmoid converts scores into probabilistically interpretable predictions between 0 and 1.
4 Hypernetwork Knowledge Graph Embeddings
HypER uses a hypernetwork to generate relation-specific convolutional filters, then scores subject–relation–object triples through projected feature maps and inner products. Its tensor formulation connects convolution with sparse, weight-tied relation transformations while using 1-N training to score all candidate objects simultaneously.
- Model architecture: HypER projects each relation embedding through a hypernetwork to produce relation-specific convolutional filter weights.The projected output is reshaped into a set of convolutional filters for processing subject entity embeddings.
- Scoring function: Subject embeddings are convolved with relation-specific filters, projected by W, passed through ReLU, and compared with every object embedding to produce triple probabilities.The model applies the logistic sigmoid element-wise to the resulting scores.
- Tensor interpretation: HypER can be represented through tensor operations in which convolution becomes multiplication by a sparse, diagonally duplicated tensor before combination with W.The resulting tensor product can be considered as generating an implicit relation-specific matrix.
- Training procedure: Training uses 1-N scoring with Adam to minimize binary cross-entropy loss over all candidate object entities, with label smoothing applied to the labels.1-N scoring evaluates (e1, r, E) simultaneously rather than training individual triples one at a time.
- Parameter comparison: 4.3M parameters are used by HypER versus 5.1M by ConvE on FB15k-237.The passage attributes the difference to HypER directly transforming relations into convolutional filters.
5 Experiments
HypER is evaluated on five standard knowledge-graph datasets using established ranking metrics and consistently achieves state-of-the-art link-prediction performance. Additional comparisons examine fairness, parameter count, the hypernetwork, filter dimensionality, and label smoothing.
- 5.1 Datasets: HypER is evaluated on FB15k-237, WN18RR, FB15k, WN18, and YAGO3-10 using standard link-prediction datasets.FB15k-237 and WN18RR remove inverse relations that can make prediction artificially easy.
- 5.2 Experimental Setup: The evaluation reports mean rank, mean reciprocal rank, hits@10, hits@3, and hits@1, favoring low mean rank and high reciprocal-rank and hits scores.Mean rank averages the true triple’s rank; the other metrics measure reciprocal rank or top-k inclusion.
- 5.3 Results: HypER consistently outperforms prior models across all datasets and metrics except mean reciprocal rank on WN18 and mean rank on four datasets.The exceptions are mean rank on WN18RR, FB15k-237, WN18, and YAGO3-10.
- 5.3 Results: With 200-dimensional embeddings and 1-N scoring, HypER remains the best-performing model overall despite improved tuning of competing models.The reimplementation was intended to provide a fair comparison with prior models.
- 5.3 Results: Matching ConvE’s parameter count does not improve ConvE, indicating that HypER’s advantage is not explained solely by having fewer parameters.ConvE was reduced from 32 to 16 feature maps for the comparison.
- 5.4 Analysis: Ablations show that the hypernetwork improves performance, 1D filters outperform reshaped 2D alternatives, and label smoothing has dataset-specific effects.Small convolutional filters extract most of the useful information; label smoothing improves WN18RR but harms FB15k.
6 Conclusion
HypER generates relation-specific convolutional filters through a hypernetwork, sharing information across relations and creating nonlinear entity–relation interactions. The authors connect this approach to tensor factorization and identify future extensions involving object embeddings and interpretability.
- 6 Conclusion: HypER generates relation-specific convolutional filters and applies them to subject entity embeddings for link prediction.Its hypernetwork shares information between relation vectors, enabling multi-task learning across relations.
- 6 Conclusion: HypER achieves state-of-the-art results across almost all metrics on multiple link prediction datasets while using relatively few parameters and resisting overfitting.The model is also described as fast and robust to overfitting.
- 6 Conclusion: Future work includes applying convolutional filters to both subject and object embeddings and studying label smoothing and convolutional feature-map interpretability.These directions aim to extend the architecture and improve insight into its learned representations.