Source-linked AI summary
Modeling Relational Data with Graph Convolutional Networks
Michael Schlichtkrull, Thomas N. Kipf, Peter Bloem, Rianne van den Berg, Ivan Titov, Max Welling
TL;DR
Knowledge bases remain incomplete, motivating methods that recover missing facts and entity attributes. This paper introduces relational graph convolutional networks for both tasks and shows that adding an R-GCN encoder to DistMult improves link prediction by 29.8% on FB15k-237 over a decoder-only baseline.
Problem
Knowledge bases remain incomplete, creating a need to recover missing facts and entity attributes for applications such as question answering and information retrieval.
Method
The paper introduces R-GCNs for relational graphs and combines their multi-step information propagation with DistMult decoding for link prediction.
Results
29.8% improvement over the decoder-only baseline on FB15k-237, while R-GCNs achieve competitive results for entity classification and link prediction.
Takeaways & Limitations
Explicit neighborhood modeling is beneficial for recovering missing knowledge-base facts, especially when an R-GCN encoder enriches a factorization decoder.
Takeaways & Limitations
The fixed normalization used to aggregate neighboring-node messages may be problematic for high-degree nodes, motivating data-dependent attention weights.
Abstract
from arXiv · showhide
Knowledge graphs enable a wide variety of applications, including question answering and information retrieval. Despite the great effort invested in their creation and maintenance, even the largest (e.g., Yago, DBPedia or Wikidata) remain incomplete. We introduce Relational Graph Convolutional Networks (R-GCNs) and apply them to two standard knowledge base completion tasks: Link prediction (recovery of missing facts, i.e. subject-predicate-object triples) and entity classification (recovery of missing entity attributes). R-GCNs are related to a recent class of neural networks operating on graphs, and are developed specifically to deal with the highly multi-relational data characteristic of realistic knowledge bases. We demonstrate the effectiveness of R-GCNs as a stand-alone model for entity classification. We further show that factorization models for link prediction such as DistMult can be significantly improved by enriching them with an encoder model to accumulate evidence over multiple inference steps in the relational graph, demonstrating a large improvement of 29.8% on FB15k-237 over a decoder-only baseline.
1 Introduction
Knowledge bases support applications such as question answering and information retrieval but remain incomplete despite extensive maintenance. The paper introduces R-GCNs for link prediction and entity classification, combining relational graph encoding with DistMult for knowledge base completion.
- Knowledge bases enable question answering and information retrieval, yet even large resources such as DBPedia, Wikidata, and Yago remain incomplete.
- The paper addresses link prediction and entity classification, recovering missing triples and assigning types or categorical properties to entities.Both tasks exploit information encoded in graph neighborhoods.
- R-GCNs produce node representations for softmax-based entity classification, with model parameters learned by optimizing cross-entropy loss.
- The link prediction model uses an R-GCN encoder and a DistMult decoder, forming an autoencoder that predicts labeled edges.
- The R-GCN-enhanced method achieves competitive benchmark results and outperforms direct optimization of vanilla DistMult, especially on the more challenging FB15k-237.
- R-GCNs are introduced as a framework for relational data, with parameter sharing and sparsity techniques enabling application to multigraphs with many relations.
2 Neural relational modeling
R-GCNs extend graph convolutional and message-passing architectures to directed, labeled multi-relational graphs by aggregating normalized, relation-specific neighbor transformations across stacked layers. Basis and block-diagonal decompositions regularize relation-specific weights and reduce parameter growth in highly multi-relational data.
- Graph representation: R-GCNs model directed, labeled multi-graphs whose edges are relation-typed triples between entity nodes.The relation set includes both canonical and inverse directions, such as “born in” and “born in inv.”
- Propagation model: Each R-GCN update accumulates transformed neighbor features through a normalized sum, using transformations specific to each relation type and direction.A special self-connection preserves information from the node’s previous layer.
- Propagation model: Stacking multiple layers enables R-GCNs to capture dependencies across several relational steps, with updates efficiently implemented using sparse matrix multiplications.The same layer update is evaluated in parallel for every node in the graph.
- Parameter regularization: Relation-specific weights can grow rapidly with the number of relations, risking overfitting on rare relations and producing very large models.This parameter-growth problem is central when applying the propagation rule to highly multi-relational data.
- Parameter regularization: R-GCNs introduce basis and block-diagonal decompositions to regularize layer weights and reduce the number of parameters.Basis decomposition shares transformations through relation-dependent coefficients, whereas block decomposition imposes structured sparsity on relation-specific matrices.
3 Entity classification
For entity classification, R-GCN layers are stacked with a per-node softmax output, and training minimizes cross-entropy over labeled nodes while ignoring unlabeled nodes. The model is trained with full-batch gradient descent.
- Entity classification: R-GCN layers are stacked for node classification, with a per-node softmax activation applied to the final layer’s output.
- Entity classification: Cross-entropy loss is minimized over labeled nodes, while unlabeled nodes are ignored.Y denotes the set of labeled node indices; t_ik is the ground-truth label for node i.
- Entity classification: The model is trained using full-batch gradient descent, with its architecture schematically depicted in Figure 3a.
4 Link prediction
Link prediction scores candidate subject–relation–object triples to recover missing facts in an incomplete knowledge graph. The model combines an R-GCN entity encoder with a DistMult decoder, trained using corrupted negative examples.
- Task definition: Link prediction assigns scores to possible subject–relation–object triples to estimate which missing edges belong to the knowledge graph.The graph is represented as G = (V, E, R), but only an incomplete edge subset is observed.
- Model architecture: The proposed graph auto-encoder maps each entity to a real-valued vector and uses a decoder to score triples from those representations.The encoder produces ei ∈ R^d, while the decoder scores triples through a function s : R^d × R × R^d → R.
- Model architecture: Unlike approaches that directly optimize one vector per entity, the model computes entity representations through an R-GCN encoder.The representations are computed as ei = h(L)i, following the graph auto-encoder perspective.
- Decoder and training: The experiments use DistMult as the scoring function, with each relation represented by a diagonal matrix.DistMult is selected because it performs well on standard link prediction benchmarks when used independently.
- Decoder and training: Training uses negative sampling by randomly corrupting either the subject or object of each observed positive triple and optimizing cross-entropy loss.The objective encourages observable triples to receive higher scores than corrupted negatives.
5 Empirical evaluation
The empirical evaluation shows that R-GCNs perform competitively for entity classification and improve link prediction when used as encoders alongside factorization decoders. Results also identify fixed message normalization as a limitation and motivate attention-based aggregation and alternative scoring functions.
- Entity classification: R-GCN achieves state-of-the-art entity-classification results on AIFB and AM.The evaluation uses RDF datasets AIFB, MUTAG, BGS, and AM, with accuracy reported on established train/test splits.
- Entity classification: MUTAG and BGS are challenging because labeled entities connect only through high-degree hubs encoding features or feature hierarchies.These datasets represent molecular graphs and rock types converted to RDF, respectively.
- Entity classification: Fixed normalization may harm performance on high-degree nodes, motivating data-dependent attention weights for message aggregation.The proposed future direction replaces 1/c_i,r with attention weights a_ij,r satisfying their relation-specific normalization constraint.
- Link prediction: On FB15k and WN18, R-GCN and R-GCN+ outperform DistMult but underperform LinkFeat, while R-GCN+ exceeds ComplEx on FB15k.The strong LinkFeat result reflects the contribution of inverse relation pairs, and combining the encoder with ComplEx is suggested as future work.
6 Related Work
R-GCNs adapt graph convolutional and message-passing neural networks to large-scale, highly multi-relational knowledge bases. The link-prediction model uses DistMult as a decoder and relates to prior factorization and path-based approaches.
- Knowledge-base completion: The link-prediction decoder relies on DistMult, a simpler special case of RESCAL that is more effective than original RESCAL for multi-relational knowledge bases.The passage also notes numerous alternative bilinear, nonlinear, and combined factorizations in statistical relational learning.
- Knowledge-base completion: Prior path-based knowledge-base methods create auxiliary triples, use paths or walks as prediction features, or combine both strategies.These strategies incorporate paths between entities into factorization objectives or edge prediction.
- Graph neural networks: R-GCNs adapt prior graph convolutional networks for large-scale, highly multi-relational data characteristic of realistic knowledge bases.The encoder is closely related to earlier graph neural network work.
- Graph neural networks: Earlier graph neural network research includes Scarselli et al. (2009), while later extensions use gating mechanisms to facilitate optimization.The cited gated extensions are Li et al. (2016) and Pham et al. (2017).
- Graph neural networks: R-GCNs are a subclass of message passing neural networks, which interpret graph models, including GCNs, through differentiable message passing.This places R-GCNs within a broader framework encompassing several previous neural models for graphs.
7 Conclusions
The paper introduces R-GCNs for link prediction and entity classification, demonstrating competitive graph-based encoding and improved link prediction with DistMult decoding. It also identifies extensions involving alternative factorization models, entity features, and scalability.
- Conclusions: R-GCNs are introduced for two statistical relation modeling problems: link prediction and entity classification.
- Conclusions: For entity classification, R-GCNs provide a competitive, end-to-end trainable graph-based encoder.
- Conclusions: For link prediction, combining an R-GCN encoder with DistMult decoding outperformed direct factorization optimization and achieved competitive benchmark results.
- Conclusions: Potential extensions include combining graph autoencoders with ComplEx, integrating entity features, and exploring improved scalability.ComplEx may better model asymmetric relations, while entity features could benefit both link prediction and entity classification.
Further experimental details on entity classification
The entity-classification experiments were rerun under a uniform evaluation protocol using canonical splits, training-only hyperparameter optimization, and one final test evaluation. The supplementary details specify baseline preprocessing and implementations, R-GCN hyperparameters, and execution conditions.
- Evaluation protocol: Baselines were rerun on the canonical train/test split, with hyperparameters optimized only on training data before a single test evaluation.This uniform protocol explains differences from previously reported numbers.
- Baseline implementations: WL and RDF2Vec use Mustard-based implementations, extract explicit instance-node features, and classify them with a linear SVM.WL uses the tree variant of the Weisfeiler-Lehman subtree kernel; RDF2Vec uses the authors’ implementation.
- Dataset preprocessing: MUTAG preprocessing removes only triples encoding the target relation, avoiding label leakage from other boolean relations that remain present.Earlier preprocessing removed all triples connecting the subject and object, allowing the boolean label to be inferred indirectly.
- R-GCN configuration: R-GCN selects the first-layer l2 penalty and basis-function count on validation performance; block decomposition does not improve results.The tested choices are Cl2 ∈ {0, 5 · 10^-4} and B ∈ {0, 10, 20, 30, 40}, with B = 0 meaning no basis decomposition.
- R-GCN configuration: R-GCN otherwise uses 50 epochs and 16 hidden units, omits dropout, and uses 10 hidden units for AM to reduce memory usage.The normalization constant is ci,r = |Nr_i|.