Source-linked AI summary
Relational Graph Attention Networks
Dan Busbridge, Dane Sherburn, Pietro Cavallo, Nils Y. Hammerla
TL;DR
The paper asks whether relational graph attention can extend graph attention beyond spectral relational models. It develops and evaluates RGAT variants against benchmarks and re-tuned RGCNs, finding competitive-or-poor performance overall with task-dependent exceptions. The authors identify evaluation and modelling considerations for future work.
Problem
RGCNs perform relational graph learning but remain spectral and share spectral deficiencies, motivating relational models away from spectral methods.
Method
The paper evaluates Within-Relation and Across-Relation Graph Attention variants with additive or multiplicative attention against established benchmarks and equivalently evaluated RGCNs.
Results
RGATs perform competitively or poorly overall; multiplicative ARGAT benefits relational inductive graph classification, while transductive relational tasks favor spectral methods such as RGCNs or WL graph kernels.
Takeaways & Limitations
WIRGAT generally pairs better with additive logits on transductive tasks, whereas ARGAT generally pairs better with multiplicative logits on inductive tasks.
Takeaways & Limitations
On MUTAG, RGCN already memorises the training set, so a more complex model that also memorises it may generalise less well; learned node embeddings may also provide insufficient signal for attention.
Abstract
from arXiv · showhide
We investigate Relational Graph Attention Networks, a class of models that extends non-relational graph attention mechanisms to incorporate relational information, opening up these methods to a wider variety of problems. A thorough evaluation of these models is performed, and comparisons are made against established benchmarks. To provide a meaningful comparison, we retrain Relational Graph Convolutional Networks, the spectral counterpart of Relational Graph Attention Networks, and evaluate them under the same conditions. We find that Relational Graph Attention Networks perform worse than anticipated, although some configurations are marginally beneficial for modelling molecular properties. We provide insights as to why this may be, and suggest both modifications to evaluation strategies, as well as directions to investigate for future work.
1 Introduction
Graph learning extends neural methods beyond Euclidean data, but relational convolution remains spectral and inherits spectral limitations. This paper introduces relational graph attention models, evaluates two attention designs against benchmarks and re-tuned RGCNs, and finds weaker-than-expected performance with limited task-specific benefits.
- Motivation: Graphs and manifolds motivate Geometric Deep Learning because many data types, including knowledge bases, molecules, and point clouds, are non-Euclidean.GDL includes spectral, spatial, and hybrid approaches.
- Limitations of existing approaches: Spectral approaches depend on the chosen basis, while spatial approaches lack shift invariance and a coordinate system.Hybrid methods trade advantages and deficiencies from both families.
- Research gap: RGCNs extend GCNs to relational graphs but remain spectral, motivating generalisations away from their spectral origins.RGCNs had achieved strong performance on node classification and link prediction, while retaining spectral deficiencies.
- Approach: RGATs extend graph attention to relational graphs through Within-Relation and Across-Relation variants with additive or multiplicative attention.The study performs extensive hyperparameter searches and evaluates transductive node classification and inductive graph classification.
- Main finding: RGATs perform worse than expected, although some configurations provide marginal benefits on inductive graph classification tasks.The study compares them with established benchmarks and a re-tuned RGCN under the same evaluation conditions.
2 RGAT architecture
RGAT extends graph attention to relational graphs by using relation-specific transformations, attention logits, and neighborhood aggregation. The architecture includes two softmax-normalization variants, additive or multiplicative logits, multi-head attention, basis decomposition, and task-specific output networks.
- Relational graph attention layer: RGAT layers transform node features separately by relation before computing attention over relation-specific neighborhoods.Each node has feature vector h_i, while relation-specific intermediate representations are produced using learnable transformations W^(r).
- Attention logits: RGAT supports additive logits derived from the relational modification of GAT attention and multiplicative logits based on query-key products.The additive mechanism uses scalar query and key representations, whereas the multiplicative mechanism permits any positive query-key dimensionality D.
- WIRGAT: WIRGAT normalizes attention independently within each relation, so nodes compete only with neighbors connected by the same relation.This implements a separate probability distribution over each relation’s neighborhood.
- ARGAT: ARGAT normalizes attention across relation-specific neighborhoods, creating one probability distribution over neighboring representations regardless of relation type.ARGAT therefore encodes relation importance as a local property of the graph.
- Propagation: Attention coefficients weight relation-specific neighborhood sums, which are aggregated and passed through an optional nonlinearity to update node representations.Multiple attention heads can enhance performance, and RGCN-style equally weighted coefficients provide the comparison mechanism.
- Parameterization: Basis decomposition is applied to transformation and attention kernels because parameter counts grow linearly with relation types and heads, potentially causing overparameterization.The paper evaluates both full and decomposed kernels, while noting that decomposition can improve generalization at the cost of increased model bias.
3 Evaluation
The evaluation tests RGAT variants on transductive RDF node classification and inductive molecular graph classification, using retrained RGCN comparisons and extensive hyperparameter searches. RGAT attention yields limited or configuration-dependent benefits, while RGCN remains stronger on MUTAG and some apparent gains are attributable to search or task characteristics.
- Evaluation design: Models are tuned with cross-validation, retrained on full training data, and evaluated across 200 seeds for transductive tasks; inductive tuning uses benchmark splits and cross-validation.Constant-attention counterparts are also evaluated to isolate attention’s contribution.
- Transductive learning: On AIFB, additive WIRGAT ranks first, while multiplicative ARGAT ranks second and RGCN third; every listed comparison is significant.The reported ordering is additive WIRGAT, multiplicative ARGAT, RGCN, additive ARGAT, then multiplicative WIRGAT.
- Transductive learning: On MUTAG, RGCN ranks first and consistently outperforms RGAT; attention changes relative mean performance by +0.66% for additive WIRGAT, +2.90% for additive ARGAT, and -1.63% for multiplicative ARGAT.The reported ranking places multiplicative ARGAT second, additive and multiplicative WIRGAT tied third, and additive ARGAT fourth.
- Interpretation: The authors attribute weak RGAT generalisation partly to training-set memorisation, insufficient signal for learning both node embeddings and attention, and objectives that do not encourage RGCN-like behaviour.No harshly regularised RGAT configuration generalised well on MUTAG.
- Inductive learning: On Tox21, multiplicative WIRGAT and ARGAT marginally outperform RGCN without significance, while attention improves their constant counterparts by 3.33% and 4.36%.The nonsignificant comparisons against RGCN have p = 0.24 and p = 0.41, respectively; additive models match RGCN.
4 Conclusion
RGATs extend graph attention with relational structure, but their performance is competitive or poor and strongly task-dependent. The evaluation finds no guaranteed significant advantage over RGCN and identifies training-set memorisation as a practical warning sign.
- RGATs apply masked self-attention that uses local relational structure and node features to assign relation-specific importance dynamically.
- The study evaluates WIRGAT and ARGAT with additive and multiplicative logits against equivalently evaluated RGCNs.
- RGAT performance is task-dependent: multiplicative ARGAT benefits inductive graph classification, while RGCN or WL is preferable for transductive relational tasks without node features.
- WIRGAT generally pairs better with additive logits and performs marginally better on transductive tasks, whereas multiplicative ARGAT performs marginally better on inductive tasks.
- No RGAT variant is guaranteed to significantly outperform RGCN; when RGCN memorises the training set, RGAT is expected to perform worse.
- Future studies should use larger, more challenging datasets because model variance is high across the reported tasks, and compare spectral methods with recurrent graph models.
A Tox21 Results
This section reports training, validation, and test performance for the models on the Wu et al. splits and summarizes graph-classification AUC across 12 tasks.
- The evaluation reports training, validation, and test performance for the models in addition to results from Wu et al. (2018).
- Table 3 summarizes graph-classification mean Area Under the Curve (AUC) across all 12 tasks over three splits.
B Hyperparameters
Hyperparameters were optimized separately for transductive and inductive tasks using specified search priors, while multihead configurations preserve a fixed total output width.
- Hyperparameter optimization uses Hyperopt with priors specified separately for transductive and inductive tasks.
- Table 4 defines the transductive-task hyperparameter search space and reduces units per head in multihead RGAT layers to keep total output units constant.
- Table 5 defines the inductive-task hyperparameter space, with batch size fixed at 64 and no bases decomposition.
C Charts
Figure 4 provides a chart representation of the Table 2 results, comparing entity-classification accuracy for established baselines, RGCN, and attention variants.
- Figure 4 charts the Table 2 results to aid interpretation.
- The chart compares entity-classification accuracy for FEAT, WL, RDF2Vec, RGCN, the authors’ RGCN implementation, and additive and red attention variants.
D Cumulative distribution functions
CDFs evaluate model performance across the full hyperparameter space rather than relying only on tuned configurations. Results vary by task: RGCN and WIRGAT are competitive on AIFB, ARGAT is less sensitive on MUTAG, and multiplicative attention is usually weaker on TOX21.
- CDF methodology: CDFs treat hyperparameter-search performance as an empirical distribution, enabling architecture comparisons across the whole search space.This avoids relying on tuned hyperparameters that may be outliers for an architecture’s generic behavior.
- Figure 5: Figure 5 compares CDFs for RGCN, ARGAT, and WIRGAT using solid additive-attention and dashed multiplicative-attention curves across AIFB, MUTAG, and TOX21.Lower CDF values indicate that a greater proportion of models achieve higher metric values.
- AIFB: On AIFB, additive and multiplicative ARGAT perform poorly across most of the hyperparameter space, while RGCN and multiplicative WIRGAT perform comparably.
- MUTAG: On MUTAG, RGCN and both WIRGAT variants cover more poor-performance regions while also covering more good-performance regions.ARGAT performance is relatively insensitive to hyperparameter choice compared with the other candidates.
- TOX21: On TOX21, multiplicative attention performs poorly across most of the search space but outperforms other models in a small region.Additive attention and RGCN perform comparably throughout and generally exceed multiplicative methods outside that region.
E Significance testing
The paper uses one-sided Mann–Whitney U tests to assess directional model comparisons without parametric assumptions. Empirical distributions from paired models are treated as samples, with results reported for AIFB, MUTAG, and TOX21.
- Test design: The one-sided Mann–Whitney U test evaluates whether one model’s performance distribution exceeds another’s without assuming a parametric response.The analysis focuses on directional movement in performance.
- Hypotheses: The null hypothesis states that the two populations are equal.
- Hypotheses: The alternative hypothesis is H1: P(X > Y) > P(Y > X), indicating that observations from X more often exceed observations from Y.
- Application: Model A and Model B empirical distributions are treated as samples from populations X and Y to compare paired models on each task.
- Results: Figures 6, 7, and 8 report the AIFB, MUTAG, and TOX21 comparison results, respectively.