Source-linked AI summary
Graph Attention Networks
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, Yoshua Bengio
TL;DR
Graph-structured data spans irregular domains where grid-based methods do not apply. The paper introduces graph attention networks that attend over neighboring node features, achieving or matching state-of-the-art performance across four transductive and inductive benchmarks.
Problem
Many important datasets lie in irregular graph-structured domains rather than grid-like structures, motivating neural methods that operate on graphs.
Method
GATs use masked self-attentional layers to compute node representations by attending over neighbors, assigning them implicitly different importances without costly matrix operations or predefined full graph structure.
Results
GATs achieved or matched state-of-the-art performance across all four benchmarks, including a 20.5% improvement over the best GraphSAGE result on PPI.
Takeaways & Limitations
The results support using attention-based graph networks for both transductive tasks and inductive settings involving unseen graphs.
Takeaways & Limitations
The implementation’s sparse-matrix constraint limits batching capabilities, especially for datasets containing multiple graphs.
Abstract
from arXiv · showhide
We present graph attention networks (GATs), novel neural network architectures that operate on graph-structured data, leveraging masked self-attentional layers to address the shortcomings of prior methods based on graph convolutions or their approximations. By stacking layers in which nodes are able to attend over their neighborhoods' features, we enable (implicitly) specifying different weights to different nodes in a neighborhood, without requiring any kind of costly matrix operation (such as inversion) or depending on knowing the graph structure upfront. In this way, we address several key challenges of spectral-based graph neural networks simultaneously, and make our model readily applicable to inductive as well as transductive problems. Our GAT models have achieved or matched state-of-the-art results across four established transductive and inductive graph benchmarks: the Cora, Citeseer and Pubmed citation network datasets, as well as a protein-protein interaction dataset (wherein test graphs remain unseen during training).
1 INTRODUCTION
Many important tasks involve graph-structured data rather than grid-like inputs, motivating neural architectures that operate on irregular domains. The paper introduces an attention-based architecture for node classification that computes node representations by attending over neighbors.
- Motivation: Graph representations capture irregular-domain data from 3D meshes, social and telecommunication networks, biological networks, and brain connectomes.These tasks cannot generally be represented with grid-like structures.
- Related work: Prior efforts extended neural networks to graphs through recursive neural networks and general graph neural networks.Early recursive models handled directed acyclic graphs, while later graph neural networks addressed a more general class of graphs.
- Related work: Graph convolution research is commonly divided into spectral and non-spectral approaches.Spectral methods use graph spectral representations, whereas non-spectral methods define convolutions directly over spatially close neighbors.
- Contribution: The paper introduces an attention-based architecture for node classification that computes each node’s hidden representation by attending over its neighbors.The method follows a self-attention strategy on graph neighborhoods.
- Contribution: The attention operation is efficient because it is parallelizable across node-neighbor pairs.This is one of the architecture’s stated properties.
2 GAT ARCHITECTURE
GATs build graph-attention networks by stacking a masked self-attentional layer that transforms node features using learned, neighborhood-specific attention weights. The layer supports efficient, inductive graph processing, while sparse implementations reduce storage but currently limit batching across multiple graphs.
- 2.1 Graph Attentional Layer: Each layer maps N node features of dimension F to node features of potentially different dimension F′ using a shared linear transformation.The input is h = {h⃗1, h⃗2, . . . , h⃗N}, with h⃗i ∈ R^F, and the output has h⃗′i ∈ R^F′.
- 2.1 Graph Attentional Layer: Masked self-attention computes coefficients for each node’s first-order neighbors, including itself, so nodes weight neighboring features according to their learned importance.The attention mechanism uses a shared function over transformed node features and injects graph structure by restricting computation to j ∈ N_i.
- 2.1 Graph Attentional Layer: The attention mechanism is a single-layer feedforward network with weight vector a⃗ ∈ R^2F′ and LeakyReLU slope α = 0.2.The coefficients use concatenated transformed node features; transposition and concatenation define the fully expanded form.
- 2.2 Multi-head Attention: Multi-head attention uses K independent attention mechanisms whose outputs are concatenated, producing KF′ features per node and stabilizing self-attention learning.On the final prediction layer, head outputs are averaged instead of concatenated, and the final classification nonlinearity is applied afterward.
- 2.3 Advantages and Limitations: O(|V|FF′ + |E|F′) is the time complexity of one GAT head, matching GCN baseline complexity without eigendecompositions or similarly costly matrix operations.Self-attention parallelizes across edges, while output-feature computation parallelizes across nodes.
- 2.3 Advantages and Limitations: GATs support directed graphs and inductive evaluation on graphs completely unseen during training, while sparse operations reduce storage complexity to linear in nodes and edges.The current sparse implementation limits batching because the tensor framework supports sparse matrix multiplication only for rank-2 tensors.
3 EVALUATION
The evaluation covers transductive citation-network benchmarks and inductive protein-interaction graphs, using GAT models tailored to each setting. GAT achieves or matches state-of-the-art performance across all four datasets, with improvements over GCNs and discriminative feature representations.
- Datasets: The experiments evaluate GAT on Cora, Citeseer, and Pubmed transductively, plus protein-protein interaction graphs inductively with completely unseen test graphs.The PPI dataset contains 20 training graphs, 2 validation graphs, and 2 test graphs, averaging 2372 nodes per graph.
- Model configurations: The inductive model is a three-layer GAT whose first two layers use K = 4 heads and F ′ = 256 features per head, while its classification layer uses K = 6 heads.The first two layers compute 1024 features each; the final layer averages 121 features per head before logistic-sigmoid classification.
- Results: GAT achieves or matches state-of-the-art performance across all four datasets and improves upon GCNs by 1.5% on Cora and 1.6% on Citeseer.The reported improvements suggest that assigning different weights to nodes in the same neighborhood may be beneficial.
- Representation analysis: First-layer GAT representations pre-trained on Cora form discernible clusters in a t-SNE 2D projection corresponding to the dataset’s seven labels.The visualization qualitatively supports the discriminative power of the learned feature representations.
4 CONCLUSIONS
The paper concludes that GATs provide efficient, parallelizable graph attention layers that assign different importances to neighboring nodes. It identifies larger batch sizes, interpretability, and graph classification as directions for future work.
- Conclusions: GATs use computationally efficient, parallelizable masked self-attentional layers that can assign different importances to nodes within a neighborhood.The layer avoids costly matrix operations and handles neighborhoods with different sizes.
- Conclusions: Future work includes overcoming practical limitations to support larger batch sizes.The paper specifically points to problems described in subsection 2.2.
- Conclusions: The authors propose analyzing attention for model interpretability and extending GATs from node classification to graph classification.These are identified as particularly interesting research directions and method extensions.