Source-linked AI summary
Graph Attention Auto-Encoders
Amin Salehi, Hasan Davulcu
TL;DR
Existing graph representation methods can lack node-attribute reconstruction, graph-structure reconstruction, label-free learning, or unified inductive capability. GATE uses stacked graph-attention encoder/decoder layers to reconstruct node features and graph structure, and it outperforms or matches supervised and unsupervised baselines in most experiments while generalizing to unseen nodes.
Problem
Existing methods often require labels, underuse node features, lack inductive learning, or fail to reconstruct both node attributes and graph structure.
Method
GATE is an unsupervised graph auto-encoder that uses stacked graph-attention encoder/decoder layers to reconstruct node features and regularizes representations to reconstruct graph structure.
Results
In most experiments, GATE outweighs state-of-the-art supervised and unsupervised baselines and naturally generalizes to unseen nodes.
Takeaways & Limitations
GATE provides a unified neural architecture for unsupervised graph representation learning across transductive and inductive tasks.
Takeaways & Limitations
The implementation cannot batch matrix multiplication for rank-3 tensors, limiting batching capability and motivating future work.
Abstract
from arXiv · showhide
Auto-encoders have emerged as a successful framework for unsupervised learning. However, conventional auto-encoders are incapable of utilizing explicit relations in structured data. To take advantage of relations in graph-structured data, several graph auto-encoders have recently been proposed, but they neglect to reconstruct either the graph structure or node attributes. In this paper, we present the graph attention auto-encoder (GATE), a neural network architecture for unsupervised representation learning on graph-structured data. Our architecture is able to reconstruct graph-structured inputs, including both node attributes and the graph structure, through stacked encoder/decoder layers equipped with self-attention mechanisms. In the encoder, by considering node attributes as initial node representations, each layer generates new representations of nodes by attending over their neighbors' representations. In the decoder, we attempt to reverse the encoding process to reconstruct node attributes. Moreover, node representations are regularized to reconstruct the graph structure. Our proposed architecture does not need to know the graph structure upfront, and thus it can be applied to inductive learning. Our experiments demonstrate competitive performance on several node classification benchmark datasets for transductive and inductive tasks, even exceeding the performance of supervised learning baselines in most cases.
1 INTRODUCTION
Graph representation learning must use both graph structure and node attributes while often operating without labels. GATE addresses these gaps with attention-based autoencoding that reconstructs both node features and graph structure and supports transductive and inductive learning.
- Most graph representation methods use structure alone, despite real-world nodes often having rich attributes.
- Supervised attribute-based methods require labels, which are unavailable in many real-world applications.
- Existing unsupervised graph embedding methods may underuse node features, lack inductive learning, or fail to unify transductive and inductive tasks.
- Prior graph auto-encoders use graph inputs but neglect to reconstruct either graph structure or node attributes.
- GATE uses stacked graph-attention encoder and decoder layers to reconstruct node attributes, while regularizing representations to reconstruct graph structure.
- GATE is presented as a unified architecture for unsupervised attributed graph representation learning in both transductive and inductive settings.
2 RELATED WORK
Prior graph embedding approaches include factorization, random-walk, and autoencoder methods, while attributed graph methods separately address supervised and unsupervised settings. A remaining gap is unified unsupervised learning for both transductive and inductive tasks.
- Graph Representation Learning: Graph embedding methods are commonly categorized as factorization-based, random-walk-based, or autoencoder-based approaches.
- Graph Representation Learning: Factorization methods use matrix decompositions or related techniques, whereas random-walk methods define node similarity through simulated walks.
- Graph Representation Learning: Factorization and random-walk approaches use shallow models, motivating autoencoders that capture nonlinear graph structures with deep networks.
- Attributed Graph Representation Learning: Attributed graph embedding methods use node features in supervised or unsupervised approaches.
- Attributed Graph Representation Learning: Many unsupervised attributed graph methods are not designed for inductive learning, while existing inductive models are not unified frameworks for transductive and inductive tasks.
3 PROBLEM STATEMENT
The problem setup provides node features and graph relations, then seeks a representation for every node. The graph is modeled with an adjacency matrix that, in the experiments, is unweighted and includes self-loops.
- The paper represents matrices with bold uppercase letters, vectors with bold lowercase letters, sets with calligraphic fonts, and matrix entries using row and column indices.
- The attributed graph setup provides a node feature matrix X and an adjacency matrix A representing node relations.
- The objective is to learn a node-representation matrix H whose columns represent individual nodes.
4 ARCHITECTURE
GATE reconstructs node features and graph structure through stacked attention-based encoder and decoder layers. Its matrix implementation supports parallel computation and sparse-graph efficiency.
- Architecture: GATE uses stacked encoder and decoder layers to reconstruct node features from graph-structured inputs.The encoder uses graph structure to generate node representations, while decoder layers reverse the process to reconstruct features.
- Encoder: Each encoder layer computes neighbor relevance with shared self-attention, normalizes coefficients with softmax, and aggregates neighboring representations.Node features initialize the representations, and each layer propagates information through each node’s neighborhood.
- Decoder: The decoder mirrors the encoder using neighbor relevance to reconstruct representations layer by layer, producing reconstructed node features after L layers.The decoder input is the encoder’s output, and its final output is the reconstructed feature matrix.
- Objective: GATE jointly minimizes node-feature and graph-structure reconstruction losses, with λ controlling the graph-structure contribution.Graph structure reconstruction encourages neighboring nodes to have similar representations.
- Matrix formulation and complexity: Sparse matrix operations make space complexity linear in the number of nodes and edges, while graph-attention and other operations can be parallelized.The matrix formulation is designed for sparse adjacency matrices and parallel computation across edges or nodes.
5 EVALUATION
The evaluation measures GATE on transductive and inductive node classification using benchmark datasets, compares it with supervised and unsupervised baselines, and examines its architectural components and learned representations.
- Experimental setup: GATE is evaluated on Cora, Citeseer, and Pubmed for both transductive and inductive node classification.The experiments use supervised and unsupervised comparison methods, with unseen test-node information withheld in the inductive setting.
- Node classification results: In transductive classification, GATE outperforms all supervised and unsupervised baselines on Cora and Pubmed and matches GAT on Citeseer.It improves over GAT by 1.9% on Pubmed and 0.2% on Cora, and over the best unsupervised baselines by 2.7% and 0.9% on those datasets.
- Component analysis: Reconstructing node features improves performance over graph auto-encoders that reconstruct only graph structure by 2.7%, 2.6%, and 1.4% on Pubmed, Citeseer, and Cora.This comparison identifies node-feature reconstruction as a substantial contributor relative to the graph-autoencoder baselines.
- Node classification results: GATE outperforms all supervised and unsupervised baselines on the three inductive benchmarks, improving over the best baselines by 2.4%, 2%, and 0.9% on Citeseer, Cora, and Pubmed.Its inductive and transductive accuracies differ by only 0.1%, 0.3%, and 0.7% on Pubmed, Citeseer, and Cora, respectively.
- Component analysis: GATE outperforms the other architecture variants on every dataset, while GATE/A performs worst, indicating the strongest contribution from self-attention among the examined components.GATE/F performs better than GATE/S on Cora and Pubmed, whereas GATE/S performs better on Citeseer.
- Qualitative analysis: The learned Cora representations form discernible t-SNE clusters, and same-class edges are generally thicker than different-class edges under averaged attention coefficients.Node colors denote classes; edge colors distinguish same-class from different-class endpoints, and thickness represents averaged attention.
6 CONCLUSION
GATE is an unsupervised graph auto-encoder that reconstructs both node features and graph structure, with experiments showing strong performance across transductive and inductive tasks. Its batching capability is limited by the tensor framework used.
- GATE reconstructs both node features and the graph structure using stacked graph-attention encoder/decoder layers.
- Experiments on three benchmark datasets cover both transductive and inductive tasks.
- In most experiments, GATE outperforms state-of-the-art supervised and unsupervised baselines while learning high-quality node representations.
- GATE naturally generalizes to unseen nodes.
- TensorFlow’s lack of rank-3 tensor matrix multiplication limits batching capability and motivates future work.