Source-linked AI summary
MGAE: Masked Autoencoders for Self-Supervised Learning on Graphs
Qiaoyu Tan, Ninghao Liu, Xiao Huang, Rui Chen, Soo-Hyun Choi, Xia Hu
TL;DR
Unsupervised graph representation learning needs a masked-autoencoding approach that can reconstruct missing structure while using limited observed edges. MGAE masks about 70% of edges, applies a GNN to the remaining graph, and uses a cross-correlation decoder; experiments report better or comparable performance across benchmark tasks and datasets.
Problem
Masked autoencoding had not been explored for graphs, and existing edge-dropping approaches typically use less than 30% masking while targeting robust representations rather than reconstructing removed edges.
Method
MGAE randomly masks a high proportion of graph edges, runs a GNN encoder on the remaining structure, and reconstructs masked edges with a tailored cross-correlation decoder.
Results
MGAE performs better or sometimes on par with state-of-the-art competitors across Planetoid and OGB benchmarks for link prediction and node classification.
Takeaways & Limitations
A 70% masking ratio enables MGAE to train on only about 30% of the original edges while maintaining effective graph representation learning.
Takeaways & Limitations
The framework assumes an undirected graph with node attributes, although attributes can be initialized as one-hot vectors or learnable parameters when unavailable.
Abstract
from arXiv · showhide
We introduce a novel masked graph autoencoder (MGAE) framework to perform effective learning on graph structure data. Taking insights from self-supervised learning, we randomly mask a large proportion of edges and try to reconstruct these missing edges during training. MGAE has two core designs. First, we find that masking a high ratio of the input graph structure, e.g., $70\%$, yields a nontrivial and meaningful self-supervisory task that benefits downstream applications. Second, we employ a graph neural network (GNN) as an encoder to perform message propagation on the partially-masked graph. To reconstruct the large number of masked edges, a tailored cross-correlation decoder is proposed. It could capture the cross-correlation between the head and tail nodes of anchor edge in multi-granularity. Coupling these two designs enables MGAE to be trained efficiently and effectively. Extensive experiments on multiple open datasets (Planetoid and OGB benchmarks) demonstrate that MGAE generally performs better than state-of-the-art unsupervised learning competitors on link prediction and node classification.
1 Introduction
MGAE addresses the open problem of masked autoencoding for graphs by masking a high proportion of edges, reconstructing them with a tailored decoder, and improving unsupervised graph representation learning across downstream tasks.
- 1 Introduction: MGAE asks how to design graph masked autoencoding and determine the edge-masking proportion needed for effective node representations.
- 1 Introduction: The framework masks a large proportion of edges, encodes only the remaining structure with a GNN, and reconstructs masked edges using cross-correlation between endpoint representations.
- 1 Introduction: MGAE is introduced as a masked graph autoencoder for graph-structured data, targeting link prediction and node classification.
- 1 Introduction: A tailored cross-correlation decoder handles noisy representations from masked structures and enables a 70% edge-masking ratio while improving effectiveness and efficiency.
- 1 Introduction: Extensive experiments show MGAE performs better or sometimes on par with state-of-the-art competitors on Planetoid and OGB benchmarks for link prediction and node classification.
2 Problem Statement
This section defines graph autoencoders as unsupervised models that learn node embeddings by preserving graph structure, using GNN message passing and an edge-reconstruction decoder.
- 2 Problem Statement: The formulation assumes an undirected graph with node attribute vectors, which may be replaced by one-hot indices or learnable parameters when attributes are unavailable.
- 2 Problem Statement: A graph autoencoder maps nodes to latent embeddings and reconstructs network edges, with performance evaluated on link prediction and node classification.
- 2 Problem Statement: The GNN encoder updates each node by aggregating its own representation with those of neighboring nodes through message passing.
- 2 Problem Statement: Figure 1 depicts MGAE masking many edges, applying the GNN to the remaining edges, and decoding the masked links from multi-granularity endpoint cross-correlations.
- 2 Problem Statement: At layer k, node representations capture neighborhood structure within k hops through aggregation and combination functions.
- 2 Problem Statement: The decoder predicts whether an edge exists from hidden node representations using an edge-wise similarity function, such as an inner product or MLP.
3 Masked Graph Autoencoder
MGAE masks a large subset of graph edges, encodes the remaining structure with a GNN, and reconstructs masked edges using a multi-granularity cross-correlation decoder. Its design combines network masking, partial-graph message propagation, cross-correlation modeling, and masked-edge reconstruction.
- Framework overview: MGAE reconstructs masked edges from partially observed graphs using network masking, a GNN encoder, a cross-correlation decoder, and a reconstruction target.The encoder operates on reserved edges, while the decoder predicts edges in the masked set.
- Network masking: A high-ratio random mask partitions edges into Emask and Ereserve, with only the reserved edges retained for encoder message propagation.The two sets satisfy Emask ∪ Ereserve = E, and random sampling is used for efficiency.
- Network masking: Undirected masking is harder than directed masking, and experiments favor undirected masking for dense Planetoid graphs but directed masking for sparse OGB graphs.Undirected masking removes both directional copies of a link, whereas directed masking can retain the reverse link.
- GNN encoder: The GNN encoder uses GCN or GraphSage backbones on a small subset such as 30% of edges, reducing training computation and memory costs.Masked edges are removed during message propagation and recovered by the decoder.
- Cross-correlation decoder: The cross-correlation decoder combines K hidden representations by modeling correlations between the head and tail nodes across different neighborhood granularities.This shared-pattern representation addresses noise and incompleteness in embeddings produced from the masked graph; K is typically small, such as K = 2.
- Reconstruction target: MGAE trains against Emask rather than Ereserve, using reconstructed edge scores from an MLP and a standard graph-based loss.Negative sampling is adopted in the experiments.
4 Experiments
MGAE is evaluated against graph autoencoder and self-supervised baselines on link prediction and node classification, across Planetoid and OGB benchmarks, while varying masking ratios. The experiments also examine encoder variants and compare MGAE with SelfTask-GNN under different masking ratios.
- 4.1 Link Prediction: MGAE outperforms graph autoencoder baselines on six link-prediction datasets in almost all cases and achieves new state-of-the-art results on Cora, PubMed, and ogb-ddi.It is comparable with the best graph autoencoder baselines on ogbl-collab and ogbl-ppa, while showing larger gains on the other four datasets.
- 4.1 Link Prediction: MGAE achieves substantial gains over self-supervised baselines on five of six link-prediction datasets, losing only to GIC on CiteSeer.The performance gap from these baselines increases on OGB datasets.
- 4.1 Link Prediction: MGAE-GCN and MGAE-SAGE do not consistently outperform each other, indicating that the best encoder varies across graph scenarios.The experiments use both GCN and GraphSage encoder variants.
- 4.1 Link Prediction: Most reported results use masking ratio ω = 0.7, so the encoder receives only 30% of the original edges and can reduce message-propagation cost.The exception is ogbl-ppa, which uses a different masking ratio in the OGB evaluation.
- 4.2 Node Classification: MGAE consistently outperforms graph-autoencoder baselines across five node-classification datasets, while exceeding self-supervised baselines by a large margin on three larger datasets.It loses to the best self-supervised result on the smaller Cora and CiteSeer datasets, but outperforms those methods on PubMed, ogbn-arxiv, and ogbn-proteins.
- 4.3 Sensitivity Analysis: MGAE performance rises with masking ratio until ω = 0.7, then declines, remaining relatively stable around ω = 0.5–0.7.Across masking ratios, MGAE generally outperforms SelfTask-GNN except below ω < 0.2 on PubMed, with the largest gap around 0.5–0.7.
5 Conclusion
MGAE is a self-supervised graph autoencoder that masks much of the graph, reconstructs masked edges with a cross-correlation decoder, and outperforms state-of-the-art baselines on link prediction and node classification.
- MGAE masks 70% of graph structure, reconstructs only masked edges, and uses undirected or directed masking to create self-supervisory tasks.
- A tailored cross-correlation decoder recovers missing edges by capturing multi-granularity cross representations between anchor-edge head and tail nodes.
- Across multiple open graph benchmarks, MGAE outperforms state-of-the-art baselines on link prediction and node classification.