Source-linked AI summary
GraphMAE2: A Decoding-Enhanced Masked Self-Supervised Graph Learner
Zhenyu Hou, Yufei He, Yukuo Cen, Xiao Liu, Yuxiao Dong, Evgeny Kharlamov, Jie Tang
TL;DR
Graph SSL seeks to address scarce labels, but masked feature reconstruction depends on input-feature discriminability. GraphMAE2 regularizes reconstruction with randomized re-masking and latent representation prediction, and consistently outperforms graph SSL baselines across datasets and settings. Its large-scale evaluation on ogbn-Papers100M demonstrates significant performance improvements.
Problem
Masked feature reconstruction in graph SSL relies on input-feature discriminability, while real-world graphs often provide less discriminative node features.
Method
GraphMAE2 regularizes target reconstruction through multi-view random re-mask decoding and latent representation prediction in embedding space.
Results
GraphMAE2 consistently offers significant outperformance over state-of-the-art graph SSL baselines across public datasets and different settings.
Takeaways & Limitations
The results support node-level signals as abundant supervision for masked graph self-supervised learning and motivate further exploration.
Takeaways & Limitations
Graph SSL augmentation choices remain insufficiently understood theoretically and still depend mostly on researchers’ empirical analysis.
Abstract
from arXiv · showhide
Graph self-supervised learning (SSL), including contrastive and generative approaches, offers great potential to address the fundamental challenge of label scarcity in real-world graph data. Among both sets of graph SSL techniques, the masked graph autoencoders (e.g., GraphMAE)--one type of generative method--have recently produced promising results. The idea behind this is to reconstruct the node features (or structures)--that are randomly masked from the input--with the autoencoder architecture. However, the performance of masked feature reconstruction naturally relies on the discriminability of the input features and is usually vulnerable to disturbance in the features. In this paper, we present a masked self-supervised learning framework GraphMAE2 with the goal of overcoming this issue. The idea is to impose regularization on feature reconstruction for graph SSL. Specifically, we design the strategies of multi-view random re-mask decoding and latent representation prediction to regularize the feature reconstruction. The multi-view random re-mask decoding is to introduce randomness into reconstruction in the feature space, while the latent representation prediction is to enforce the reconstruction in the embedding space. Extensive experiments show that GraphMAE2 can consistently generate top results on various public datasets, including at least 2.45% improvements over state-of-the-art baselines on ogbn-Papers100M with 111M nodes and 1.6B edges.
1 INTRODUCTION
Graph self-supervised learning addresses scarce graph labels, while masked feature reconstruction is vulnerable to limited input-feature discriminability. GraphMAE2 regularizes reconstruction through randomized re-masking and latent prediction, achieving strong results across graph scales and datasets.
- Real-world graph applications often lack sufficient task-specific labels, especially on billion-scale graphs.
- Generative graph SSL masks node features or edges and learns to recover the original input; GraphMAE applies this paradigm to randomly masked node features.
- GraphMAE’s masked-feature reconstruction relies on discriminable input features, which are often approximate and less discriminative descriptions of graph nodes.
- GraphMAE2 regularizes target reconstruction with multi-view random re-mask decoding and latent representation prediction.Random re-masking regularizes input-feature reconstruction, while latent prediction reconstructs masked features in embedding space.
- GraphMAE2 consistently outperforms state-of-the-art graph SSL baselines across public datasets and settings, with both decoding strategies contributing to improvements.
- 111M nodes and 1.6B edges: experiments on ogbn-Papers100M show significant improvements from extending GraphMAE2 to large-scale graphs.
2 METHOD
GraphMAE2 addresses vulnerabilities in masked feature reconstruction by regularizing decoding with randomized re-masking and latent-space prediction, while extending training to large graphs.
- GraphMAE2 Framework: For large-scale graphs, GraphMAE2 uses local clustering to form node-associated clusters and addresses challenges associated with existing sampling approaches.The framework is designed for graph SSL settings where prior work largely focused on small graphs or reused supervised sampling methods.
- Masked Autoencoding on Graphs: GraphMAE2 masks node features, encodes the corrupted graph with a GNN, and reconstructs the original features from the resulting representations.The masked graph is processed by an encoder, and a decoder predicts masked features against the originals.
- GraphMAE2 Framework: GraphMAE2 regularizes decoding because graph node features may be noisy or insufficiently discriminative for reliable masked reconstruction.The framework is motivated by the vulnerability of feature reconstruction to input-feature distinguishability and unexpected noise.
- GraphMAE2 Framework: Multi-view random re-mask decoding repeatedly corrupts encoded representations and forces every view to reconstruct input features.The multiple views expose the decoder to partially observed embeddings and regularize reconstruction against memorizing input patterns.
- GraphMAE2 Framework: Latent representation prediction reconstructs masked-node targets in embedding space, using targets generated from the unmasked graph.A target-generation network produces latent targets, while the masked graph’s encoded representations are projected for prediction.
3 EXPERIMENTS
Experiments evaluate GraphMAE2 across graph datasets of varied scales using linear probing, fine-tuning, and ablation studies. GraphMAE2 generally outperforms the compared baselines, while both decoding strategies and local clustering contribute to its performance.
- Large-scale and small-scale evaluation: GraphMAE2 achieves better linear-probing results than all self-supervised baselines across the evaluated datasets.It improves upon GraphMAE by 1.91% on MAG-Scholar-F and 2.35% on Papers100M.
- Large-scale and small-scale evaluation: In semi-supervised fine-tuning, GraphMAE2 achieves state-of-the-art performance across the large-scale benchmarks, except under 1% training data on Products.On ogbn-Products, pretraining improves performance by above 5.1%, while 5% labeled data reaches 80.52%.
- Large-scale and small-scale evaluation: GraphMAE2 generally outperforms the baselines on small-scale datasets, although its improvement over GraphMAE is less significant than on large graphs.The authors relate this pattern to less noisy bag-of-words input features.
- Ablation studies: Both latent representation prediction and multi-view random re-masking improve performance, with latent prediction producing larger ablation drops.Removing latent prediction drops accuracy by -1.58% on ogbn-Products and -1.91% on ogbn-Papers100M, versus -0.45% and -0.73% without random re-masking.
- Ablation studies: Local clustering outperforms GraphSAINT and Cluster-GCN by 0.57% and 1.49% on average, respectively.The authors attribute the advantage to densely connected local subgraphs that preserve useful neighborhood information.
4 RELATED WORK
Graph SSL comprises contrastive and generative approaches, alongside scalable GNN training strategies. Related work highlights augmentation dependence in contrastive learning and reconstruction-based generative methods.
- Graph SSL is broadly divided into contrastive learning and generative learning according to the learning paradigm.
- Contrastive methods: Contrastive methods learn representations through mutual-information maximization, graph diffusion, subgraph discrimination, augmentation agreement, or group discrimination.
- Contrastive methods: Negative-sample-free methods use online and target encoders or feature-level objectives instead of expensive negative samples.
- Contrastive methods: Most graph contrastive methods rely on complex graph augmentations, whose theoretical understanding remains limited.
- Generative methods: Graph autoencoders generally reconstruct graph structure, node attributes, neighborhoods, or combinations of structural and attribute information.
- Scalable graph neural networks: Scalable GNNs use either sampled mini-batches or decoupled training paradigms, with subgraph sampling methods including GraphSAGE, GraphSAINT, and ClusterGCN.
5 CONCLUSION
The paper studies masked feature prediction for graph SSL and introduces GraphMAE2 to regularize reconstruction. Its decoding strategy improves performance on realistic large-scale benchmarks and supports further investigation of node-level supervision.
- GraphMAE2 addresses concerns that limited input-feature discriminability hinders masked feature prediction by imposing regularization on prediction.
- The framework focuses on decoding with latent representation targets and randomness in input reconstruction.
- The novel decoding strategy significantly boosts performance in realistic large-scale benchmarks.
- The work concludes that node-level signals can provide abundant supervision for masked graph SSL and deserve further exploration.
A.1 Theorem for Local Clustering.
The appendix states a local-clustering theorem for retrieving a targeted cluster in an unweighted graph and defines conductance for non-empty node sets.
- For an unknown targeted cluster T, the theorem guarantees a substantial subcluster T_g under an inverse-mixing-time condition.It requires vol(T_g) ≥ vol(T)/2 and concerns seeds in T_g.
- The conductance of a non-empty node set S is defined using crossing edge weight divided by the smaller volume of S and its complement.
A.2 Finetuning Results for Supervised Learning
Fine-tuning with all training labels evaluates pretrained encoders using accuracy across datasets. GraphMAE2 improves over random initialization on three datasets, with a smaller gain on ogbn-Papers100M.
- Fine-tuning results report accuracy(%) for all datasets under mini-batch training with full labels.
- 1.22%-3.17% improvement over random initialization is observed on ogbn-Arxiv, Products, and MAG-Scholar-F.
- 0.2% improvement over random initialization is observed on ogbn-Papers100M, the only reported exception.The passage attributes this to an overly well-labeled training set under the dataset split.
A.3 Linear Probing Results on ogbn-Arxiv with Full-graph Training.
On ogbn-Arxiv, GraphMAE2 is evaluated against baselines using full-graph training, with results reported as linear-probing accuracy.
- 0.2% improvement over GraphMAE is reported for GraphMAE2 under full-batch training on ogbn-Arxiv.
- Table 9 reports linear probing results using accuracy(%).
A.4 Implementation Notes
The framework is implemented in PyTorch and evaluated on a Linux system with substantial compute and specified software versions; large-scale pretraining uses AdamW with cosine decay.
- Experiments run on Linux with 1007G RAM and eight NVIDIA A100 GPUs with 80GB memory each.
- The implementation uses Python 3.9, PyTorch 1.12.0, OGB 1.3.3, and CUDA 11.3.
- Pretraining uses AdamW with cosine learning-rate decay and no warmup.
- Table 10 contains additional hyperparameters for large-scale datasets.
A.5 Baselines
For large-scale graphs, GraphMAE2 is compared with established graph SSL baselines using a common GAT backbone and adapted sampling-compatible implementations where possible.
- Four large-dataset baselines—GRACE, BGRL, CCA-SSG, and GraphMAE—are evaluated with the GAT backbone.Their public code was adapted to integrate sampling algorithms for large-scale graphs.
- GGD results are taken from its original paper because its authors have not released the code.