PaperlayerView exact revision on arXiv

Source-linked AI summary

GraphMAE: Self-Supervised Masked Graph Autoencoders

Zhenyu Hou, Xiao Liu, Yukuo Cen, Yuxiao Dong, Hongxia Yang, Chunjie Wang, Jie Tang

arXiv:2205.10803v3cs.LG

TL;DR

Graph autoencoders have lagged contrastive graph SSL, which relies on elaborate training and difficult-to-interpret augmentations. GraphMAE addresses this gap with masked feature reconstruction, scaled cosine error, re-mask decoding, and GNN decoding, and shows advantages across three tasks and 21 datasets. The findings support further exploration of generative graph pretraining.

  • Problem

    Graph autoencoders have lagged contrastive graph SSL, while contrastive methods depend on complicated training strategies and graph augmentations that are difficult to interpret.

  • Method

    GraphMAE is a masked graph autoencoder that reconstructs masked features using scaled cosine error, re-mask decoding, and GNN decoders.

  • Results

    Across 21 datasets and three graph-learning tasks, GraphMAE generates performance advantages over state-of-the-art contrastive SSL approaches and can sometimes match or outperform supervised baselines.

  • Takeaways & Limitations

    The work demonstrates the potential of generative self-supervised pretraining for graph representation learning and calls for more in-depth exploration.

  • Takeaways & Limitations

    Graph augmentations remain difficult to interpret, with no theoretical verification that handcrafted strategies are label-invariant and optimal.

Abstract

from arXiv · show

Self-supervised learning (SSL) has been extensively explored in recent years. Particularly, generative SSL has seen emerging success in natural language processing and other AI fields, such as the wide adoption of BERT and GPT. Despite this, contrastive learning-which heavily relies on structural data augmentation and complicated training strategies-has been the dominant approach in graph SSL, while the progress of generative SSL on graphs, especially graph autoencoders (GAEs), has thus far not reached the potential as promised in other fields. In this paper, we identify and examine the issues that negatively impact the development of GAEs, including their reconstruction objective, training robustness, and error metric. We present a masked graph autoencoder GraphMAE that mitigates these issues for generative self-supervised graph pretraining. Instead of reconstructing graph structures, we propose to focus on feature reconstruction with both a masking strategy and scaled cosine error that benefit the robust training of GraphMAE. We conduct extensive experiments on 21 public datasets for three different graph learning tasks. The results manifest that GraphMAE-a simple graph autoencoder with careful designs-can consistently generate outperformance over both contrastive and generative state-of-the-art baselines. This study provides an understanding of graph autoencoders and demonstrates the potential of generative self-supervised pre-training on graphs.

1 INTRODUCTION

GraphMAE addresses four weaknesses identified in graph autoencoders by combining masked feature reconstruction, scaled cosine error, re-mask decoding, and GNN decoding. Across 21 datasets and three graph-learning tasks, it outperforms contrastive approaches and sometimes matches or exceeds supervised baselines.

  • Motivation: Graph SSL has favored contrastive methods, whose training commonly depends on complex architectures, negative sampling, and heuristic graph augmentations.These dependencies include momentum or exponential-moving-average encoders, constructed negatives, and graph augmentations whose effectiveness can be difficult to verify.
  • Challenges: Existing GAEs lag behind contrastive SSL because they often emphasize structure reconstruction, use uncorrupted feature reconstruction, rely on sensitive MSE, and employ weak MLP decoders.Structure reconstruction favors link prediction and clustering, while MSE can suffer from varied feature norms and dimensionality-related instability.
  • GraphMAE Design: GraphMAE reconstructs masked node features rather than graph structure, using masking to improve robustness and avoid trivial identity solutions.The design is motivated by denoising autoencoders and the small feature dimensions typical of graph nodes.
  • GraphMAE Design: GraphMAE uses scaled cosine error to address feature-vector magnitude variation and down-weight easy reconstruction samples.The scaling makes high-confidence cosine errors decay faster when the scaling factor γ exceeds 1.
  • GraphMAE Design: GraphMAE re-masks encoder outputs for masked nodes and uses GNN decoders instead of the commonly used MLP decoders.These choices increase decoder expressiveness while preserving the masked reconstruction setup.
  • Results: Across 21 datasets and three graph-learning tasks, GraphMAE generates performance advantages over state-of-the-art contrastive SSL approaches.The tasks include node classification, graph classification, and transfer learning; in many cases, it also matches or outperforms supervised baselines.

2 RELATED WORK

Related graph SSL work divides into contrastive and generative methods. Contrastive approaches depend on elaborate sampling, architectural, and augmentation choices, while earlier generative methods generally underperform and graph autoencoders often produce unsatisfactory classification results.

  • Overview: Graph self-supervised methods are categorized into contrastive and generative approaches according to their architectures and objectives.
  • Contrastive Self-Supervised Graph Learning: Contrastive graph learning relies on negative sampling, architectural constraints, and high-quality data augmentation.Examples include negative queues or in-batch negatives, momentum or EMA-based asymmetric encoders, and feature-, proximity-, or sampling-based augmentations.
  • Contrastive Self-Supervised Graph Learning: Graph augmentations are difficult to interpret because handcrafted strategies lack a theoretical guarantee of label invariance or optimality.
  • Generative Self-Supervised Graph Learning: Earlier generative graph methods substantially lagged contrastive methods in graph representation learning.
  • Generative Self-Supervised Graph Learning: Graph autoregressive models factorize joint distributions into conditionals, but their dependence on decoding order is poorly suited to most unordered graphs.
  • Generative Self-Supervised Graph Learning: Graph autoencoders avoid decoding orders and have used link prediction, feature reconstruction, or both, but typically perform unsatisfactorily on node and graph classification.Earlier GAEs commonly reconstruct structure without masking, and some combine structural and feature objectives.
  • Generative Self-Supervised Graph Learning: Attribute-masking studies have lagged state-of-the-art contrastive methods because they did not address other critical defects of existing GAEs.

3 THE GraphMAE APPROACH

GraphMAE reframes graph autoencoding around masked node-feature reconstruction and introduces design choices for robust self-supervised representation learning. Its framework combines masking, re-mask decoding with expressive GNN decoders, and scaled cosine error to address weaknesses in prior GAEs.

  • Decoder design: GraphMAE re-masks encoded representations of selected nodes and uses GNN decoders instead of the MLPs common in previous GAEs.The decoder reconstructs a masked node from neighboring unmasked latent representations, and can use architectures such as GAT or GIN.
  • Feature reconstruction: GraphMAE reconstructs masked node features rather than graph structure, aligning its training objective with downstream classification representations.The framework uses partially observed node features and the input adjacency matrix to reconstruct selected nodes.
  • Robust training: Masking corrupts input features before encoding, reducing the risk that a vanilla autoencoder learns an identity function.A learnable [MASK] token replaces features for a sampled subset of nodes, and the decoder reconstructs those original features.
  • Reconstruction criterion: Scaled cosine error down-weights easy reconstruction samples, addressing limitations of MSE for continuous node features with varying norms.With scaling factor γ≥1, high-confidence predictions whose cosine errors are below 1 decay faster toward zero.
  • Design rationale: GraphMAE is designed to remedy prior GAE weaknesses involving reconstruction objectives, trivial solutions, decoder expressiveness, and reconstruction loss design.The approach is presented as a simple, scalable framework intended to strengthen generative graph pretraining.

4 EXPERIMENTS

GraphMAE is evaluated as a general self-supervised framework across node classification, graph classification, and molecular transfer learning. Across 21 benchmarks, it shows competitive or superior results, while ablations support masking, scaled cosine error, and expressive decoders.

  • Experimental scope: GraphMAE evaluates unsupervised node classification, graph classification, and molecular property transfer learning using standard experimental procedures.The experiments cover 21 benchmarks and compare against state-of-the-art contrastive and generative methods.
  • Node classification: GraphMAE achieves the best or competitive results across node-classification benchmarks and provides strong generalization to unseen PPI and Reddit nodes.The evaluation compares GraphMAE with contrastive, generative, and supervised baselines.
  • Graph classification: GraphMAE outperforms all self-supervised baselines on five of seven graph-classification datasets and remains competitive on the other two.The result holds despite these datasets using relatively less informative one-hot node features.
  • Transfer learning: GraphMAE achieves the best average transfer-learning scores and a small edge over previous best results on two molecular property prediction tasks.The model is pretrained on 2 million unlabeled ZINC15 molecules and finetuned on eight scaffold-split benchmarks.
  • Ablation studies: SCE improves over MSE by 1.5%–8.0% in node classification, while offering a smaller performance edge on graph classification.MSE also improves graph-classification accuracy because the inputs are discrete one-hot encodings.
  • Ablation studies: Masking input features is vital for avoiding trivial solutions, re-masking improves accuracy by 0.1%–1.9%, and GNN decoders typically outperform MLP decoders.Among GNN decoders, GIN performs better for graph-level tasks, while GAT is more suitable for node classification; replacing GAT with GCN reduces Cora and PubMed performance by about 2.9% and 2.0%.

5 CONCLUSION

GraphMAE addresses common graph autoencoder issues through masked feature reconstruction and a scaled cosine error. Experiments demonstrate effectiveness and generalizability across graph classification benchmarks.

  • GraphMAE identifies common issues faced by graph autoencoders in generative self-supervised graph learning.
  • GraphMAE uses masked feature reconstruction with a scaled cosine error as its reconstruction criterion.
  • Experiments across node and graph classification benchmarks demonstrate GraphMAE’s effectiveness and generalizability.

A APPENDIX

The appendix compares attributed-masking methods for node classification and notes a dataset mismatch affecting the PubMed comparison. Evaluation distinguishes finetuning from linear-probe settings.

  • Table 5 compares GraphMAE with other attributed-masking methods in node classification.
  • “FT” denotes downstream-task finetuning, whereas “LP” denotes training a linear classifier for classification.
  • The PubMed dataset used differs from the dataset used by other baselines.

A.1 Results in the PPI Dataset

On PPI, increasing model size helps self-supervised performance substantially but helps supervised performance little. GraphMAE can exceed the supervised counterpart at a very large hidden size, with baseline discrepancies attributed to evaluation procedures.

  • Increasing model parameters highly boosts self-supervised performance on PPI but helps little in the supervised setting.
  • At hidden size 2048×4, GraphMAE outperforms the supervised counterpart on PPI, although the model is much larger.
  • Reported BGRL and GRACE PPI results differ from prior work because the linear classifier was trained until convergence during evaluation.

A.2 Ablation on the Encoder Architecture

Encoder choice varies by task in GraphMAE’s ablations. GIN performs better for graph-level tasks, while GAT is generally preferable for node classification under the reported training setting.

  • Under GraphMAE’s training setting, GAT could be a better option in most cases.

A.3.1 Environment.

Experiments use specified Linux-server environments and node-classification optimization settings, alongside performance and ablation studies on PPI, Ogbn-arxiv, and IMDB-B.

  • Environment: Most experiments use Intel Xeon E5-2680 v4 CPUs, 256GB RAM, NVIDIA 2080Ti GPUs, and specified Python, PyTorch, DGL, CUDA, and scikit-learn versions.Ogbn-arxiv and Reddit node-classification experiments instead use Xeon Gold 6240 CPUs and NVIDIA 3090 GPUs because they require large memory.
  • Performance studies: On PPI, Figure 4 evaluates GAT with 4 attention heads against other baselines and reports that GraphMAE could outperform a supervised model.The figure passage also states that self-supervised methods benefit from larger model size.
  • Ablation studies: Figure 5 studies the effects of mask ratio and scaling factor γ in Ogbn-arxiv and IMDB-B.
  • Optimization: Node classification uses Adam with β1 = 0.9, β2 = 0.999, ε = 1×10^-8, initial learning rate 0.001, cosine decay without warmup, and PReLU activation.

A.3.2 Model Configuration.

Model configurations vary by task, with separate optimization settings for graph classification and molecular transfer learning and dataset-specific statistics reported in Tables 7–9.

  • Graph classification: Graph classification generally uses initial learning rate 0.00015 with cosine decay, while SVM evaluation searches C over {10^-3, ..., 10}.
  • Molecular transfer learning: Molecular transfer learning uses a single-layer GIN decoder, mask rate 0.25, and 100-epoch pretraining.
  • Dataset configuration: Table 7 reports node-classification dataset statistics and hyper-parameters, distinguishing multi-class classification (“s”) from multi-label classification (“m”).
  • Dataset configuration: Table 8 provides statistics and hyper-parameters for graph-classification datasets.
  • Dataset configuration: Table 9 reports molecular-property-prediction dataset statistics and identifies ZINC as the pre-training dataset.

A.4 Baselines

Baselines combine supervised and unsupervised graph-learning methods, using prior reports, official implementations, reproduced results, and matched GNN backbones for fair comparison.

  • Node classification: Node-classification comparisons include supervised GCN and GAT baselines, with some results taken from CogDL when not previously reported.
  • Node classification: GRACE, BGRL, and CCA-SSG serve as state-of-the-art contrastive-learning baselines for graph node classification.
  • Node classification: The authors use the same GNN backbone as GraphMAE, tune contrastive baselines on validation data, and fix bugs in the official CCA-SSG code.
  • Node classification: MVGRL results come from DGL reproductions, while GPT-GNN is implemented from official code adapted for homogeneous networks.
  • Graph classification and transfer learning: Graph-classification and molecular-transfer results use prior papers when available, with official GraphCL and JOAO code run for IMDB-MULTI.
Loading 2205.10803v3…