PaperlayerView exact revision on arXiv

Source-linked AI summary

Self-Supervised Learning of Graph Neural Networks: A Unified Review

Yaochen Xie, Zhao Xu, Jingtun Zhang, Zhengyang Wang, Shuiwang Ji

arXiv:2102.10757v5cs.LG

TL;DR

Limited or unavailable labels motivate self-supervised learning for GNNs, but graph-specific methods remain difficult to organize and compare. The paper surveys and unifies contrastive and predictive approaches, summarizes settings and datasets, and introduces a standardized testbed. Its supported conclusion is a unified framework for analyzing existing methods and guiding future evaluation and development, while optimal graph views and large-scale efficiency remain open challenges.

  • Problem

    Graph SSL is still emerging, while labels may be expensive, limited, imbalanced, or unavailable and graph structure creates application-specific challenges.

  • Method

    The survey unifies contrastive GNN methods through mutual-information objectives and view generation, categorizes predictive methods by self-generated labels, and provides datasets and a standardized testbed.

  • Results

    The paper provides a unified review showing that existing contrastive methods can be represented through choices of mutual-information estimator, view generation, and graph encoder.

  • Takeaways & Limitations

    The unified treatment clarifies similarities and differences among graph SSL methods and supports future methodological development and empirical evaluation.

  • Takeaways & Limitations

    Optimal graph view generation remains unclear and existing methods face memory and efficiency problems when scaled to extremely large graphs.

Abstract

from arXiv · show

Deep models trained in supervised mode have achieved remarkable success on a variety of tasks. When labeled samples are limited, self-supervised learning (SSL) is emerging as a new paradigm for making use of large amounts of unlabeled samples. SSL has achieved promising performance on natural language and image learning tasks. Recently, there is a trend to extend such success to graph data using graph neural networks (GNNs). In this survey, we provide a unified review of different ways of training GNNs using SSL. Specifically, we categorize SSL methods into contrastive and predictive models. In either category, we provide a unified framework for methods as well as how these methods differ in each component under the framework. Our unified treatment of SSL methods for GNNs sheds light on the similarities and differences of various methods, setting the stage for developing new methods and algorithms. We also summarize different SSL settings and the corresponding datasets used in each setting. To facilitate methodological development and empirical comparison, we develop a standardized testbed for SSL in GNNs, including implementations of common baseline methods, datasets, and evaluation metrics.

1 INTRODUCTION

Self-supervised learning addresses limited or unavailable labels by learning from unlabeled data, but applying it to graph neural networks remains challenging because graph structure creates domain-specific issues. This survey unifies graph SSL methods, summarizes settings and datasets, and provides a standardized testbed for comparison.

  • Self-supervised learning enables representation learning from unlabeled data when labels are expensive, limited, imbalanced, or unavailable.
  • SSL methods are broadly divided into contrastive models using data-data pairs and predictive models using data-label pairs with self-generated labels.
  • Applying SSL to GNNs is emerging and faces challenges arising from the uniqueness of graph-structured data.
  • The survey unifies contrastive GNN methods through a mutual-information framework covering view transformations, objectives, and component choices.
  • It categorizes predictive methods by how labels are obtained, including graph reconstruction, property prediction, and self-training.
  • The authors summarize SSL task settings and datasets and develop a standardized testbed with baseline implementations and benchmarks for empirical comparison.
  • Graph-specific review is needed because image-oriented taxonomies omit graph view-generation approaches and graph-specific predictive tasks.

2 PROBLEM FORMULATION

The formulation represents graphs with adjacency and feature information, learns node- or graph-level encoders from unlabeled graph distributions, and supports representation learning, pretraining, and auxiliary learning paradigms.

  • 2.1 Notations: An attributed undirected graph is represented as G = (V, E, α), with node attributes, edges, adjacency matrix A, and feature matrix X.
  • 2.1 Notations: The survey organizes SSL methods into contrastive and predictive branches, with further categories based on view generation, objectives, and self-generated labels.
  • 2.1 Notations: The goal is to learn an encoder f producing H = f(A, X) for node-level or graph-level downstream prediction tasks.
  • 2.1 Notations: Self-supervision learns graph encoders from the unlabeled-graph distribution P by minimizing a task-specific self-supervised loss L_ssl(f, P).
  • 2.2 Paradigms for Self-Supervised Learning: Unsupervised representation learning uses only P, and the resulting representations support downstream tasks such as linear classification and clustering.
  • 2.2 Paradigms for Self-Supervised Learning: Unsupervised pretraining first trains an encoder on unlabeled graphs, then initializes supervised fine-tuning with a prediction head and labeled data.
  • 2.2 Paradigms for Self-Supervised Learning: This pretraining–fine-tuning paradigm supports semi-supervised learning when datasets overlap and transfer learning when domains differ.
  • 2.2 Paradigms for Self-Supervised Learning: Auxiliary learning jointly trains a supervised primary task and a self-supervised task, balancing their losses with a positive weight λ.

3 CONTRASTIVE LEARNING

Graph contrastive learning trains GNN encoders by contrasting representations from jointly sampled views of the same graph with independently sampled views. The survey unifies methods around view transformations, encoders, and objectives, while highlighting differences in inference, estimators, and graph-specific view generation.

  • Framework: Contrastive learning discriminates jointly sampled view pairs from independently sampled view pairs, treating views from the same graph as positives.Multiple views are generated by applying transformations to each graph, and representations of similar instances are encouraged to agree.
  • Framework: A contrastive framework is specified by transformations that generate views, encoders that represent them, and an objective that optimizes encoder parameters.Views may be graphs or subgraphs, and encoders can be identical or share weights.
  • Objectives: Contrastive objectives usually maximize agreement between view representations through mutual-information estimation, including DV, JS, and InfoNCE lower bounds.Some methods instead use non-bound objectives, and projection heads distinguish parametric from non-parametric estimators.
  • Inference: During inference, methods either merge multiple view representations, use one main encoder and view, or apply a shared encoder directly to the given graph.Merging can use summation or concatenation.
  • Objectives: Triplet and BPR losses can coincide with InfoNCE under specific settings, including one negative sample and corresponding margin or similarity choices.The triplet margin corresponds to the NT-Xent temperature when there is only one negative sample.
  • View Generation: Graph view transformations should match downstream data properties: edge perturbation can help social-network tasks but harm some molecular datasets, while subgraph sampling may help molecules.Molecular properties can depend on bonds in functional groups, making edge perturbation potentially damaging.

4 PREDICTIVE LEARNING

Predictive learning trains graph encoders with self-generated labels, organizing methods by how prediction targets are obtained. The survey covers reconstruction, variational, property-prediction, meta-path, and self-training approaches.

  • Framework: Predictive methods jointly train a graph encoder and prediction head using informative labels self-generated from the input graph.The prediction head may be removed when the learned encoder is used for downstream representation learning or pre-training.
  • Self-Training: Self-training uses pseudo-labels iteratively, while theory based on an expansion assumption explains how correctly pseudo-labeled examples can denoise incorrect ones.The framework repeatedly updates the pseudo-label set through clustering and prediction before retraining the graph neural network.
  • Graph Reconstruction: Graph reconstruction predicts observed graph components, including adjacency matrices, node attributes, or other masked graph information.GAE reconstructs the adjacency matrix, while MGAE reconstructs randomly corrupted feature matrices.
  • Graph Reconstruction: Graph autoencoders vary their decoder and regularization designs, including symmetric architectures, Laplacian sharpening, adversarial regularization, and stochastic latent-variable models.GALA uses Laplacian sharpening, ARGA/ARVGA use adversarial regularization, and SIG-VAE uses stacked stochastic layers.
  • Graph Reconstruction: Attribute masking pre-trains graph encoders by reconstructing masked node attributes from representations computed on corrupted input features.A linear projection can serve as the decoder, and masked edge attributes can also be reconstructed when available.
  • Property Prediction: Meta-path prediction trains heterogeneous-graph encoders to classify whether node pairs are connected by specified meta-paths.For K meta-paths, separate prediction heads are trained for the corresponding binary classification tasks.

5 SUMMARY OF LEARNING TASKS AND DATASETS

The survey organizes evaluation around graph-level inductive and node-level transductive learning, while also covering node-level inductive settings. It catalogs chemical, protein, social-network, citation, and larger graph datasets.

  • Graph-Level Learning: Graph-level inductive learning predicts graph properties across separate sets of graphs.Common datasets include chemical molecules, proteins, and social networks.
  • Node-Level Transductive Learning: Node-level transductive learning uses a large graph and labels for only a portion of its nodes during training.CORA, CITESEER, and PUBMED are common citation-network datasets, while Coauthor-CS, Coauthor-Physics, Amazon-Photos, and Amazon-Computers provide larger, more challenging graphs.
  • Graph-Level Learning: Chemical molecule datasets support unsupervised representation learning and self-supervised pre-training followed by fine-tuning.NCI1 and MUTAG are commonly used for unsupervised graph representation learning.
  • Graph-Level Learning: Protein datasets represent amino acids as nodes and connect amino acids located less than 6 Angstroms apart.They are used for biological-function prediction and unsupervised representation learning.
  • Graph-Level Learning: Social-network datasets represent entities as nodes and social connections as edges, with COLLAB, REDDIT-B, and IMDB-B used for graph classification.These datasets are not typically used for transfer learning because social networks differ across datasets.
  • Node-Level Inductive Learning: Node-level inductive learning separates training and testing nodes either within one graph or across separate graphs.A PPI dataset contains 395K unlabeled proteins for pre-training and 88K labeled proteins with 40 biological functions for fine-tuning.

6 AN OPEN-SOURCE LIBRARY

The survey introduces DIG-sslgraph, an open-source platform for implementing and comparing self-supervised GNN methods. It combines unified frameworks, standardized data interfaces, baseline implementations, and evaluation tools.

  • Library: DIG-sslgraph provides a unified, customizable framework for contrastive learning methods and standardized evaluation of self-supervised GNNs.The module is built with PyTorch and PyTorch Geometric.
  • Library: The library simplifies method development by letting users specify objectives, view-generation functions, and encoders within a common contrastive base class.It pre-implements InfoGraph, GRACE, MVGRL, and GraphCL for node-level or graph-level tasks.
  • Library: The open-source DIG library is publicly available for methodological development and empirical comparison.The paper provides a GitHub URL for the library.
  • Evaluation: DIG-sslgraph supports fair comparisons under common settings for unsupervised graph representation learning, semi-supervised graph classification, and unsupervised node-level representation learning.Its data interface includes datasets from TUDataset and citation-network collections.

7 CHALLENGES AND FUTURE DIRECTIONS

The survey identifies unresolved challenges in graph SSL involving task-specific views, predictive-method theory, domain knowledge, scalability, and explainability. It presents these as directions for future work.

  • View Generation: The optimal graph views for contrastive learning remain unclear and depend on the downstream task.Existing adversarial and search-based approaches have limited transformation search spaces.
  • Predictive Learning Theory: Predictive methods lack a unified theoretical framework to guide the design of pretext tasks for different downstream tasks.The information bottleneck principle may interpret some methods, but further investigation is needed.
  • Domain Knowledge: Domain knowledge offers additional self-supervision for graph tasks in areas such as biomedicine and quantum physics.Current domain-knowledge tasks include motif prediction, leaving richer task designs for future study.
  • Scaling and Efficiency: Scaling SSL to larger graphs remains difficult because contrastive methods require multiple views and often larger mini-batches.Memory and computation costs can increase sharply, limiting application to extremely large graphs.
  • Explainability: Explainability for SSL-trained GNNs requires further study, especially for task-agnostic explanations without a downstream prediction head.Drug discovery is identified as a setting where understanding molecular functional groups behind predictions is important.

8 CONCLUSION

The survey unifies self-supervised learning methods for GNNs into contrastive and predictive frameworks, while organizing datasets, evaluations, and open-source software for comparison.

  • Existing contrastive GNN methods are unified as mutual-information maximization over graph views, specified by an estimator, view generation, and encoder.The framework also discusses options for each component and guides their selection.
  • Predictive GNN methods are categorized into graph reconstruction, property prediction, and self-training according to how labels are generated.
  • The review summarizes graph datasets by domain and associated learning tasks to support future evaluation experiments.
  • A unified treatment spanning methodologies, datasets, evaluations, and open-source software is intended to foster methodological development and facilitate empirical comparison.

APPENDIX A HETEROGENEOUS AND DYNAMIC GRAPHS

Contrastive and predictive frameworks can be adapted to heterogeneous and dynamic graphs when their encoders properly incorporate graph structure and attributes.

  • Heterogeneous graphs include node and edge types that provide richer topological and feature information than typical homogeneous graphs.
  • Existing contrastive and predictive frameworks can handle heterogeneous and dynamic graphs with suitable encoders, including edge-attribute message passing networks and R-GCNs.

DICTIVE MODELS

Contrastive and predictive methods learn informative graph representations through different training objectives and exhibit distinct computational, theoretical, and architectural trade-offs.

  • Contrastive methods maximize mutual information between representations of different graph parts, whereas predictive methods preserve graph properties by predicting selected characteristics.
  • Contrastive methods usually cost more computationally but generally outperform most predictive methods on downstream classification performance.
  • Recent invariance-regularization predictive methods can achieve performance on par with state-of-the-art contrastive methods.
  • Predictive pretext tasks are more flexible for incorporating domain knowledge, while contrastive methods receive stronger theoretical guidance from mutual information.
  • GNN encoders update node representations by combining each node’s representation with aggregated neighbor representations through component functions defining the aggregation strategy.
  • Graph-level representations commonly use permutation-invariant sum or mean READOUT functions over node-level representations.

C.2 Effects of Graph Encoders

Encoder choice affects self-supervised GNN performance, with the best architecture depending on pretraining, task, and graph structure.

  • GIN often benefits most from pretraining because its greater expressiveness can exploit learned representations more effectively.GIN achieves the highest performance gain from pretraining, despite slightly lower performance than less expressive GNNs without pretraining.
  • GraphCL reports that GIN outperforms GCN and GAT in semi-supervised node classification.
  • GCN-based encoders outperform other GNN encoders for SUBG-CON because GCN is more suitable for handling subgraphs.

APPENDIX D COMPARISON OF CONTRASTIVE OBJECTIVES

The survey compares contrastive objectives for graph learning, finding that objective performance depends on the downstream task, batch size, and graph views.

  • JS and InfoNCE are the most commonly used graph contrastive objectives because both provide lower bounds on mutual information.
  • InfoNCE generally outperforms JS but is more sensitive to the number of negative samples and requires many negatives to remain competitive.With limited mini-batch sizes, JS may be preferable.
  • The JS estimator generally performs best for graph classification, whereas InfoNCE or NT-Xent achieves overall best performance on node classification.The comparison used batch sizes of 32–256 for graph classification and 2–8 for node classification.
  • In one graph contrastive framework, the triplet margin loss achieves the best performance among triplet, logistic, and BPR objectives.BPR is equivalent to InfoNCE with N = 1, while logistic loss is treated as equivalent to the JS estimator in that comparison.

APPENDIX E SUMMARY OF SSL METHODS FOR GNNS

The survey summarizes both contrastive and predictive SSL methods for GNNs and provides supplementary tables to support their comparison.

  • Contrastive methods reviewed in the survey are summarized in Supplementary Table 1.
  • Predictive methods reviewed in the survey are summarized in Supplementary Table 2.
  • The developed DIG-sslgraph library is documented through an overview in Supplementary Figure 1.

APPENDIX G EFFICIENCY AND DOWNSTREAM ACCURACY OF DIG IMPLEMENTATIONS

The appendices document standardized summaries and comparisons for SSL methods, including DIG implementations, efficiency, downstream accuracy, and evaluation components.

  • Supplementary Table 3 compares training time, GPU memory, and downstream accuracy between original and DIG implementations under a common experimental environment.Experiments use the unsupervised setting, the same environment and device, a single NVIDIA V100 GPU, and standard dataset splits.
  • Table 2 organizes contrastive GNN methods by objective, representation level, view generation, and targeted downstream-task level.Graph-level methods are marked G and node-level methods N; link prediction is treated as node-level.
  • Table 3 categorizes predictive GNN methods by supervision source, sub-category, pretext task, and self-supervision training paradigm.The paradigms include unsupervised representation learning, unsupervised pretraining, and auxiliary learning.
  • Table 4 compares four pre-implemented DIG methods: GraphCL, MBGRL, InfoGraph, and GRACE.The comparison covers efficiency through training time and GPU memory, alongside downstream accuracy.
  • The sslgraph library within DIG provides customizable contrastive frameworks, pre-implemented contrastive models, data interfaces, and evaluation tools for contrastive and predictive methods.
Loading 2102.10757v5…