PaperlayerView exact revision on arXiv

Source-linked AI summary

Self-supervised Learning on Graphs: Contrastive, Generative,or Predictive

Lirong Wu, Haitao Lin, Zhangyang Gao, Cheng Tan, Stan. Z. Li

arXiv:2105.07342v4cs.LGcs.AI

TL;DR

Graph learning relies on labels that are often expensive, limited, or inaccessible, motivating self-supervised learning for graphs. The survey categorizes graph SSL methods and reviews them through a unified mathematical, implementation, and experimental framework. It also identifies technical limitations and future directions, including the need for more principled graph augmentation strategies.

  • Problem

    Graph learning often depends on abundant, carefully labeled data even though graph annotations can be expensive, limited, and inaccessible.

  • Method

    The survey categorizes graph SSL as contrastive, generative, or predictive and unifies mathematical, implementation, dataset, evaluation, and experimental analyses.

  • Results

    The survey provides a comprehensive review, a unified mathematical framework, implementation summaries, and experimental studies of graph SSL methods.

  • Takeaways & Limitations

    The taxonomy and consolidated resources support clearer comparisons and future development of graph self-supervised learning.

  • Takeaways & Limitations

    Graph SSL still lacks sufficiently efficient and principled augmentation strategies because graph data are non-Euclidean.

Abstract

from arXiv · show

Deep learning on graphs has recently achieved remarkable success on a variety of tasks, while such success relies heavily on the massive and carefully labeled data. However, precise annotations are generally very expensive and time-consuming. To address this problem, self-supervised learning (SSL) is emerging as a new paradigm for extracting informative knowledge through well-designed pretext tasks without relying on manual labels. In this survey, we extend the concept of SSL, which first emerged in the fields of computer vision and natural language processing, to present a timely and comprehensive review of existing SSL techniques for graph data. Specifically, we divide existing graph SSL methods into three categories: contrastive, generative, and predictive. More importantly, unlike other surveys that only provide a high-level description of published research, we present an additional mathematical summary of existing works in a unified framework. Furthermore, to facilitate methodological development and empirical comparisons, we also summarize the commonly used datasets, evaluation metrics, downstream tasks, open-source implementations, and experimental study of various algorithms. Finally, we discuss the technical challenges and potential future directions for improving graph self-supervised learning. Latest advances in graph SSL are summarized in a GitHub repository https://github.com/LirongWu/awesome-graph-self-supervised-learning.

1 INTRODUCTION

Graph SSL addresses the cost and scarcity of graph labels by learning from unlabeled data through pretext tasks. This survey organizes graph SSL into contrastive, generative, and predictive methods and provides a unified mathematical and empirical review.

  • Motivation: Graph learning often depends on abundant labels that are expensive or inaccessible, limiting applications and increasing risks of over-fitting, poor generalization, and weak robustness.These constraints are especially relevant where annotation requires expert knowledge.
  • Motivation: SSL learns transferable knowledge from unlabeled data through pretext tasks before applying downstream supervision.Its success in computer vision and natural language processing motivates its extension to graphs.
  • Motivation: Graph SSL is promising because graphs combine node features with structure, encode expert-rule information, and often support transductive learning.These properties provide graph-specific sources for designing pretext tasks.
  • Challenges: The paper identifies graph-specific challenges arising from non-rigid topology and correlations among nodes rather than independent samples.These differences complicate direct transfer of SSL designs from images and text.
  • Survey scope: The survey categorizes graph SSL methods as contrastive, generative, and predictive, respectively using inter-data comparisons, intra-data information, and self-generated labels.The taxonomy is illustrated through different sources of self-supervision.
  • Contributions: The survey contributes mathematical summaries, implementation details, datasets, metrics, downstream tasks, open-source codes, and experimental comparisons.It reports 41 datasets and 71 surveyed methods, compared with 20 datasets and 18 methods in one earlier survey and 47 methods in another.

2 PROBLEM STATEMENT

The paper formalizes graph data, graph-level and node-level representations, downstream tasks, GNN encoders, and three graph SSL training strategies. These strategies differ in whether pretraining is fine-tuned, jointly optimized, or frozen before downstream learning.

  • Graph definitions: A graph is represented by nodes and edges, with neighborhoods defined by shortest-path distance and adjacency entries indicating edge presence.Attributed, dynamic, heterogeneous, and spatial-temporal graphs extend this basic representation in different ways.
  • Downstream tasks: Graph downstream tasks are organized into node-level, link-level, and graph-level tasks, using node or graph embeddings followed by optional prediction heads.Examples include node classification, link prediction, and graph regression.
  • Graph neural networks: GNNs update node representations by aggregating neighborhood messages and combining them with prior representations across layers.Graph-level tasks additionally require a READOUT function to aggregate node representations into a graph embedding.
  • Training strategy: Pre-training and Fine-tuning initializes an encoder with self-supervised learning and then optimizes it under downstream supervision.The pretrained parameters θinit initialize the downstream encoder.
  • Training strategy: Joint Learning trains the encoder simultaneously with pretext and downstream tasks, combining their objectives through a trade-off hyperparameter.The pretext task can function as auxiliary multi-task supervision or regularization.
  • Training strategy: Unsupervised Representation Learning freezes pretrained parameters and trains downstream tasks using the resulting representations only.Compared with fine-tuning, this strategy cannot further optimize the encoder using downstream supervision.

3 CONTRASTIVE LEARNING

Contrastive graph SSL is organized around augmentation, pretext-task, and contrastive-objective designs. These components generate views at local, contextual, or global scales and support same-scale or cross-scale contrasting.

  • 3.1 A Unified Perspective: Graph contrastive methods can be unified through data augmentation strategies, pretext tasks, and contrastive objectives.The survey treats graph encoder design as outside its main scope.
  • 3.1 A Unified Perspective: Two views from the same instance form positive pairs, while views from different instances form negative pairs whose agreement is contrasted.Representations are produced from augmented views, potentially using identical or shared-weight encoders.
  • 3.2 Data Augmentation: Graph augmentations are feature-based, structure-based, sampling-based, or adaptive, reflecting the difficulty of transferring image augmentations to non-Euclidean graphs.Feature and structure transformations modify attributes or connectivity; sampling transforms both adjacency and features.
  • 3.2 Data Augmentation: Feature-based augmentation masks or shuffles node and edge attributes, whereas structure-based augmentation perturbs edges, inserts nodes, or diffuses connectivity.Masking locations may be sampled or manually assigned, with constant or noise-based replacement values.
  • 3.2 Data Augmentation: Sampling-based augmentation selects nodes and connected edges using uniform, ego-net, random-walk, importance, or knowledge-based strategies.Ego-net sampling selects an L-hop neighborhood and is described as a special form of BFS sampling.
  • 3.2.4 Adaptive Augmentation: Adaptive augmentation can use gradient information to guide edge removal after independently masking node attributes to create two views.GROC computes contrastive loss between the two stochastic views and uses edge gradients to guide augmentation.
  • 3.4 Contrastive Objectives: Contrastive objectives include mutual-information estimators and triplet-margin losses; mutual-information maximization is not required for contrastive learning.Quadruplet loss adds inter-class constraints and samples negatives more randomly than triplet margin loss.

4 GENERATIVE LEARNING

Generative graph SSL uses graph reconstruction as self-supervision, while predictive methods obtain labels from node properties, context, self-training, or domain knowledge. Reconstruction tasks vary in whether they recover adjacency, node or edge attributes, or masked graph content.

  • 4 GENERATIVE LEARNING: Generative methods treat information embedded in graph data as self-supervision and use graph decoders to reconstruct restricted or corrupted inputs.The survey categorizes generative methods by how reconstruction is performed.
  • 4.1 Graph Autoencoding: Graph completion masks one node’s features and reconstructs them from unmasked neighborhood features.Only unmasked neighboring nodes are used for reconstruction in the described graph-completion setup.
  • 4.1 Graph Autoencoding: Node attribute masking reconstructs multiple nodes’ features simultaneously without requiring neighboring features used for reconstruction to remain unmasked.This distinguishes it from graph completion.
  • 4.1 Graph Autoencoding: Edge attribute masking randomly masks features of an edge set to encourage learning of edge-relation information.The task is designed for graphs with known edge features.
  • 4.1 Graph Autoencoding: Node attribute denoising adds Gaussian noise to node features and reconstructs the clean features.Node embedding denoising instead reconstructs clean node features from noisy embeddings.
  • 4.1 Graph Autoencoding: Adjacency matrix reconstruction perturbs graph edges to obtain a corrupted adjacency matrix and requires reconstruction of the original structure.Because adjacency matrices are usually sparse, cross-entropy may be used instead of MAE in practice.
  • 4.2 Autoregressive Generation: GPT-GNN iteratively reconstructs masked nodes and their connected edges through an autoregressive graph-generation framework.Each iteration generates a masked node and its connected edges, updating the graph for the next step.

5 PREDICTIVE LEARNING

Predictive graph SSL methods generate informative labels from graph data and learn data-label relationships through node properties, contextual information, or iterative/self-training strategies.

  • Predictive methods generate informative labels from graph data and use them as supervision for data-label relationships.
  • 5.1 Node-Property Prediction (NP): Node Property Prediction extracts statistical labels such as node degree or local clustering coefficient through a mapping from graph data to labels.
  • 5.2 Context-based Prediction (CP): Context-based predictive methods construct regression- or classification-based pretext tasks from underlying graph structure and contextual information.
  • 5.2 Context-based Prediction (CP): Distance2Clusters partitions the graph into K clusters, selects the highest-degree node in each cluster as a center, and predicts distances to those centers.
  • 5.2 Context-based Prediction (CP): Distance2Labeled uses task-specific label information instead of unsupervised clustering to find centers, with reported advantages in efficiency and performance over Distance2Cluster.
  • 5.3 Self-Training (ST): Self-training iteratively uses high-confidence predictions as pseudo-labels, moves selected nodes into the labeled set, and retrains the graph encoder.

6 SUMMARY OF THE IMPLEMENTATION

The survey organizes implementation information around task levels, datasets, metrics, code, and comparative experiments to support empirical evaluation of graph SSL methods.

  • The survey records graph properties, pretext tasks, augmentations, objectives, training strategies, and publication years for surveyed works.
  • Graph SSL evaluation covers node-level, link-level, and graph-level tasks, including node and graph classification, graph regression, and link prediction.
  • Common metrics include ROC-AUC and Accuracy for graph classification, MAE for graph regression, and ROC-AP, ROC-PR, and ROC-AUC for link prediction.
  • The survey provides statistics for 41 datasets spanning citation, social, protein, molecule, and other graph categories.
  • Implementation resources include hyperlinks to open-source codes for 71 surveyed works and a GitHub repository tracking graph SSL advances and available code.
  • For fair comparison, the survey evaluates node and graph classification performance across 15 commonly used graph datasets.

7 DISCUSSION

The survey organizes graph SSL methods and compares their development, strengths, and unresolved challenges. It highlights gaps in pretext-task coverage, theoretical foundations, explainability, augmentation, negative sampling, and transfer to downstream tasks.

  • Method development: Graph SSL methods are organized into contrastive, generative, and predictive approaches, with related work tracing their development and inheritance connections.The survey also discusses seminal contributions and provides a timeline of key milestones.
  • Pros and cons: Classical graph SSL algorithms trade off innovation, accessibility, effectiveness, and efficiency across pioneering, knowledge-based, and subsequent method categories.Pioneering methods introduced new tasks or graph types but often had lower downstream performance and higher computational complexity than later work.
  • Open challenges: Pretext-task design remains limited for spatial-temporal and heterogeneous graphs, often using only node- or graph-level information instead of richer temporal or relation information.Most existing work concentrates on attribute graphs.
  • Open challenges: Graph SSL lacks sufficient theoretical foundations because many methods are designed intuitively and evaluated mainly through empirical experiments.The survey identifies theoretical analysis of contrastive losses and their relationship to mutual information as an open direction.
  • Open challenges: Graph augmentation strategies remain insufficient because image-based transformations do not directly fit non-Euclidean graph data and current graph augmentations often rely on node or edge modifications.The survey calls for applicable, adaptable, and better-guided augmentation designs.
  • Open challenges: Negative-sample selection remains inefficient, especially because hard negatives are difficult to estimate without label information and image-specific methods may not transfer directly to graphs.The survey frames narrowing the gap between ideal and practical contrastive learning as requiring further exploration.
  • Open challenges: Graph SSL also needs better explainability and graph-specific techniques for transferring pretrained knowledge to downstream tasks.Current work does not clearly identify what feature patterns, structures, or feature-structure relationships self-supervision learns.

8 CONCLUSION

The paper surveys graph self-supervised learning through a unified mathematical framework and detailed empirical and implementation summaries. It concludes by identifying technical limitations and future directions, including stronger theoretical foundations and explainability.

  • Contributions: The survey develops a unified mathematical framework for graph SSL and summarizes implementation details, similarities, and differences across existing methods.It also conducts a detailed experimental study of self-supervised learning.
  • Future directions: The paper identifies technical limitations and proposes future directions for graph SSL beyond downstream-task performance.The stated priorities include theoretical foundations and explainability.

A. CONTRASTIVE METHODS

The survey covers contrastive graph SSL methods that learn representations by contrasting augmented or differently scaled graph views, including local, contextual, and global information. It presents diverse frameworks involving distillation, label contrast, mutual-information maximization, and hierarchical subgraph selection.

  • A. CONTRASTIVE METHODS: IGSD iteratively performs teacher-student distillation on augmented graphs, updating the teacher through an exponential moving average of student parameters.The framework uses graph encoders, a READOUT function, a prediction head, and symmetric consistency loss.
  • A. CONTRASTIVE METHODS: Other surveyed contrastive methods apply mixup, supervised label contrast, dynamic subgraph sampling, multiplex mutual-information maximization, expert linking, or unsupervised environment representation learning.These designs adapt contrastive learning to settings including dynamic graphs, multiplex graphs, and domain transfer.
  • A. CONTRASTIVE METHODS: KS2L combines intra-model and cross-model knowledge distillation after two linear mappings and graph encoders produce node embedding matrices.Its objective includes separate distillation components for the two models.
  • A. CONTRASTIVE METHODS: Contrast-Reg uses an InfoNCE regularizer to contrast semantically similar node pairs against negative pairs and theoretically addresses representation norm and variance.The stated analysis links these properties to improved generalization performance.
  • A. CONTRASTIVE METHODS: SUGAR selects salient subgraphs with reinforcement learning and maximizes mutual information so subgraph embeddings reflect global structural properties.Selected subgraphs are pooled into a sketched graph for hierarchical representation learning.

B. GENERATIVE METHODS

Generative graph SSL reconstructs masked attributes, structures, or task-relevant targets from graph representations. The surveyed methods apply these pretext tasks to general graph learning and domains such as medication recommendation, recommendation cold starts, and transferable graph encoders.

  • B. GENERATIVE METHODS: Node Attribute Masking masks multiple node feature vectors simultaneously and trains a model to reconstruct the masked features.Unlike graph completion, message passing need not rely only on unmasked neighboring features.
  • B. GENERATIVE METHODS: G-BERT uses self-prediction to reconstruct masked medical codes and dual-prediction to reconstruct one embedding type from another.The framework combines GNNs and BERT for medication recommendation.
  • B. GENERATIVE METHODS: SLAPS reconstructs node attributes to infer a task-specific latent graph structure before applying a graph neural network.The inferred structure is part of the downstream graph-learning pipeline.
  • B. GENERATIVE METHODS: Pretrain-Recsys simulates cold-start scenarios, uses an attention-based meta aggregator, and reconstructs target embeddings as its self-supervised task.The aggregator is intended to reduce the impact of cold-start neighbors.
  • B. GENERATIVE METHODS: Graph-Bert pre-trains with node attribute reconstruction and structure recovery, then transfers the model directly or with fine-tuning when supervised labels are available.Its pre-trained representations are used for downstream tasks.

C. PREDICTIVE METHODS

Predictive graph SSL uses graph-derived signals such as distances, centrality rankings, local label distributions, topology transformations, and global context as pretext targets. The survey also includes applications that construct pseudo-labels or exploit unlabeled program data.

  • C. PREDICTIVE METHODS: PairwiseDistance predicts discretized shortest-path categories between sampled node pairs to preserve global topology information.Distances are grouped into four categories, with paths longer than four truncated to reduce computation and noisy long-range influence.
  • C. PREDICTIVE METHODS: TopoTER perturbs connected and disconnected edge pairs, then predicts transformation types from differences between original and perturbed node embeddings.The four targets distinguish adding, removing, keeping connections, and keeping disconnections.
  • C. PREDICTIVE METHODS: Centrality Score Ranking treats relative orders from four centrality measures as pseudo-labels for predicting node rank scores.The measures are eigencentrality, betweenness, closeness, and subgraph centrality.
  • C. PREDICTIVE METHODS: ContextLabel constructs each node’s local label distribution using labeled nodes and Label Propagation assignments within a k-hop neighborhood.The resulting objective is formulated as a multi-class classification problem.
  • C. PREDICTIVE METHODS: HCM uses global context prediction to model local and global contextual information for anomaly detection on attributed networks.The task is motivated by anomalous nodes differing from normal nodes in structures and attributes.
  • C. PREDICTIVE METHODS: SEF derives pseudo-labels from Louvain clustering and uses learned edge-attention coefficients as edge features, while DrRepair pre-trains on unlabeled programs represented as feedback graphs.These methods extend predictive graph SSL to node classification and program repair.

E. SUMMARY OF IMPLEMENTATION DETAILS

The survey summarizes implementation details for the reviewed graph SSL methods, covering downstream task levels, task-specific evaluation metrics, and commonly used datasets.

  • E. SUMMARY OF IMPLEMENTATION DETAILS: Implementation tables report whether downstream tasks are node-, link-, or graph-level and pair evaluation metrics with specific tasks.They also list commonly used datasets across the surveyed methods.

F. SUMMARY OF COMMON DATASETS

The survey compiles common graph datasets and provides accompanying open-source implementation resources to support graph self-supervised learning research.

  • Table A5 summarizes common graph datasets and reports their category, graph and node statistics, edge statistics, node-attribute dimensionality, class number, and citation papers.
  • Table A6 summarizes open-source code for the surveyed works, marking unavailable implementations as “N.A.”.
  • The survey maintains a GitHub repository of recent graph self-supervised learning advances and associated code, with realtime updates as papers and implementations become available.

H. EXPERIMENTAL STUDY RESULTS

The experimental study compares graph self-supervised methods across node- and graph-classification benchmarks under fixed dataset splits. Results indicate dataset-dependent performance, scale-related reliability concerns, and remaining gaps for generative and predictive methods.

  • 23 methods are compared on 8 datasets for node classification, while 11 representative methods are compared on 7 datasets for graph classification using accuracy.
  • Fixed dataset splits are adopted to support fair comparisons, while unavailable or impractical implementations are reported with dashes.
  • No single method performs best across all datasets, indicating that graph self-supervised algorithms depend on graph properties and lack sufficient generality.
  • Small-dataset success may not transfer to large datasets, so evaluations should include graphs of different scales because small-scale results may be biased.
  • Most generative and predictive methods target node-level tasks and remain behind state-of-the-art contrastive methods, leaving their potential for further exploration.

I. TIMELINE

The survey provides a timeline of graph self-supervised learning milestones and organizes supporting material across method summaries, implementation details, datasets, codes, and benchmark tables.

  • The timeline lists publication dates based on arXiv records and shows inheritance connections between methods.
  • The survey includes appendix summaries of surveyed papers organized by pretext-task motivation.
  • Implementation-detail tables summarize downstream task levels, evaluation metrics, and datasets used by surveyed works.
  • Additional appendix tables cover common datasets, surveyed open-source codes, and node- and graph-classification accuracy results.
  • Figure A1 traces key milestones in graph self-supervised learning and marks generative, contrastive, and predictive methods with distinct colors.
Loading 2105.07342v4…