PaperlayerView exact revision on arXiv

Source-linked AI summary

Graph Self-Supervised Learning: A Survey

Yixin Liu, Ming Jin, Shirui Pan, Chuan Zhou, Yu Zheng, Feng Xia, Philip S. Yu

arXiv:2103.00111v5cs.LG

TL;DR

Graph learning has relied heavily on manual labels, while graph SSL must address irregular topology and dependencies between nodes. This survey unifies and classifies graph SSL methods, reviews their resources and applications, and identifies challenges including limited theory, robustness, and coverage of complex graph types. Its central outcome is a comprehensive synthesis organized into four method categories with accompanying comparisons, datasets, implementations, and future directions.

  • Problem

    Graph deep learning commonly depends on costly manual labels, while graph SSL faces topology and node-dependency challenges not present in regular-grid CV and NLP data.

  • Method

    The survey mathematically formalizes graph SSL, organizes methods into four categories, and synthesizes approaches, resources, applications, and future directions.

  • Results

    The paper provides a comprehensive review with mathematical summaries, method comparisons, datasets, evaluation resources, open-source codes, and application coverage.

  • Takeaways & Limitations

    The unified taxonomy and collected resources support systematic understanding and empirical study of graph SSL across methods and application domains.

  • Takeaways & Limitations

    Graph SSL still lacks a solid theoretical foundation, and most methods assume perfect inputs despite noisy real-world data and GNN vulnerability to adversarial attacks.

Abstract

from arXiv · show

Deep learning on graphs has attracted significant interests recently. However, most of the works have focused on (semi-) supervised learning, resulting in shortcomings including heavy label reliance, poor generalization, and weak robustness. To address these issues, self-supervised learning (SSL), which extracts informative knowledge through well-designed pretext tasks without relying on manual labels, has become a promising and trending learning paradigm for graph data. Different from SSL on other domains like computer vision and natural language processing, SSL on graphs has an exclusive background, design ideas, and taxonomies. Under the umbrella of graph self-supervised learning, we present a timely and comprehensive review of the existing approaches which employ SSL techniques for graph data. We construct a unified framework that mathematically formalizes the paradigm of graph SSL. According to the objectives of pretext tasks, we divide these approaches into four categories: generation-based, auxiliary property-based, contrast-based, and hybrid approaches. We further describe the applications of graph SSL across various research fields and summarize the commonly used datasets, evaluation benchmark, performance comparison and open-source codes of graph SSL. Finally, we discuss the remaining challenges and potential future directions in this research field.

1 INTRODUCTION

Graph self-supervised learning addresses the label dependence of graph deep learning, whose irregular topology and correlated nodes require designs distinct from SSL for images and text. This survey formalizes and organizes the field while reviewing methods, resources, applications, and future directions.

  • Graph deep learning has often relied on manually labeled data, whose collection and annotation can be prohibitive for large-scale or domain-demanding datasets.
  • SSL reduces manual-label dependence by learning from automatically derived supervision in handcrafted auxiliary pretext tasks.
  • Graph SSL differs from CV and NLP because graph data occupy irregular non-Euclidean spaces and nodes are correlated through topology.
  • The survey proposes a mathematically formalized framework and categorizes graph SSL into generation-based, auxiliary property-based, contrast-based, and hybrid methods.
  • It reviews classical and recent approaches with fine-grained classifications, mathematical descriptions, comparisons, datasets, benchmarks, implementations, applications, and open directions.
  • Compared with related surveys, it focuses specifically on graph-structured data and offers a unified encoder-decoder framework, finer taxonomy, and broader resources and applications.

2 DEFINITION AND NOTATION

This section defines graph SSL terminology and graph notation, including manual versus pseudo labels, graph structures, attributes, and common GNN representations. It also introduces the notation used to describe graph encoders, readouts, and graph data.

  • Term Definitions: Manual labels are human-annotated, whereas pseudo labels are acquired automatically from data and generally cost less to obtain.
  • Notations: A plain graph is G = (V, E), with nodes V, edges E, neighborhoods N(v_i), and adjacency matrix A representing topology.
  • Notations: An attributed graph associates nodes and/or edges with features, commonly represented by G = (V, E, X) when only nodes have features.
  • Notations: Most reviewed methods use GNN encoders to transform raw node features X into compact representations H using graph connectivity A.
  • Notations: Readout functions aggregate node-level representations H into graph-level representations h_G.

3 FRAMEWORK AND CATEGORIZATION

The paper formalizes graph SSL with an encoder-decoder framework and categorizes methods by pretext objectives, training schemes, and downstream task scale. These dimensions organize reconstruction, auxiliary-property, contrastive, and hybrid approaches for learning graph representations.

  • Unified Framework: Graph SSL uses an encoder to produce node representations and a pretext decoder to define self-supervised objectives over unlabeled graph data.The learned encoder can then support downstream supervised tasks through a downstream decoder.
  • Graph SSL Categories: The framework distinguishes generation-based, auxiliary property-based, contrastive-based, and hybrid methods according to pretext decoders and objective functions.Generation methods reconstruct features or adjacency; auxiliary-property methods use graph properties; contrastive methods compare augmented instances; hybrid methods combine objectives.
  • Graph SSL Categories: Generation-based methods reconstruct graph features or structure, whereas auxiliary property-based methods use regression or classification objectives over crafted graph properties.Auxiliary properties may include node degree, distances to clusters, graph partitions, or cluster indices.
  • Graph SSL Categories: Contrastive methods estimate agreement between differently augmented instances, while hybrid methods combine multiple pretext decoders or training objectives.Contrastive objectives maximize agreement between positive pairs and minimize agreement between negative samples; hybrid methods must balance their components.
  • Training Schemes: Graph SSL training is organized into Pre-training and Fine-tuning, Joint Learning, and Unsupervised Representation Learning schemes.PF initializes fine-tuning with a pretrained encoder, JL jointly optimizes pretext and downstream tasks, and URL freezes the encoder during downstream training on the same dataset.
  • Downstream Tasks: Downstream tasks are divided by prediction target into node-level, link-level, and graph-level tasks.Node tasks predict node properties, link tasks predict edge or node-pair properties, and graph tasks predict properties for individual graphs.

4 GENERATION-BASED METHODS

Generation-based graph SSL reconstructs graph information from original or perturbed inputs, targeting either features or structure. These branches use different decoders and losses, yielding representations with different information emphasis.

  • Generation-based methods: Generation-based methods reconstruct input graph data and commonly use full graphs or subgraphs to recover features, structure, or components of both.They obtain supervision directly from the input data rather than manual labels.
  • Feature generation: Feature generation approaches recover feature information from original or perturbed graphs, including masked features, noisy features, and clean features.Masked feature regression predicts missing node or edge features from unmasked information, while denoising methods recover raw features from corrupted inputs.
  • Structure generation: Structure generation approaches recover topological information, most often by reconstructing the full or partial adjacency matrix.GAE encodes the original graph into node embeddings and decodes them into adjacency probabilities using a binary cross-entropy objective.
  • Structure generation: Structure-generation variants reconstruct masked edges or discarded connections instead of rebuilding the entire graph.Denoising Link Reconstruction randomly drops existing edges and learns to recover them with a similarity-based decoder.
  • Comparison: Structure and feature generation use distinct decoder and loss designs, with structure generation emphasizing node-pair information and feature generation emphasizing node-level knowledge.The difference follows from their respective edge-reconstruction and feature-reconstruction targets.

5 AUXILIARY PROPERTY-BASED METHODS

Auxiliary property-based graph SSL derives supervision from automatically available node-, link-, and graph-level properties. The survey divides these methods into classification with discrete pseudo-labels and regression with continuous property targets.

  • Overview: Auxiliary property-based methods use graph-derived properties as pseudo-label supervision, avoiding the manually annotated labels used in supervised learning.Their training paradigm resembles supervised learning because both use sample-label pairs, but pseudo-labels are generated from graph data.
  • Taxonomy: The survey divides auxiliary property-based methods into auxiliary property classification and auxiliary property regression.Classification uses discrete pseudo-labels, whereas regression predicts continuous property values.
  • Auxiliary property classification: Auxiliary property classification creates discrete pseudo-labels, trains a classifier with cross-entropy loss, and includes clustering-based and pair relation-based methods.Clustering-based methods assign nodes to groups, while pair relation-based methods label node pairs according to contextual relationships.
  • Pair relation-based methods: Pair relation-based classification can predict shortest-path distance or compare centrality scores between node pairs.S2GRL uses shortest-path distance as the label, while Centrality Score Ranking predicts the relative ordering of centrality values.
  • Auxiliary property regression: Auxiliary property regression predicts continuous node- or pair-wise properties with mean squared error.Examples include node degree, distances to graph-cluster centers, and cosine similarity between node features.

6 CONTRAST-BASED METHODS

Contrast-based graph SSL creates augmented graph views and learns representations by maximizing agreement between semantically similar instances while minimizing agreement with unrelated ones. The survey organizes these methods around augmentations, contrastive objectives, mutual-information estimation, and same-scale versus cross-scale designs.

  • Overview: Contrast-based methods maximize mutual information between similar augmented instances and minimize it between unrelated instances.The approach uses a pretext decoder to estimate agreement between graph instances.
  • Graph Augmentations: Graph augmentations modify node attributes, graph structure, or both to generate alternative graph instances for contrastive learning.The survey covers feature masking, feature shuffling, edge modification, graph diffusion, and subgraph sampling.
  • Attributive Augmentations: Node feature masking zeros selected node vectors or feature channels, while node feature shuffle partially perturbs feature assignments across nodes.Masking may be random or adaptive, with importance-based masking assigning higher probabilities to less important nodes.
  • Structural Augmentations: Structural augmentations either drop or insert edges through edge modification or inject global topology through weighted graph diffusion.Diffusion connects nodes with indirectly connected neighbors using weighting coefficients and transition matrices.
  • Hybrid Augmentations: Hybrid augmentation combines attribute and topology changes; subgraph sampling selects nodes and their linkages using uniform, random-walk, or importance-based sampling.Contrastive methods may also combine multiple strategies, such as edge dropping with feature masking or diffusion with subgraph sampling.
  • Contrastive Objectives: Contrastive pretext tasks are divided into same-scale and cross-scale methods, with same-scale approaches further separated into node-level and graph-level contrasts.Same-scale methods compare instances at equal scales, whereas cross-scale methods compare representations at different scales.

7 HYBRID METHODS

Hybrid graph SSL combines multiple pretext tasks in a weighted or unweighted multi-task objective. The survey covers combinations within generation, between generation and contrast, among contrastive tasks, and among auxiliary property tasks.

  • Overview: Hybrid methods combine two or more self-supervised objectives in a multi-task learning framework.The overall objective is typically a weighted sum, with trade-off weights assigned to individual losses.
  • Generation Combinations: Generation-based hybrids jointly reconstruct masked attributes and graph structure, as in GPT-GNN’s feature and edge generation tasks.GPT-GNN masks nodes and edges before training the encoder with attribute-generation and edge-generation objectives.
  • Generative–Contrastive Combinations: Other hybrids combine generative and contrastive objectives, such as adjacency reconstruction with mutual-information maximization.GMI uses a JSD-based contrastive target for node embeddings and neighboring features alongside BCE adjacency reconstruction.
  • Contrastive Combinations: Some methods integrate several contrastive views, including subgraph–subgraph, graph–graph, and graph–contextual comparisons.GraphLoG uses InfoNCE for three contrastive objectives, while other methods combine same-scale and cross-scale contrasts.
  • Auxiliary Property Combinations: Auxiliary property-based hybrids jointly model structural and semantic properties such as denoising links, centrality, clusters, contextual properties, and motifs.These tasks are used to capture transferable generic graph structures or node- and graph-level properties.

8 EMPIRICAL STUDY

The empirical study compares graph SSL methods for node classification under unsupervised representation learning and label-accessible settings. Results indicate that carefully designed contrastive and hybrid methods are competitive, while direct transfer of objectives from visual learning is not consistently beneficial.

  • Experimental Setup: The study evaluates representative graph SSL methods on node classification across four benchmark datasets.It compares methods whose encoders use only SSL pretext tasks with methods trained using accessible labels, alongside GCN and GAT baselines.
  • Results: Methods trained with accessible labels generally outperform methods using only unsupervised pretext-task training.The comparison separates these settings into PF/JL and URL groups, respectively.
  • Results: Early random-walk contrastive and autoencoder-based generative methods perform worse than most newer graph SSL approaches.The survey attributes this pattern to simpler unsupervised targets that do not fully exploit available graph supervision signals.
  • Results: Advanced visual-domain objectives such as BYOL and Barlow Twins do not consistently outperform other methods when directly applied to graph SSL.The result indicates that borrowing self-supervised objectives from other domains does not always improve graph performance.
  • Results: Representative contrast-based methods outperform generalization-based and auxiliary property-based methods in the reported comparisons.The survey presents this pattern as evidence of the effectiveness of contrastive pretext tasks and remaining room for other approaches.
  • Results: Hybrid methods show competitive performance, with some surpassing supervised baselines in the node-classification comparisons.The survey links this outcome to combining supervision signals from diverse perspectives.

9 PRACTICAL APPLICATIONS

Graph SSL has been applied across disciplines, including recommender systems. In recommendation, surveyed methods address cold-start users and items, pre-training, selection bias, and message dropout through reconstruction and contrastive objectives.

  • Recommender Systems: Graph SSL supports recommender-system applications involving cold-start users and items, recommendation pre-training, selection bias, and message dropout.The survey identifies these as issues addressed by recent graph SSL recommendation methods.
  • Recommender Systems: Reconstruction-based pretext tasks can pre-train GNNs on cold-start users and items.This application targets recommendation settings where user or item interaction information is limited.
  • Recommender Systems: Contrastive tasks are used for hypergraph representation learning in social- and session-based recommendation.The survey cites separate methods for social recommendation and session-based recommendation.
  • Recommender Systems: A debiased graph contrastive module is used to address message dropout and reduce selection bias in GNN-based recommender systems.The method introduces a debiased loss within the graph contrastive learning module.

10 FUTURE DIRECTIONS

Graph SSL still faces theoretical, interpretability, robustness, and graph-type limitations. Future work should develop stronger foundations, safer models, and pretext tasks tailored to complex graphs.

  • Theoretical Foundation: Graph SSL lacks a solid theoretical foundation, with many methods designed intuitively and evaluated mainly through empirical experiments.The survey calls for connections between empirical SSL and graph signal processing or spectral graph theory.
  • Interpretability and Robustness: Most graph SSL methods prioritize downstream performance with black-box models, leaving learned representations and predictions insufficiently explainable and robust.The survey highlights risk-sensitive applications such as fraud detection and vulnerability to adversarial attacks.
  • Pretext Tasks for Complex Types of Graphs: Current work concentrates on attributed graphs, while relatively few methods address heterogeneous or spatial-temporal graphs.Complex graphs require pretext tasks that capture distinctive characteristics such as temporal dynamics.
  • Pretext Tasks for Complex Types of Graphs: Future directions include SSL tasks for complex graphs and broader graph types such as hypergraphs.The survey emphasizes making specific data characteristics central to pretext-task design.
  • Augmentation for Graph Contrastive Learning: Graph contrastive learning has fewer diverse augmentation strategies than computer vision, motivating further study of graph-specific augmentations.The passage begins a comparison with image augmentations that preserve semantic information across views.

11 CONCLUSION

The survey organizes graph SSL into a unified framework and four categories while reviewing its development, notation, GNN foundations, and common losses. It also provides resources for understanding graph SSL methods and applications.

  • 11 CONCLUSION: The survey presents a unified framework and systematic taxonomy covering generation-based, auxiliary property-based, contrast-based, and hybrid graph SSL approaches.It also compares methods mathematically and collects datasets, evaluation methods, performance comparisons, and open-source codes.
  • 11 CONCLUSION: Graph SSL developed from random-walk node embeddings and graph autoencoders toward contrastive learning and graph property mining.The timeline identifies DeepWalk, LINE, node2vec, GAE/VGAE, and DGI among representative methods.
  • 11 CONCLUSION: The paper formalizes core graph-learning components including GNN node representations, aggregation and combination functions, and graph-level readout.Readout functions fuse node representations into graph representations using permutation-invariant or pooling operations.
  • 11 CONCLUSION: The paper introduces MSE, CE, and BCE as commonly used losses for graph-learning formulations.BCE is presented as the binary form of cross-entropy for scalar predictions and targets.

APPENDIX C DOWNSTREAM TASKS

The appendix defines node-, link-, and graph-level downstream tasks and describes their supervised objectives and evaluation. Graph-classification results show cross-scale contrastive methods generally outperform same-scale methods, while SSL can sometimes hurt downstream performance.

  • Node-level tasks: Node-level tasks include node regression and classification, with node representations passed to decoders and cross-entropy typically used for classification.Training uses labeled nodes and indexes their embeddings from the full embedding matrix.
  • Link-level tasks: Link-level tasks include edge classification and link prediction, using pairs of node embeddings as decoder inputs.The objective is defined over a labeled training edge set with cross-entropy serving as the loss for edge classification.
  • Graph-level tasks: Graph-level classification and regression use graph representations formed by aggregating node embeddings before decoding.The graph-classification objective is trained over a labeled set of graphs.
  • Graph classification evaluation: Cross-scale contrast-based methods generally perform better than same-scale methods on graph-level learning tasks.The survey suggests that cross-scale contrast may improve global graph-level representations.
  • Graph classification evaluation: SSL pre-trained encoders sometimes negatively affect downstream tasks, and graph-level SSL lacks unified evaluation benchmarks.The appendix identifies both negative transfer and inconsistent dataset usage as areas needing further investigation.

APPENDIX E DATASETS

The appendix surveys benchmark datasets spanning citation, co-purchase, social, and biochemical graphs, and summarizes their structures, features, labels, and resources. It also points readers to experimental and implementation summaries.

  • Citation networks: Citation-network datasets represent papers or authors as nodes, relationships such as citations or co-authorship as edges, and textual context as node features.Labels commonly indicate fields of study.
  • Co-purchase networks: Amazon co-purchase graphs represent goods as nodes and frequent co-purchases as edges, with product-review features and category labels.The cited examples are Amazon Computers and Amazon Photo.
  • Social networks: Social-network datasets model users or posts and their interactions, with labels representing communities or researcher fields depending on the dataset.Examples include Reddit discussion data and COLLAB ego-networks.
  • Resource summaries: The survey provides summaries of benchmark datasets, experimental results, and open-source implementations.These resources are organized in Tables 7–9.
  • Bio-chemical graphs: Biochemical benchmarks include protein-interaction, molecular, and chemical-compound graphs with labels for properties such as enzyme status, carcinogenicity, or cancer-cell activity.Examples include PPI, MUTAG, PROTEINS, PTC, NCI-1, and OGB molecule benchmarks.

APPENDIX F OPEN-SOURCE IMPLEMENTATIONS

The survey collects available open-source implementations of reviewed graph SSL approaches and provides their source-code hyperlinks in Table 9. It also lists several application-specific methods and their SSL pretext tasks or objectives.

  • Open-source implementations are collected for reviewed graph SSL approaches when code is available, with hyperlinks provided in Table 9.
  • COAD uses same-scale contrastive learning to pre-train an expert encoder for zero-shot expert linking.
  • SLAPS combines denoising node-feature generation with classification for graph structure learning, while SCRL uses prototype-based pretext tasks for few-label graph learning.
  • SDGE trains a community-detection model with a same-scale contrastive learning objective.
Loading 2103.00111v5…