Source-linked AI summary
Self-supervised Learning on Graphs: Deep Insights and New Direction
Wei Jin, Tyler Derr, Haochen Liu, Yiqi Wang, Suhang Wang, Zitao Liu, Jiliang Tang
TL;DR
The paper addresses how GNNs can more fully exploit unlabeled nodes for node classification, given that graph structure and dependencies make image- and text-based SSL difficult to transfer. It empirically studies basic graph pretext tasks and proposes SelfTask, whose advanced tasks achieve state-of-the-art performance on real-world graph datasets.
Problem
GNNs use unlabeled nodes through neighborhood aggregation, but this does not thoroughly exploit their information, while graph structure and non-i.i.d. dependencies complicate SSL design.
Method
The paper empirically analyzes basic attribute- and structure-based pretext tasks, then uses the findings to design advanced SelfTask pretext tasks for GNN node classification.
Results
SelfTask pretext tasks achieve state-of-the-art performance on various real-world graph datasets, and ContextLabel improves GCN by 1.4%, 1.1% and 3.0% on Cora, Citeseer and Pubmed, respectively.
Takeaways & Limitations
Empirical insights into when and why graph SSL works can inspire advanced pretext tasks that improve GNN node-classification performance.
Takeaways & Limitations
SelfTask-Distance2Labeled is not scalable on Reddit because it requires shortest-path distances from labeled to unlabeled data.
Abstract
from arXiv · showhide
The success of deep learning notoriously requires larger amounts of costly annotated data. This has led to the development of self-supervised learning (SSL) that aims to alleviate this limitation by creating domain specific pretext tasks on unlabeled data. Simultaneously, there are increasing interests in generalizing deep learning to the graph domain in the form of graph neural networks (GNNs). GNNs can naturally utilize unlabeled nodes through the simple neighborhood aggregation that is unable to thoroughly make use of unlabeled nodes. Thus, we seek to harness SSL for GNNs to fully exploit the unlabeled data. Different from data instances in the image and text domains, nodes in graphs present unique structure information and they are inherently linked indicating not independent and identically distributed (or i.i.d.). Such complexity is a double-edged sword for SSL on graphs. On the one hand, it determines that it is challenging to adopt solutions from the image and text domains to graphs and dedicated efforts are desired. On the other hand, it provides rich information that enables us to build SSL from a variety of perspectives. Thus, in this paper, we first deepen our understandings on when, why, and which strategies of SSL work with GNNs by empirically studying numerous basic SSL pretext tasks on graphs. Inspired by deep insights from the empirical studies, we propose a new direction SelfTask to build advanced pretext tasks that are able to achieve state-of-the-art performance on various real-world datasets. The specific experimental settings to reproduce our results can be found in \url{https://github.com/ChandlerBang/SelfTask-GNN}.
1 Introduction
The paper studies how self-supervised learning can better exploit unlabeled nodes for GNN-based node classification. It analyzes graph-specific challenges and opportunities, then proposes SelfTask based on empirical insights.
- GNNs naturally use labeled and unlabeled nodes, but simple neighborhood aggregation cannot fully exploit the information in unlabeled nodes.
- Graph SSL is difficult to transfer from images and text because graph nodes have complex attributes, topology, and non-i.i.d. dependencies.
- Graph complexity also provides rich structural and attribute information from which diverse self-supervised pretext tasks can be constructed.
- The paper empirically studies when, why, and which SSL strategies work with GNNs for node classification.
- SelfTask uses findings from basic pretext-task analyses to design advanced tasks that achieve state-of-the-art performance on various graph datasets.
2 Problem Statement
The paper formalizes SSL for semi-supervised node classification on a graph with adjacency structure, node features, and partially labeled nodes. A self-supervised loss is integrated with the task-specific loss to improve generalization on unlabeled data.
- A graph is represented as G = (A, X), where A encodes node links and X contains node features.
- The node-classification setting provides labels for only a subset of graph nodes, leaving the remaining nodes unlabeled.
- The paper defines SSL for GNNs as constructing a pretext task and loss Lself that integrate with the task-specific loss Ltask.
- The resulting GNN is trained to generalize better on unlabeled graph data.
3 Basic Pretext Tasks on Graphs
The paper develops graph-specific pretext tasks from structural and attribute information, using local, global, node-level, and pairwise perspectives. These tasks guide GNN representations to preserve connectivity, topology, or attribute information.
- 3 Basic Pretext Tasks on Graphs: Basic graph SSL tasks draw self-supervised signals from graph structure or node attributes.
- 3.1 Local Structure Information: Local structure tasks use node properties or node pairs to preserve neighborhood information and connectivity.
- 3.1.1 Local Structure Information: NodeProperty predicts node degree as a local structural signal, while EdgeMask reconstructs randomly masked edges using linked and unlinked node pairs.
- 3.1.1 Local Structure Information: EdgeMask uses node embeddings, cross-entropy loss, and binary link labels to guide GNNs toward local connectivity information.
- 3.1.2 Global Structure Information: Global structure tasks capture a node’s graph-wide position through pairwise distances or distances to cluster centers.
- 3.2 Attribute Information: Attribute-based tasks reconstruct masked features or preserve similarity relationships between nodes, with PCA used before AttributeMask for high-dimensional sparse features.
4 Preliminary Analysis
The paper compares joint and two-stage strategies for integrating self-supervised pretext tasks into GNNs, then analyzes which information helps node classification and why. Its empirical analysis favors joint training and global structure-based self-supervision, motivating task-specific pretext tasks.
- Training Strategies: Joint training optimizes self-supervised and downstream losses simultaneously, whereas two-stage training pre-trains on the pretext task before fine-tuning downstream.Joint training uses shared feature extraction with separate adaptation processes; two-stage training shares the feature extractor after pre-training.
- Training Strategies: Joint training outperforms two-stage training in most settings and requires tuning only the self-supervision weight λ, while two-stage performance is highly sensitive to strategy choices.The authors therefore identify joint training as the better integration strategy in their empirical analysis.
- Empirical Findings: SSL pretext tasks improve the vanilla GCN, but their utility varies substantially across task types and information sources.The experiments evaluate six basic pretext tasks on Cora, Citeseer, and Pubmed using joint and two-stage strategies.
- Empirical Findings: Global self-supervision based on PairwiseDistance, Distance2Cluster, and PairwiseAttrSim improves node classification, including over 2% improvement on Cora, whereas NodeProperty, EdgeMask, and AttributeMask improve GCN by less than 0.3%.Across datasets, the best-performing method is consistently one that includes an SSL pretext task, with the strongest results coming from global structure information.
- Why SSL Works: GCN representations already preserve NodeProperty information and some EdgeMask information, limiting the additional benefit of corresponding pretext tasks.PairwiseDistance shows smaller representation-versus-feature differences, while Pubmed’s original features already exceed 80% accuracy on EdgeMask.
- Why SSL Works: The proposed direction extends structural or attribute equivalence to regular task equivalence by constructing neighborhood-label-based pretext vectors for node classification.This task-specific formulation uses labeled-node information to define similarity at the downstream-task level.
5 Advanced Pretext Tasks on Graphs
SelfTask addresses the limits of structure- or attribute-only supervision by incorporating task-specific label information into graph pretext tasks. The section develops distance-, context-, ensemble-, and correction-based variants that use labeled nodes and iteratively refined labels.
- Structure- or attribute-only pretext tasks may add little when GCN already preserves those information types, motivating task-specific SelfTask objectives.SelfTask extends regular task equivalence to class-label information.
- SelfTask-Distance2Labeled predicts each node’s distance vector to labeled nodes, combining global structure with task-specific label information.For each class, the method uses average, minimum, and maximum shortest-path lengths from an unlabeled node to labeled nodes.
- ContextLabel constructs a neighbor label-distribution context vector for each unlabeled node using graph structure, attributes, and currently labeled nodes.The context considers nodes within a k-hop neighborhood and produces a K-dimensional label distribution.
- Label information can be extended to unlabeled nodes with Label Propagation using structure or Iterative Classification using both structure and attributes.The framework also permits ensembling LP and ICA predictions before constructing context-label distributions.
- CorrectedLabel enhances ContextLabel by alternating GNN training with label correction based on class prototypes selected from embedding similarity.Prototype density identifies the top-p most mutually similar nodes in each class, whose similarities guide corrected labels.
6 Experiments
Experiments evaluate SelfTask on benchmark graph datasets under transductive node classification, using joint SSL-GNN training and parameter analyses. SelfTask generally improves GCN and baselines, with particularly strong gains when labels are sparse.
- Experimental Settings: Experiments use Cora, Citeseer, Pubmed, and Reddit benchmark datasets in the transductive setting.Reddit contains 232,965 nodes, 57,307,946 edges, 210 classes, and 5,414 node features.
- Experimental Settings: The evaluation uses a 2-layer GCN backbone and joint training, with λ searched over {1, 5, 10, 50, 100, 500}.The SSL loss uses first-layer hidden representations followed by a linear task-specific head.
- Experimental Settings: SelfTask-Distance2Labeled is not scalable on Reddit because it requires shortest-path distances between labeled and unlabeled nodes.
- Performance Comparison: SelfTask-ContextLabel-ICA improves GCN by 1.4%, 1.1%, and 3.0% on Cora, Citeseer, and Pubmed, respectively, achieving state-of-the-art performance.Most SelfTask pretext tasks outperform Self-Training and M3S, while basic SSL tasks usually yield smaller or less consistent gains.
- Fewer Labeled Samples: SelfTask remains effective with very limited labels and improves GCN by 7.2% with 5 samples per class on Citeseer.Across the sparse-label settings studied, SelfTask consistently outperforms the reported baselines.
- Parameter Analysis: Label correction outperforms no correction for α > 0 on the Cora parameter analysis.The comparison evaluates SelfTask-CorrectedLabel-ICA while varying λ and α.
7 Related Work
Related work situates the paper at the intersection of self-supervised learning and graph neural networks. Prior studies develop graph SSL signals from clustering, global context, and established spectral or spatial GNN formulations.
- Self-Supervised Learning: Self-supervised learning generates additional supervised signals through pretext tasks to address limited labeled training data.The related work surveys image-domain SSL before turning to graph methods.
- Self-Supervised Learning: Prior graph SSL methods include clustering-based updates and global node-context supervision for learning node embeddings.
- Graph Neural Networks: GNNs are commonly categorized as spectral or spatial methods, with GCN presented as a simplified spectral approach following ChebNet.
8 Conclusion
The paper studies SSL for GNN node classification through empirical analysis of basic pretext tasks, then proposes SelfTask to construct advanced task-specific pretext tasks. Experiments on real-world datasets show that the advanced method achieves state-of-the-art performance.
- The study empirically examines basic graph SSL pretext tasks to understand when and why SSL works with GNNs.
- It investigates which strategies best integrate SSL with GNNs for node classification.
- SelfTask builds advanced pretext tasks that exploit task-specific self-supervised information.
- Extensive experiments on real-world datasets show that the advanced method achieves state-of-the-art performance.