Source-linked AI summary
When Does Self-Supervision Help Graph Convolutional Networks?
Yuning You, Tianlong Chen, Zhangyang Wang, Yang Shen
TL;DR
Self-supervision was rarely explored for GCNs despite its use in CNNs for learning transferable, generalized, and robust representations from unlabeled data. This paper systematically compares incorporation mechanisms and graph-specific tasks, finding that properly designed self-supervision improves GCN generalizability and robustness. The study also identifies limitations of pretraining and finetuning and self-training, while integrating multi-task self-supervision into adversarial training.
Problem
Self-supervision’s introduction to GCNs operating on graph data was rarely explored, leaving open how to incorporate it, which tasks help, and whether it affects adversarial robustness.
Method
The paper compares three incorporation mechanisms, investigates node clustering, graph partitioning, and graph completion, and integrates multi-task self-supervision into graph adversarial training.
Results
Properly designed task forms and incorporation mechanisms improve GCN generalizable standard performance and robustness against various graph attacks.
Takeaways & Limitations
Self-supervision benefits GCNs when task forms and incorporation mechanisms are appropriately designed, with task suitability depending on dataset pseudo-label quality and prior complementarity.
Takeaways & Limitations
Self-training can show performance-gain saturation as labeling rates increase and restricts the types of self-supervision tasks that can be incorporated.
Abstract
from arXiv · showhide
Self-supervision as an emerging technique has been employed to train convolutional neural networks (CNNs) for more transferrable, generalizable, and robust representation learning of images. Its introduction to graph convolutional networks (GCNs) operating on graph data is however rarely explored. In this study, we report the first systematic exploration and assessment of incorporating self-supervision into GCNs. We first elaborate three mechanisms to incorporate self-supervision into GCNs, analyze the limitations of pretraining & finetuning and self-training, and proceed to focus on multi-task learning. Moreover, we propose to investigate three novel self-supervised learning tasks for GCNs with theoretical rationales and numerical comparisons. Lastly, we further integrate multi-task self-supervision into graph adversarial training. Our results show that, with properly designed task forms and incorporation mechanisms, self-supervision benefits GCNs in gaining more generalizability and robustness. Our codes are available at https://github.com/Shen-Lab/SS-GCNs.
1. Introduction
The paper presents the first systematic study of self-supervision in GCNs, asking how it affects classification, task design, and adversarial robustness. It evaluates incorporation mechanisms and graph-specific pretext tasks, finding that properly designed self-supervision can improve robustness without larger models or additional data.
- Motivation: GCNs extend CNNs to graph-structured data and are evaluated here mainly on transductive semi-supervised node classification.This setting contains many unlabeled nodes and relatively few labeled nodes.
- Motivation: Self-supervision is motivated by its use of unlabeled data to learn transferable and generalized representations through pretraining or multi-task learning.The paper notes that pretext tasks must be designed to support downstream-related features.
- Research questions: The study asks whether self-supervision improves GCN classification performance, which pretext-task designs are useful, and whether it affects adversarial robustness.These questions organize the paper’s systematic assessment.
- Approach: The authors compare pretraining and finetuning, self-training, and multi-task learning as mechanisms for incorporating self-supervision into GCNs.Multi-task learning is presented as a regularization term in GCN training and is compared favorably with the other schemes.
- Approach: They study node clustering, graph partitioning, and graph completion as self-supervised tasks based on graph properties.The paper further reports that different models and datasets may prefer different tasks.
2. Related Work
Prior work established graph-based semi-supervised learning and several self-supervision paradigms in CNNs, but self-supervision in GCNs remained scarcely studied. The paper positions its work against the limited GCN literature and graph attack-defense research.
- Graph-based semi-supervised learning: Graph-based semi-supervised methods commonly assume that nodes joined by stronger edges are more likely to share labels.Earlier approaches include mincuts, Boltzmann machines, and graph random walks.
- Self-supervised learning: CNN self-supervision mainly uses pretraining and finetuning or simultaneous multi-task learning with supervised and self-supervised objectives.These paradigms motivate the paper’s examination of analogous mechanisms for GCNs.
- Self-supervision in GCNs: Before this study, only one recent work had applied self-supervision to GCNs, using node clustering through self-training.That approach was reported to suffer performance saturation and degradation and to restrict the available self-supervised tasks.
- Adversarial attack and defense: Graph attack-defense research includes edge and feature attacks, direct training on perturbed graphs, graph refinement, and adversarial contrasting learning.This literature motivates evaluating whether self-supervision can contribute to GCN robustness.
3. Method
The paper develops three ways to incorporate self-supervision into GCNs, then focuses on multi-task learning with graph-specific auxiliary tasks. It evaluates these mechanisms and tasks, finding multi-task learning most effective among the schemes.
- Three Schemes: The study examines pretraining and finetuning, self-training, and multi-task learning as three mechanisms for incorporating self-supervision into GCNs.The mechanisms are evaluated for transductive semi-supervised node classification.
- Three Schemes: Pretraining and finetuning initializes the feature extractor with a self-supervised task, then trains it on the supervised target task.The feature extractor parameters learned during pretraining initialize supervised training.
- Three Schemes: Self-training iteratively adds highly confident pseudo-labeled unlabeled nodes to the labeled set for subsequent training rounds.The paper discusses M3S as the prior GCN self-supervision approach using this pipeline.
- Multi-task Learning: Multi-task learning shares a feature extractor between target and auxiliary tasks while assigning separate linear transformations and weighting supervised and self-supervised losses.The self-supervised objective acts as a data-driven regularizer during training.
- GCN-Specific Tasks: Multi-task learning is experimentally the most effective of the three schemes and supports a toolkit of self-supervised tasks using node and edge information.The toolkit includes node clustering, graph partitioning, and graph completion; different models and datasets may prefer different tasks.
4. Experiments
Experiments compare three ways to incorporate self-supervision into GCNs and show that multi-task learning most consistently improves generalizable performance, especially with suitable tasks and architectures. Integrating multi-task self-supervision into adversarial training also improves robustness across graph attacks.
- Self-Supervision Helps Generalizability: Graph partitioning generally benefits GCN, GAT, and GIN across all three datasets, while node clustering does not benefit SOTAs on PubMed.Task effectiveness varies with dataset characteristics, pseudo-label quality, and whether the self-supervised priors complement the architecture.
- Self-Supervision Helps Generalizability: All three self-supervised tasks improve most tested GNN architectures on some datasets, but gains are smaller for architectures with stronger existing priors, such as GMNN and GraphMix.GMNN sees little improvement from graph completion, while GraphMix’s feature refinement mitigates graph completion’s additional benefit.
- Self-Supervision Helps Generalizability: Self-supervision produces modest gains in inductive fully supervised node classification and appears more beneficial in semi-supervised or few-shot learning.The experiments evaluate standard performance across GCN architectures and datasets, with node-classification accuracy as the target metric.
- Self-Supervision Helps Adversarial Robustness: Introducing self-supervision into adversarial training improves GCN robustness against varying attack types and perturbation intensities.Node clustering and graph partitioning provide feature and link priors respectively, while graph completion improves adversarial accuracy by around 4.5% against link attacks and over 8.0% against combined link-and-feature attacks on Cora.
- Self-Supervision Helps Generalizability: Multi-task learning consistently benefits GCN standard performance, whereas pretraining and finetuning often loses self-supervised information and self-training is most useful in few-shot settings.Pretraining and finetuning helps Cora but not Citeseer or PubMed; self-training gains can diminish as labeling rates increase.
5. Conclusion
The paper systematically studies self-supervision for GCNs across standard and adversarial settings, comparing incorporation mechanisms and designing self-supervised tasks. It concludes that properly designed tasks and mechanisms can improve both generalizability and robustness.
- 5. Conclusion: The study compares standard and adversarial GCN performance across three incorporation mechanisms and three self-supervised learning tasks.It analyzes optimization effects, evaluates data-driven regularization, and integrates multi-task self-supervision into graph adversarial training.
- 5. Conclusion: Properly designed self-supervised task forms and incorporation mechanisms benefit GCNs in generalizability and robustness.The paper also relates task and mechanism design to data characteristics, target tasks, and neural network architectures.