Source-linked AI summary
Contrastive Self-supervised Learning for Graph Classification
Jiaqi Zeng, Pengtao Xie
TL;DR
Limited labeled graph data makes graph classification prone to overfitting. The paper proposes CSSL-Pretrain and CSSL-Reg, using augmented graphs and contrastive objectives, and reports effectiveness across various graph classification datasets.
Problem
Graph classification often has limited labeled training graphs, making classification models prone to overfitting.
Method
The paper pretrains graph encoders on unlabeled graphs with CSSL or jointly optimizes classification and a CSSL-based regularizer using augmented graphs.
Results
Experiments on various graph classification datasets demonstrate the effectiveness of CSSL-Pretrain and CSSL-Reg, which outperform baseline approaches.
Takeaways & Limitations
CSSL provides two ways to use unlabeled graphs or an auxiliary objective to alleviate overfitting in graph classification.
Abstract
from arXiv · showhide
Graph classification is a widely studied problem and has broad applications. In many real-world problems, the number of labeled graphs available for training classification models is limited, which renders these models prone to overfitting. To address this problem, we propose two approaches based on contrastive self-supervised learning (CSSL) to alleviate overfitting. In the first approach, we use CSSL to pretrain graph encoders on widely-available unlabeled graphs without relying on human-provided labels, then finetune the pretrained encoders on labeled graphs. In the second approach, we develop a regularizer based on CSSL, and solve the supervised classification task and the unsupervised CSSL task simultaneously. To perform CSSL on graphs, given a collection of original graphs, we perform data augmentation to create augmented graphs out of the original graphs. An augmented graph is created by consecutively applying a sequence of graph alteration operations. A contrastive loss is defined to learn graph encoders by judging whether two augmented graphs are from the same original graph. Experiments on various graph classification datasets demonstrate the effectiveness of our proposed methods.
Introduction
The paper addresses overfitting in graph classification when labeled training graphs are limited by proposing two contrastive self-supervised learning approaches. They use unlabeled graphs or a joint CSSL objective, and experiments on various datasets demonstrate effectiveness.
- Introduction: Limited training graphs can make graph classification models overfit and perform less well on test data.The paper motivates this issue with costly data collection in biomedical studies.
- Introduction: The paper proposes CSSL-Pretrain and CSSL-Reg to address overfitting in graph classification.Both approaches are based on contrastive self-supervised learning.
- Introduction: CSSL creates augmented graphs through consecutive edge or node deletion and insertion operations, then distinguishes whether two augmentations share an original graph.This defines the graph-level contrastive task used by the proposed methods.
- Introduction: CSSL-Pretrain learns graph encoders from widely available unlabeled graphs before finetuning them on labeled graphs, while CSSL-Reg jointly optimizes classification and CSSL tasks.The pretraining approach avoids relying on human-provided labels, whereas the regularizer adds CSSL to supervised learning.
- Introduction: The paper reports that CSSL-Pretrain and CSSL-Reg outperform baseline approaches and demonstrate effectiveness in alleviating overfitting.The listed contributions also report effectiveness across various datasets.
Related Works
Related work covers classic graph representation learning, graph neural networks, contrastive self-supervised learning, and emerging self-supervised methods for graph representations.
- Graph Representation Learning: Classic graph representation methods include embedding approaches such as DeepWalk, LINE, and HARP.These methods learn node representations using random walks, edge sampling, or hierarchical representation learning.
- Graph Neural Networks: Graph neural network approaches are categorized into spectral and message-passing methods.GCN and HGP-SL aggregate node-level representations into graph-level representations.
- Contrastive Self-Supervised Learning: Contrastive self-supervised learning has been applied to image encoding, data-efficient recognition, and reinforcement-learning representation learning.Examples include MoCo, Sim-CLR, contrastive predictive coding, and contrastive unsupervised representations.
- Graph Self-Supervised Learning: Graph self-supervised methods learn representations through contextual node prediction, subgraph instance discrimination, or mutual-information objectives.The cited approaches include Peng et al., GCC, and InfoGraph.
Methods
The paper applies contrastive self-supervised learning to graph augmentation and uses it either to pretrain graph encoders or to regularize classification jointly. The contrastive task distinguishes augmented graphs originating from the same original graph.
- Contrastive Self-supervised Learning on Graphs: CSSL creates augmented graphs through consecutive edge and node alteration operations, then predicts whether two augmented graphs share an original graph.The operations include edge deletion, node deletion, edge insertion, and node insertion.
- Contrastive Self-supervised Learning on Graphs: The contrastive objective uses graph embeddings and a prediction multilayer perceptron to distinguish similar from dissimilar augmented-graph pairs.Similarity is measured with cosine similarity and a temperature parameter.
- CSSL-Pretrain: CSSL-Pretrain first trains a graph encoder on unlabeled graphs, discards the contrastive prediction head, and finetunes the encoder for labeled graph classification.The pretrained encoder initializes the classification model.
- CSSL-Reg: CSSL-Reg shares one graph encoder between classification and contrastive tasks, using the CSSL loss as a data-dependent regularizer.Its objective combines classification and CSSL losses with λ as a tradeoff parameter.
- Encoder: The methods are presented as applicable to any graph encoder, although the study uses the HGP-SL encoder.HGP-SL alternates graph convolution and graph pooling layers.
Experiments
Experiments evaluate the proposed methods on five graph classification datasets against graph kernels, GNNs, pooling methods, and self-supervised baselines. CSSL-Pretrain and CSSL-Reg generally improve classification performance and reduce train-test gaps, while CSSL-Reg is sensitive to regularization strength and augmentation design.
- Experimental Setup: Experiments use five graph classification datasets, with 80% training, 10% validation, and 10% testing splits repeated 10 times.The datasets cover protein and chemical-compound graphs with binary labels, and results report averages with standard deviations.
- Baselines: The study compares against graph kernels, graph neural networks, graph pooling methods, self-supervised methods, and CSSL-Freeze.The baselines include GRAPHLET, SP, WL, GCN, GraphSAGE, GAT, multiple pooling methods, InfoGraph, and GCC.
- Classification Results: CSSL-Reg and CSSL-Pretrain outperform baseline approaches for graph classification, demonstrating effectiveness in alleviating overfitting.The methods also generally produce smaller train-test accuracy differences than HGP-SL.
- Self-supervised Comparisons: CSSL methods outperform other self-supervised methods because judging whether augmented graphs share an origin requires capturing holistic graph-level features.The paper contrasts this with baseline self-supervision performed locally at individual nodes.
- Method Comparison: On 4 out of the 5 datasets, CSSL-Reg performs better than CSSL-Pretrain and has a smaller train-test difference.The paper attributes this to jointly learning the classification and CSSL tasks through regularization.
- Sensitivity Analysis: CSSL-Reg accuracy increases as λ rises from zero, then drops when regularization becomes too strong and dominates the classification loss.Figure 6 examines classification accuracy as the regularization parameter changes.
- Ablation Study: Randomly selecting graph alteration operations performs better than deterministic selection in CSSL-Reg.The paper states that random alterations create a more difficult origin-judgment task, which can produce more robust representations.
Conclusions and Future Works
The paper proposes CSSL-Pretrain and CSSL-Reg to alleviate overfitting in graph classification and reports effectiveness across various graph classification datasets. Future work will extend these approaches to other graph learning problems and develop additional graph self-supervised tasks.
- Conclusions: The paper proposes CSSL-Pretrain and CSSL-Reg to use contrastive self-supervised learning for alleviating overfitting in graph classification.CSSL-Pretrain pretrains encoders, whereas CSSL-Reg jointly minimizes classification loss and a CSSL-based regularizer.
- Conclusions: Experiments on various graph classification datasets demonstrate the effectiveness of the proposed methods.
- Future Works: Future work will extend the approaches to graph completion and node classification and develop other graph self-supervised learning methods.One proposed direction is predicting which augmented graph is closer to the original graph.