AutoGCL: Automated Graph Contrastive Learning via Learnable View Generators

Yihang Yin; Qingzhong Wang; Siyu Huang; Haoyi Xiong; Xiang Zhang

Source-linked AI summary

AutoGCL: Automated Graph Contrastive Learning via Learnable View Generators

Yihang Yin, Qingzhong Wang, Siyu Huang, Haoyi Xiong, Xiang Zhang

arXiv:2109.10259v2Sep 21, 2021cs.LG

TL;DR

Graph contrastive learning needs views that preserve graph semantics while providing useful variation, but pre-defined augmentations may not adapt to inputs. AutoGCL learns input-conditioned node-level view generators and jointly trains them with the encoder and classifier. Across semi-supervised, unsupervised, and transfer-learning settings, it outperforms counterparts on most datasets and tasks, while visualizations show preserved discriminative structures.

Problem
Pre-defined graph augmentations may change semantic labels and do not adapt well to input graphs, while scalable graph generative models remain difficult to use.
Method
AutoGCL learns input-conditioned probability distributions over node-level augmentations and jointly trains view generators, graph encoders, and classifiers end-to-end.
Results
AutoGCL outperforms counterparts on most datasets and tasks across semi-supervised, unsupervised, and transfer-learning settings, while visualizations show preserved discriminative structures.
Takeaways & Limitations
Learnable view generators can produce contrastive graph views with similar semantic information and different topological properties.
Takeaways & Limitations
Excessive minimization of the contrastive loss can promote fine-tuning overfitting and make decision-boundary samples harder to separate.

Abstract

from arXiv · show

Contrastive learning has been widely applied to graph representation learning, where the view generators play a vital role in generating effective contrastive samples. Most of the existing contrastive learning methods employ pre-defined view generation methods, e.g., node drop or edge perturbation, which usually cannot adapt to input data or preserve the original semantic structures well. To address this issue, we propose a novel framework named Automated Graph Contrastive Learning (AutoGCL) in this paper. Specifically, AutoGCL employs a set of learnable graph view generators orchestrated by an auto augmentation strategy, where every graph view generator learns a probability distribution of graphs conditioned by the input. While the graph view generators in AutoGCL preserve the most representative structures of the original graph in generation of every contrastive sample, the auto augmentation learns policies to introduce adequate augmentation variances in the whole contrastive learning procedure. Furthermore, AutoGCL adopts a joint training strategy to train the learnable view generators, the graph encoder, and the classifier in an end-to-end manner, resulting in topological heterogeneity yet semantic similarity in the generation of contrastive samples. Extensive experiments on semi-supervised learning, unsupervised learning, and transfer learning demonstrate the superiority of our AutoGCL framework over the state-of-the-arts in graph contrastive learning. In addition, the visualization results further confirm that the learnable view generators can deliver more compact and semantically meaningful contrastive samples compared against the existing view generation methods.

1 Introduction

Graph contrastive learning can reduce reliance on graph labels, but conventional augmentations may disrupt graph semantics. AutoGCL addresses this with learnable, adaptive view generation and joint end-to-end training, improving performance across multiple learning settings.

Graph labeling requires substantial effort from professional annotators, motivating unsupervised and self-supervised GNN pretraining.
Graph augmentations can severely disrupt semantics and properties, making image-style transformations less suitable for graph contrastive learning.
AutoGCL learns input-conditioned node-level augmentation distributions and uses differentiable sampling to generate adaptive contrastive views.
AutoGCL improves state-of-the-art graph contrastive learning performance on most datasets across semi-supervised, unsupervised, and transfer-learning tasks.
Visualization results indicate that AutoGCL better preserves input semantic structures than pre-defined view generators.
Its joint strategy trains view generators, graph encoders, and classifiers end-to-end using view similarity, contrastive, and classification losses.

2 Related Work

Graph contrastive learning relies on views whose augmentation policies strongly affect performance, yet learnable graph augmentation remains underexplored. Prior methods vary in their view construction, while AutoGCL learns node-level augmentation distributions to preserve semantic structures and remain differentiable.

GNNs produce node embeddings by recursively aggregating neighbor information, with k layers covering each node’s k-hop neighborhood.
Graph-level representations for classification are obtained by applying a READOUT function and MLP layers to node embeddings.
Existing graph contrastive methods use diverse views, including diffusion, edge dropping, node dropping, attribute masking, perturbation, and subgraph sampling.
Augmentation policies are task-dependent, and their choice can substantially affect contrastive-learning performance.
InfoMin learns view generation with a flow-based model, whereas JOAO learns distributions over fixed augmentations and AD-GCL learns edge dropping.
AutoGCL learns probability distributions over node-level augmentations and reports semantic-structure preservation, end-to-end differentiability, and efficient training.

3 Methodology

AutoGCL learns node-wise graph augmentations conditioned on input structure, then jointly trains view generators, encoder, and classifier to produce diverse but label-preserving contrastive views.

3.2 Learnable Graph View Generator: The learnable generator supports simultaneous node dropping and attribute masking while avoiding differentiable edge generation with O(N^2) adjacency prediction.Edges connected to dropped nodes are removed, while the edge table serves only as guidance for aggregation and does not receive gradients.
3.2 Learnable Graph View Generator: AutoGCL learns node-wise probabilities over dropping, keeping, or masking each node’s attributes instead of using fixed augmentation ratios.The generator uses GIN embeddings and gumbel-softmax sampling; dropped nodes also have their incident edges removed.
3.3 Contrastive Pre-training Strategy: Two separate view generators create positive view pairs whose augmentation choices are encouraged to differ through a similarity loss.The similarity loss minimizes mutual information between the generators’ sampled augmentation-choice matrices.
3.3 Contrastive Pre-training Strategy: AutoGCL defines contrastive, similarity, and classification losses, with classification loss encouraging label-preserving augmentations in semi-supervised pre-training.The contrastive loss treats two augmented views from the same graph as a positive pair, while classification uses the original and augmented graphs where applicable.
3.3 Contrastive Pre-training Strategy: The framework combines two view generators, a graph encoder, and a classifier, while naive training alternates augmentation generation, view sampling, and parameter updates.The joint strategy was reported to generate label-preserving augmentation and outperform the naive strategy.
3.3 Contrastive Pre-training Strategy: Semi-supervised training alternates contrastive and supervised objectives, while unsupervised and transfer learning use joint minimization of contrastive loss.The naive strategy also includes original data during contrastive training to encourage preservation of label-related information.

4 Experiment

AutoGCL is evaluated across semi-supervised, unsupervised, and transfer-learning graph classification settings, with comparisons against established methods and training strategies. Results generally favor AutoGCL, while visualizations and strategy analyses examine semantic preservation and joint training.

Unsupervised Learning: AutoGCL achieves the best unsupervised-learning results on PROTEINS, NCI1, IMDB-binary, and REDDIT-Multi-5K, and second-best results on MUTAG, DD, and REDDIT-binary.It outperforms GraphCL, JOAO, and AD-GCL on these comparisons.
Transfer Learning: On transfer-learning benchmarks, AutoGCL performs best on most datasets, including BBBP, ClinTox, MUV, and BACE.Its average gain across datasets is around 1.5%.
Transfer Learning: 83.26±1.13 ROC-AUC is reported for AutoGCL on BACE, compared with 78.51±0.80 for AD-GCL.AttrMasking slightly outperforms AutoGCL on Tox21 and ToxCast.
Semi-Supervised Learning: The joint-strategy is compared with augmentation-only and naive contrastive training, achieving relatively high accuracy on most semi-supervised datasets and the best performance on PROTEINS and COLLAB.On other datasets, it can achieve the second-best performance.
View Visualization: On MNIST-Superpixel, AutoGCL’s generated views are more likely to retain key nodes, preserving semantic features while providing variance for contrastive learning.The view generators and classifier were jointly trained until test accuracy on generated views reached 90%.
Joint Training Strategy: The joint-strategy jointly trains the view generators and classifier for 30 epochs, whereas the naive-strategy uses contrastive pre-training followed by fine-tuning.The COLLAB comparison reports that joint training considerably alleviates over-fitting and learns better representations faster.

5 Conclusion

The paper presents AutoGCL as a learnable graph augmentation approach that jointly trains view generators, encoders, and classifiers. Across multiple learning settings, it reports advantages over competing methods and visual evidence of preserved discriminative structures.

Method: AutoGCL uses GIN-based learnable view generators and alternately optimizes them with graph encoders and classifiers to preserve graph semantic labels.The joint learning strategy is designed to generate different views while retaining discriminative structures.
Results: Across semi-supervised, unsupervised, and transfer-learning tasks, AutoGCL outperforms counterpart methods on most datasets and tasks.The conclusion summarizes results across several datasets and evaluation settings.
Conclusion: Generated graph views preserve discriminative input structures, while t-SNE visualization indicates that joint training is a better choice for semi-supervised graph representation learning.These conclusions are based on the paper’s visualization and classification experiments.

A.1 An Insight into GraphCL Augmentations

The appendix examines how GraphCL’s fixed augmentation choices affect graph contrastive learning. It finds that augmentation type and intensity materially influence performance, motivating learnable augmentation policies.

GraphCL Augmentations: GraphCL includes node dropping, edge perturbation, sub-graph selection, and node attribute masking as graph augmentations.Each operation modifies graph topology or node attributes in a different way.
Existing Augmentation Policies: Existing graph contrastive learning methods use uniform augmentation-policy sampling and a fixed hyper-parameter called “aug ratio.”The ratio controls the portion of nodes or edges selected for augmentation and is commonly held constant across experiments.
Ablation Findings: Ablation results show that edge perturbation and subgraph augmentation provide limited or negative contributions to graph contrastive learning.The appendix also notes that subgraph augmentation is contained within the potential view space of node dropping.
Ablation Findings: The choice of “aug ratio” has a considerable effect on final performance, making a constant ratio inappropriate across settings.This observation supports learning augmentation policies rather than relying only on fixed augmentation intensity.

A.2 The Effectiveness of Our Learnable Graph Augmentations

The appendix evaluates learnable graph augmentations through ablations and MNIST-Superpixel visualizations. AutoGCL’s generator is reported to retain key semantic nodes while producing sufficiently varied contrastive views.

MNIST-Superpixel Setup: MNIST-Superpixel contains 60,000 training samples and 10,000 testing samples, with 75 nodes per graph whose attributes represent super-pixel intensity.The dataset provides graph views with visually interpretable node attributes.
Visualization Setup: The learnable view generator and classifier were jointly trained until test accuracy on generated views reached 90%.The experiment used node dropping as the only topological augmentation.
View Comparison: Compared with GraphCL’s node dropping at aug ratio = 2, AutoGCL is more likely to retain key nodes from the original graph.The comparison reports semantic-feature preservation together with enough variance for contrastive learning.
View Visualization: Figure 4 colors non-zero-attribute nodes red and other nodes blue, with greater redness indicating larger node-attribute values.This color encoding supports visual comparison of semantic node retention across generated views.
Training-Strategy Analysis: The joint-strategy is compared with the naive-strategy using accuracy and t-SNE visualizations of training stages.The reported comparison concerns representation learning and classification behavior during the two strategies.

A.3 Analysis for Joint Training Strategy

The joint strategy produces better graph representations and mitigates over-fitting compared with naive pretraining and fine-tuning. Its learned views become diverse while preserving semantic similarity during joint training.

Joint versus naive strategy: On COLLAB, the joint strategy achieves higher test accuracy than the naive strategy, whose accuracy declines and indicates over-fitting.Both strategies use 5-layer GIN backbones; the comparison trains on 5000 graphs across 3 classes.
Representation learning: Joint training makes same-label graphs cluster while improving the decision boundary for classification with label supervision.The naive contrastive process gradually clusters same-label graphs, but its decision boundary remains difficult to recognize before fine-tuning.
Representation learning: During joint training, similarity and contrastive losses both decrease, indicating more diverse views whose representations remain close enough.This pattern is associated with view generators learning to preserve the input graph's semantic label.
Joint versus naive strategy: Figure 7 compares loss behavior between the naive-strategy and joint-strategy.The figure provides a direct loss comparison for the two training strategies.