Source-linked AI summary
An Empirical Study of Graph Contrastive Learning
Yanqiao Zhu, Yichen Xu, Qiang Liu, Shu Wu
TL;DR
GCL has achieved strong results without human annotations, but the reasons for its success and the contributions of individual design choices remain insufficiently evidenced. This paper defines a four-dimensional GCL design space and evaluates it with controlled experiments, finding that sparse topology views and task-aligned contrasting modes are effective while providing PyGCL for standardized experimentation.
Problem
The paper addresses the lack of rigorous empirical evidence about how individual GCL components affect performance.
Method
The authors taxonomy GCL across augmentations, contrasting modes, objectives, and negative mining, then test these choices in controlled experiments across benchmark tasks and datasets.
Results
Experiments suggest that sparse topology augmentations and task-aligned contrasting modes are effective recipes for GCL, while InfoNCE improves consistently but requires many negative samples.
Takeaways & Limitations
PyGCL packages modular GCL components, standardized evaluation, and experiment management to support comparison and development of GCL algorithms.
Takeaways & Limitations
The study covers four design dimensions and node- and graph-level classification and graph-level regression, leaving other model factors and downstream tasks unexamined.
Abstract
from arXiv · showhide
Graph Contrastive Learning (GCL) establishes a new paradigm for learning graph representations without human annotations. Although remarkable progress has been witnessed recently, the success behind GCL is still left somewhat mysterious. In this work, we first identify several critical design considerations within a general GCL paradigm, including augmentation functions, contrasting modes, contrastive objectives, and negative mining techniques. Then, to understand the interplay of different GCL components, we conduct extensive, controlled experiments over a set of benchmark tasks on datasets across various domains. Our empirical studies suggest a set of general receipts for effective GCL, e.g., simple topology augmentations that produce sparse graph views bring promising performance improvements; contrasting modes should be aligned with the granularities of end tasks. In addition, to foster future research and ease the implementation of GCL algorithms, we develop an easy-to-use library PyGCL, featuring modularized CL components, standardized evaluation, and experiment management. We envision this work to provide useful empirical evidence of effective GCL algorithms and offer several insights for future research.
1 Introduction
This paper studies why Graph Contrastive Learning succeeds by evaluating four design dimensions through controlled experiments. It reports practical recipes for augmentations, contrasting modes, objectives, and negative mining, and introduces PyGCL.
- Controlled experiments examine augmentation functions, contrasting modes, contrastive objectives, and negative mining strategies across benchmark tasks and datasets.
- The study addresses the lack of rigorous empirical evidence on how individual GCL components affect performance.
- Sparse topology augmentations provide the strongest benefits, while combining topology and feature augmentation can improve performance further.
- Same-scale contrasting is generally desirable, and contrasting modes should match the granularity of downstream tasks.
- InfoNCE improves performance consistently across settings but requires many negative samples; negative-sample-free objectives may reduce computational burden without compromising performance.
- Embedding-similarity-based negative mining yields limited improvements, while PyGCL provides modular components, standardized evaluation, and experiment management.
2 A General Paradigm of GCL and its Design Dimensions
The paper formulates GCL as a modular pipeline that augments graphs, encodes multiple views, selects contrasting samples, and optimizes a contrastive objective. Its design space covers augmentations, contrasting modes, objectives, and negative mining.
- Problem formulation: A graph encoder maps node features and structure to low-dimensional node embeddings, with a readout producing graph-level representations for graph-oriented tasks.
- A general paradigm of GCL: The framework decomposes GCL into data augmentation, contrasting mode, contrastive objective, and negative mining strategy.
- A general paradigm of GCL: Each training iteration samples two stochastic transformations, encodes both views with a shared GNN, and optionally applies a readout to obtain graph representations.
- Data augmentations: Topology augmentations include edge and node transformations or diffusion, while feature augmentations include masking and dropout.
- Contrasting modes: Contrasting modes operate locally, globally, or across scales, and their applicability depends on downstream task granularity.
- Contrastive objectives: The study compares negative-sample objectives such as InfoNCE, JSD, and triplet loss with negative-sample-free BL, Barlow Twins, and VICReg objectives.
- Negative mining strategies: The evaluated negative mining strategies are HNM, DCL, HBNM, and CNM, although negative mining has received limited attention in existing GCL literature.
3 Empirical Studies
Controlled experiments across unsupervised node- and graph-classification benchmarks examine topology and feature augmentations, stochasticity, contrasting modes, objectives, and negative mining. The findings favor sparse views, task-aligned same-scale contrasting, InfoNCE consistency, and caution with similarity-based negative mining.
- Evaluation configurations: Experiments cover unsupervised node and graph classification across academic-network and molecular datasets, using standardized linear evaluation with ten random splits.Models are pretrained unsupervised, then embeddings are evaluated with a linear classifier; averaged accuracies and standard deviations are reported.
- Data augmentations: Topology augmentations that produce sparser graph views generally improve performance, while excessive edge addition introduces noise and degrades embeddings.Edge removal generally outperforms edge addition; overly high removal can also hurt, indicating that augmentation strength has a useful range.
- Data augmentations: Feature augmentations provide additional benefits, with compositional structure-and-feature augmentation outperforming topology-only settings in general.Feature masking slightly outperforms feature dropout, although their difference is not significant; both remain useful alongside topology transformations.
- Data augmentations: Deterministic augmentations such as PPR and MDK should be paired with stochastic augmentations, because the joint scheme improves vanilla deterministic augmentation.The paper connects this combination to better approximation of the noise distribution used by contrastive objectives.
- Contrasting modes and objectives: Same-scale contrasting generally performs best: local-local for node-level classification and global-global for graph-level tasks.The reported pattern aligns the contrasting mode with the granularity of the downstream task.
- Contrasting modes and objectives: InfoNCE yields consistent improvements among negative-sample-based objectives, while Bootstrapping Latent and Barlow Twins can match counterparts without explicit negatives; similarity-based mining offers limited benefit.InfoNCE requires many negative samples, whereas negative-sample-free objectives reduce computational burden; embedding-similarity mining can confuse semantic positives with hard negatives.
4 Conclusion, Limitations, and Outlook
The paper presents a taxonomy and controlled empirical study of four GCL design dimensions, offers practical guidelines, and releases PyGCL. It acknowledges limited design coverage, downstream-task scope, and theoretical justification, while proposing automated augmentation and deeper analysis of pretext–downstream gaps.
- The paper categorizes GCL by data augmentations, contrasting modes, contrastive objectives, and negative mining strategies, then studies these choices empirically.
- PyGCL is an open-sourced PyTorch library providing modular GCL components, standardized evaluation, and experiment management.
- Limitations: The study covers only four design dimensions, excluding factors such as projection heads and graph encoder choices.
- Limitations: The evaluation includes node- and graph-level classification and graph-level regression, but not tasks such as link prediction or community detection.
- Limitations: The work is empirical and leaves performance guarantees, contrasting-mode mechanisms, and graph-domain hard-negative selection without theoretical justification.
- Outlook: Future directions include automatically learning augmentation functions, understanding pretext–downstream performance gaps, and developing structure-aware negative sampling.
A Reproducibility of Experiments
PyGCL organizes graph contrastive learning into modular components covering augmentations, contrasting architectures and modes, objectives, negative mining, evaluation, and experiment management. Its utilities support composing augmentations, reproducing experiments, and implementing both negative-sample-based and negative-sample-free approaches.
- PyGCL implements graph augmentation, contrasting architectures and modes, contrastive objectives, and negative mining strategies as its four main GCL components.
- Graph augmentations: The toolkit provides a universal augmentation interface and supports composing arbitrary numbers of augmentations or selecting augmentations randomly.
- Contrasting architectures and modes: GCL architectures are grouped into negative-sample-based and negative-sample-free approaches, with single-branch, dual-branch, bootstrap-style, and within-embedding variants.
- Contrasting architectures and modes: PyGCL provides Local-Local, Global-Global, and Global-Local contrasting modes, distinguishing same-scale from cross-scale comparisons.
- Contrastive objectives: Supported objectives include InfoNCE, JSD, Triplet Margin, Bootstrapping Latent, Barlow Twins, and VICReg, with single-positive variants for InfoNCE and Triplet losses.
- Negative mining strategies: Negative mining support includes hard negative mixing, conditional negative sampling, debiased contrastive objectives, and hardness-aware negative sampling.
- Evaluation and reproducibility: PyGCL includes embedding evaluators, dataset-split utilities, training and experiment-management tools, and trial scripts for reproducing experiments.
B Experimental Protocols
The experiments use controlled searches and standardized unsupervised evaluation across node- and graph-level benchmarks. Hyperparameters, training procedures, datasets, and reporting protocols are specified to support comparable results.
- Implementation details: Experiments grid-search embedding dimensions, learning rates, GNN layers, weight decay, InfoNCE temperature, and graph-dataset batch sizes.
- Implementation details: The authors first search the entire design space and then report representative settings to support the empirical observations.
- Evaluation protocols: Models use early stopping with a window size of 50, and evaluation uses ten random splits reporting averaged accuracies and standard deviations.
- Evaluation protocols: The study evaluates unsupervised node classification and graph classification using linear evaluation with frozen embeddings and labeled-data classifiers.
C.1 Large-Scale Evaluation
Large-scale evaluation covers molecular graph classification and regression, using task-specific metrics and controlled comparisons of augmentations, contrasting modes, and objectives. Results favor topology augmentations, global-global contrasting for graph tasks, and InfoNCE under most settings.
- Datasets and tasks: The large-scale evaluation uses ogbg-molhiv for molecular graph classification and PCQM4M-10K for molecular graph regression.
- Datasets and tasks: ogbg-molhiv is evaluated with ROC-AUC under scaffold splitting, while PCQM4M-10K predicts HOMO-LUMO energy gaps using MAE.
- Augmentation schemes: ER and ND topology augmentations produce competitive performance consistently across both large-scale datasets.
- Augmentation schemes: RWS is inferior to the other topology augmentations on these datasets, while feature augmentations alone do not achieve satisfying performance.
- Contrasting modes and objectives: Global-global contrasting yields competitive graph-task performance, and InfoNCE outperforms other objectives under most settings.
C.2 Ablation Studies on Batch Normalization of the Bootstrapping Latent Loss
The ablation examines Batch Normalization (BN) placement in three components of the bootstrapping latent loss for node classification.
- The study compares using BN or omitting it in the GNN encoder, projector, and predictor.The experiments evaluate BN placement across three critical components of the bootstrapping latent loss.
D Discussions on Negative Mining Strategies for GCL
The discussion finds that embedding-similarity-based hard-negative mining can select false negatives in graph data. Similarity increases are associated with more positive samples among nominal negatives, complicating hard-negative selection.
- Embedding-similarity mining may select hard but false negatives because GNN smoothing makes semantically similar nodes appear close.The schemes were originally designed for grid data and measure relative hardness using embedding dot products.
- Selecting hard negatives solely by embedding similarity can produce adverse learning signals for the contrastive objective.Poor-quality node embeddings at the beginning of training further obstruct reliable selection of true hard negatives.
- Figure 7 shows more positive samples among negatives as their semantic similarity to an anchor increases.The histogram uses negatives and semantic similarity scores from a randomly selected Wiki-dataset anchor node.
- Graph contrastive learning addresses label scarcity by learning representations without relying on human annotations.GCL constructs graph views and contrasts positive samples against negatives, making negative selection a central design concern.
F.1 Data Augmentation
The paper organizes graph contrastive learning around topology and feature augmentations, contrasting modes, and contrastive objectives. It describes how graph views are generated and how objectives compare positive and negative representations.
- Data Augmentation: Topology augmentation modifies the adjacency matrix to perturb graph structure.The study considers edge perturbation, node dropping, random-walk subgraphs, and diffusion-based transformations.
- Data Augmentation: Edge perturbation randomly adds or removes edges, while node dropping masks all adjacent edges of selected nodes.Edge perturbation includes Edge Removing, Edge Adding, and Edge Flipping variants; node dropping assigns each node a drop probability.
- Data Augmentation: Diffusion adds global structural information but usually densifies graphs, so the study uses sparse PPR and MDK transformations.Personalized PageRank is followed by hard-threshold sparsification, while Markov Diffusion Kernels are also considered.
- Data Augmentation: Feature augmentation modifies the node-feature matrix through Feature Masking or element-wise Feature Dropout.Feature Masking zeros randomly selected dimensions, whereas Feature Dropout independently masks feature entries.
- Contrasting Modes: Contrasting modes define positive and negative sets at different graph granularities, including local-local and global-local comparisons.Local-local contrasts corresponding node representations across views, while global-local aligns graph-level anchors with node embeddings.
- Contrastive Objectives: Contrastive objectives train encoders to increase agreement between positives and discrepancy between negatives, with InfoNCE, JSD, TM, BL, BT, and VICReg considered.InfoNCE, JSD, and TM use explicit negative comparisons, while BL, BT, and VICReg are discussed among objectives without explicit negative construction.
- Contrastive Objectives: VICReg combines similarity, variance, and covariance terms and is reported as more stable than Barlow Twins and insensitive to normalization tricks.Its coefficients control the importance of the three terms.
F.4 Negative Mining Strategies
Negative mining strategies seek more informative negatives by correcting false-negative sampling or emphasizing hard and semi-hard examples. In GCL, embedding-similarity mining remains constrained by the risk of selecting false negatives.
- Large batches or sampling sizes provide more negatives and more informative contrastive training signals.This follows the presumption that nodes or graphs other than the anchor are dissimilar and therefore negative.
- Negative mining methods include debiasing same-label samples, upweighting hard negatives, mixing hard samples, and selecting semi-hard negatives.DCL, HBNM, HNM, and CNS implement these strategies in different ways.
- HBNM concentrates negative sampling around examples with high anchor similarity using an exponential distribution controlled by β.DCL is a special case of HBNM when β = 0.
- Embedding-similarity mining strategies originally designed for grid data may select hard but false negatives in graph data.The methods measure relative positive/negative hardness using dot products of embeddings, while GNN smoothing can make semantically related nodes similar.