PaperlayerView exact revision on arXiv

Source-linked AI summary

SimGRACE: A Simple Framework for Graph Contrastive Learning without Data Augmentation

Jun Xia, Lirong Wu, Jintao Chen, Bozhen Hu, Stan Z. Li

arXiv:2202.03104v3cs.LGcs.SI

TL;DR

Existing graph contrastive learning methods struggle to preserve graph semantics without costly, dataset-specific augmentation selection. SimGRACE instead contrasts representations from an original graph encoded by a GNN and a perturbed encoder, while AT-SimGRACE applies adversarial encoder perturbation for robustness; the paper reports competitive or better performance with flexibility and efficiency.

  • Problem

    GCL augmentation selection commonly requires manual trial-and-error, cumbersome search, or expensive domain knowledge, limiting efficiency and general applicability.

  • Method

    SimGRACE contrasts two representations of the original graph produced by a GNN encoder and its perturbed version, while AT-SimGRACE perturbs the encoder adversarially.

  • Results

    The methods achieve competitive or better performance than state-of-the-art GCL methods across generalizability, transferability, robustness, and efficiency, including strong semi-supervised results.

  • Takeaways & Limitations

    Encoder perturbation provides a flexible, efficient alternative to semantic-preserving graph augmentations, and adversarial encoder perturbation enhances graph contrastive learning robustness.

  • Takeaways & Limitations

    The theoretical formulation assumes a specific perturbation setup, including an L2 norm ball and Gaussian weight perturbations with variance tied to weight magnitude.

Abstract

from arXiv · show

Graph contrastive learning (GCL) has emerged as a dominant technique for graph representation learning which maximizes the mutual information between paired graph augmentations that share the same semantics. Unfortunately, it is difficult to preserve semantics well during augmentations in view of the diverse nature of graph data. Currently, data augmentations in GCL that are designed to preserve semantics broadly fall into three unsatisfactory ways. First, the augmentations can be manually picked per dataset by trial-and-errors. Second, the augmentations can be selected via cumbersome search. Third, the augmentations can be obtained by introducing expensive domain-specific knowledge as guidance. All of these limit the efficiency and more general applicability of existing GCL methods. To circumvent these crucial issues, we propose a \underline{Sim}ple framework for \underline{GRA}ph \underline{C}ontrastive l\underline{E}arning, \textbf{SimGRACE} for brevity, which does not require data augmentations. Specifically, we take original graph as input and GNN model with its perturbed version as two encoders to obtain two correlated views for contrast. SimGRACE is inspired by the observation that graph data can preserve their semantics well during encoder perturbations while not requiring manual trial-and-errors, cumbersome search or expensive domain knowledge for augmentations selection. Also, we explain why SimGRACE can succeed. Furthermore, we devise adversarial training scheme, dubbed \textbf{AT-SimGRACE}, to enhance the robustness of graph contrastive learning and theoretically explain the reasons. Albeit simple, we show that SimGRACE can yield competitive or better performance compared with state-of-the-art methods in terms of generalizability, transferability and robustness, while enjoying unprecedented degree of flexibility and efficiency.

1 INTRODUCTION

Graph contrastive learning methods often depend on dataset-specific augmentation choices that are difficult to select efficiently while preserving semantics. SimGRACE addresses this challenge by perturbing encoders instead of graph data, and AT-SimGRACE extends the approach for robustness.

  • Motivation: Existing GCL augmentation strategies rely on manual tuning, automated search, or expensive domain knowledge, limiting generality and practicality.GraphCL uses dataset-specific trial-and-error; JOAO automates selection but adds computation and still depends on a configured augmentation pool; MoCL uses costly molecular knowledge.
  • Proposed framework: SimGRACE obtains contrastive views from the original graph processed by an encoder and its perturbed version, avoiding graph data augmentation.The framework uses encoder perturbation to generate correlated views while other graphs in the minibatch provide negative pairs.
  • Robustness: AT-SimGRACE perturbs the encoder adversarially to improve robustness with less computational overhead than per-graph adversarial transformations.The method replaces supervised classification loss with contrastive loss and applies adversarial perturbation to the encoder.
  • Empirical evaluation: The paper reports competitive or better performance across multiple graph datasets and dimensions including generalizability, transferability, robustness, and efficiency.The comparison covers social and biochemical datasets of multiple scales and types.

2 RELATED WORK

Graph self-supervised learning uses designed pretext tasks, while graph contrastive learning includes local–global representation comparisons and augmentation-based approaches. Existing augmentation selection can be time-consuming, inconvenient, expensive, or narrowly specialized, motivating SimGRACE’s encoder-perturbation framework.

  • SimGRACE feeds the original graph into an encoder and its perturbed version, then maximizes agreement between their projected representations with a contrastive loss.
  • Graph self-supervised learning learns representations without supervision through designed pretext tasks, including neighborhood-context, node-attribute, generative, and graph-property prediction.
  • Graph contrastive learning includes methods that maximize mutual information between graph-level and substructure-level representations of different granularity.
  • JOAO automatically selects GraphCL augmentations through unified bi-level optimization, but this process is time-consuming and inconvenient.
  • MoCL incorporates domain knowledge into molecular graph augmentations to preserve semantics, but the knowledge is expensive and the method is limited to molecular graphs.

3 METHOD

SimGRACE forms two graph views by perturbing a GNN encoder rather than augmenting graph data, then learns agreement between their projected representations with contrastive loss. AT-SimGRACE extends this framework with adversarial encoder perturbations and links robustness to flattening the loss landscape.

  • 3.1 SimGRACE: SimGRACE uses an encoder and its perturbed version to extract two graph-level representations from the same original graph.The perturbation adds Gaussian noise to each layer's weights, scaled by coefficient η.
  • 3.1 SimGRACE: A two-layer MLP projection head maps the representations to latent vectors, and NT-Xent enforces agreement between positive and negative pairs.Positive pairs come from the same graph's two views; negatives are drawn from other perturbed representations in the minibatch.
  • 3.1 SimGRACE: The contrastive objective uses 2N representations from a minibatch of N graphs and treats the other N−1 perturbed representations as negatives for each graph.Similarity is measured with cosine similarity, and the final loss is computed across all positive pairs in the minibatch.
  • 3.2 Why can SimGRACE work well?: Alignment measures positive-pair proximity, while uniformity measures how embeddings of random samples scatter on the hypersphere.These metrics are used to analyze why contrastive learning produces useful representations.
  • 3.2 Why can SimGRACE work well?: On MUTAG, SimGRACE improves alignment and uniformity; it achieves better alignment than GraphCL and outperforms MoCL overall through stronger uniformity.MoCL achieves better alignment than SimGRACE but only a small uniformity gain, whereas GraphCL has a smaller alignment gain.
  • 3.3 AT-SimGRACE: AT-SimGRACE replaces supervised loss with contrastive loss and adversarially perturbs the encoder instead of each graph to improve computational efficiency.Its optimization uses an L2 norm ball and gradient ascent for inner perturbation updates followed by minibatched SGD updates of the encoder weights.
  • 3.4 Theoretical Justification: The theoretical analysis explains robustness by showing that AT-SimGRACE optimizes the worst-case sharpness of the loss landscape.The method bounds max_Δ[L(θ + Δ)] − L(θ), connecting adversarial training to flatter loss behavior.

4 EXPERIMENTS

The experiments evaluate SimGRACE and AT-SimGRACE across generalizability, transferability, robustness, efficiency, and hyperparameter sensitivity using graph benchmarks and transfer datasets.

  • Generalizability: RQ1 tests whether SimGRACE outperforms competitors in unsupervised and semi-supervised settings.Unsupervised evaluation uses SVM classification with 10-fold cross-validation; semi-supervised evaluation uses pre-training followed by fine-tuning.
  • Robustness: RQ3 tests whether AT-SimGRACE improves performance against various adversarial attacks.The robustness experiments compare AT-SimGRACE with existing competitors under adversarial attack settings.
  • Efficiency: RQ4 measures SimGRACE efficiency in time and memory relative to competitors.The experiment explicitly asks whether SimGRACE is more efficient than competing methods.
  • Hyperparameters Sensitivity: RQ5 examines sensitivity to perturbation magnitude η, training epochs, and batch size.These hyperparameters are evaluated as potential sources of sensitivity in SimGRACE.

4.2 Unsupervised and semi-supervised learning (RQ1)

SimGRACE performs strongly across unsupervised, semi-supervised, transfer-learning, and adversarial-robustness evaluations, with especially favorable results on molecular and protein-related tasks.

  • Unsupervised learning: SimGRACE outperforms other baselines and ranks in the top three on all unsupervised-learning datasets.It generally performs better on biochemical molecules than augmentation-based methods because molecular semantics are more fragile.
  • Semi-supervised learning: With 1% labels, SimGRACE outperforms previous baselines by a large margin or matches state-of-the-art performance.With 10% labels, it performs comparably to GraphCL and JOAO(v2).
  • Transferability: SimGRACE shows competitive or better transferability than other pre-training schemes, especially on the PPI dataset.The paper notes that no pre-training scheme is universally beneficial, particularly in out-of-distribution transfer learning.
  • Adversarial robustness: AT-SimGRACE boosts GNN robustness dramatically compared with training from scratch under three adversarial attacks.The evaluation uses synthetic graphs and compares performance across RandSampling, GradArgmax, and RL-S2V attacks.

4.5 Efficiency (Training time and memory cost) (RQ4)

SimGRACE is substantially more efficient than GraphCL and JOAOv2, while its performance also depends on perturbation magnitude, batch size, and training duration.

  • Training time and memory cost: 40–90 times faster than JOAOv2 and 2.5–4 times faster than GraphCL in training time.These comparisons exclude GraphCL’s manual augmentation trial-and-error time.
  • Training time and memory cost: SimGRACE requires less computational memory than GraphCL and JOAOv2, especially on large-scale social graphs such as COLLAB and RDT-B.
  • Magnitude of the perturbation: Performance is usually lowest at η=0, while larger perturbations improve performance within an appropriate range.Without perturbation, the method compares original samples as negatives and the positive-pair loss becomes zero.
  • Magnitude of the perturbation: Over-large perturbations degrade performance because graph semantics are not preserved.
  • Batch size and epochs: Larger batch sizes or more training epochs generally improve SimGRACE performance by providing more negative samples.The reported fair-comparison setting uses batch size 128 and 20 epochs.

5 CONCLUSIONS

The paper concludes that SimGRACE provides a flexible and efficient augmentation-free framework for graph contrastive learning. It also introduces adversarial training schemes to improve robustness and identifies future extensions beyond graph data.

  • Conclusions: SimGRACE matches or outperforms state-of-the-art methods across graph datasets with varied scales and types.
  • Conclusions: The framework removes tedious manual tuning, cumbersome search, and expensive domain knowledge while offering flexibility, efficiency, and ease of use.
  • Conclusions: AT-SimGRACE uses adversarial training schemes to enhance robustness and provides a principled theoretical explanation.
  • Future work: Future work includes testing encoder perturbation in computer vision and natural language processing and applying pretrained GNNs.

A.1 Unsupervised learning & Semi-supervised learning

The appendix reports dataset statistics and experimental settings for unsupervised and semi-supervised learning. Semi-supervised experiments use 1% and 10% label rates, with repeated 10-fold evaluations.

  • Unsupervised learning & Semi-supervised learning: Table 7 provides dataset statistics for the unsupervised and semi-supervised experiments.
  • Unsupervised learning & Semi-supervised learning: Semi-supervised experiments use 1% and 10% label rates when there are over 10 samples for each class.
  • Unsupervised learning & Semi-supervised learning: Unsupervised and semi-supervised results are obtained from five runs, each corresponding to a 10-fold evaluation, with mean and standard deviation reported.

A.2 Transfer learning

The appendix documents the datasets and split roles used for transfer learning. ZINC-2M and PPI-306K serve as pretraining datasets, while the remaining listed datasets are used for fine-tuning.

  • Transfer learning: Table 8 reports dataset statistics for transfer learning.
  • Transfer learning: ZINC-2M and PPI-306K are used for pretraining, while the remaining listed datasets are used for fine-tuning.

B GNN ARCHITECTURES IN VARIOUS SETTINGS

The experiments use different GNN encoder architectures and configurations across unsupervised, semi-supervised, transfer-learning, and adversarial-robustness settings, matching previous competitors where applicable.

  • Unsupervised experiments use a three-layer GIN encoder with 32 hidden dimensions.
  • Semi-supervised experiments use a five-layer ResGCN encoder with 128 hidden dimensions.
  • Transfer-learning experiments use GIN with the default setting from [15].
  • Adversarial-robustness experiments use Structure2vec as the GNN-based encoder, following [7].
Loading 2202.03104v3…