PaperlayerView exact revision on arXiv

Source-linked AI summary

Adversarial Graph Augmentation to Improve Graph Contrastive Learning

Susheel Suresh, Pan Li, Cong Hao, Jennifer Neville

arXiv:2106.05819v4cs.LGcs.AI

TL;DR

Label scarcity motivates self-supervised graph representation learning, but traditional GCL can encode redundant information that harms downstream robustness. AD-GCL adversarially optimizes graph augmentations to reduce redundancy, with learnable edge dropping as a practical instantiation, and achieves gains across unsupervised, transfer, and semi-supervised settings.

  • Problem

    Traditional GCL may capture redundant information irrelevant to downstream tasks, producing brittle representations, while label scarcity limits supervised GNN training.

  • Method

    AD-GCL pairs GCL with adversarial training, using a GNN augmenter to optimize graph augmentations that decrease redundant information; its practical instantiation uses learnable edge dropping.

  • Results

    AD-GCL achieves significant performance gains and high mean ranks over state-of-the-art baselines across unsupervised, transfer, and semi-supervised learning settings.

  • Takeaways & Limitations

    AD-GCL provides a theoretically motivated way to improve GCL by optimizing graph augmentations adversarially toward minimal sufficient information for graph identification.

  • Takeaways & Limitations

    Evaluation results depend on the downstream classifier protocol; nonlinear SVMs can improve even randomly initialized untrained GIN performance relative to a linear classifier.

Abstract

from arXiv · show

Self-supervised learning of graph neural networks (GNN) is in great need because of the widespread label scarcity issue in real-world graph/network data. Graph contrastive learning (GCL), by training GNNs to maximize the correspondence between the representations of the same graph in its different augmented forms, may yield robust and transferable GNNs even without using labels. However, GNNs trained by traditional GCL often risk capturing redundant graph features and thus may be brittle and provide sub-par performance in downstream tasks. Here, we propose a novel principle, termed adversarial-GCL (AD-GCL), which enables GNNs to avoid capturing redundant information during the training by optimizing adversarial graph augmentation strategies used in GCL. We pair AD-GCL with theoretical explanations and design a practical instantiation based on trainable edge-dropping graph augmentation. We experimentally validate AD-GCL by comparing with the state-of-the-art GCL methods and achieve performance gains of up-to $14\%$ in unsupervised, $6\%$ in transfer, and $3\%$ in semi-supervised learning settings overall with 18 different benchmark datasets for the tasks of molecule property regression and classification, and social network classification.

1 Introduction

Graph contrastive learning addresses label scarcity by learning from augmented graph views, but conventional InfoMax can retain redundant information. AD-GCL adversarially optimizes graph augmentation to reduce redundancy and improves results across multiple learning settings and benchmark tasks.

  • Motivation: Label scarcity makes self-supervised GNN learning important for graph data, including applications such as identifying drug-molecule pharmacological effects.Graph labels can require substantial time and resources to obtain.
  • Motivation: Traditional GCL maximizes correspondence between representations of the same graph in different augmented forms.This follows the mutual-information maximization principle.
  • Motivation: InfoMax may encode redundant information irrelevant to downstream tasks, producing brittle representations and degrading downstream performance.Information bottleneck methods instead seek minimal sufficient task-relevant information, but require downstream-task knowledge.
  • AD-GCL: AD-GCL pairs GCL with adversarial training so an augmenter removes redundant information while the encoder maximizes correspondence with augmented graphs.The method is designed for graph-level tasks and is supported by theoretical explanations.

2 Notations and Preliminaries

The paper represents attributed graphs with message-passing GNNs that iteratively aggregate neighborhood information and pool node states into graph vectors. These encoders have limited expressive power and cannot distinguish every graph.

  • Graph and GNN preliminaries: An attributed graph consists of node and edge sets, optionally carrying F-dimensional node and edge attributes.The neighborhood of a node v is denoted N_v.
  • Graph representation learning: Graph representation learning maps graphs to low-dimensional vectors that can be used by a downstream model to predict graph labels.The formulation assumes graphs and labels are independently sampled from an underlying joint distribution.
  • Graph and GNN preliminaries: Message-passing GNNs initialize node representations from node attributes and update them iteratively using neighborhood information.Aggregation and update functions are trainable.
  • Graph and GNN preliminaries: After K message-passing iterations, the graph representation is obtained by pooling the final node representations.The pooling operation maps node-level states to a graph-level vector.
  • Graph contrastive learning: GCL applies InfoMax so graph representations preserve information that distinguishes graphs from one another.The objective is expressed through mutual information between random variables.
  • Expressivity limitation: Message-passing GNN encoders are not injective because graphs indistinguishable by the 1-WL test receive identical representations.This limitation creates a specific challenge for graph contrastive learning.

3 Adversarial Graph Contrastive Learning

AD-GCL combines InfoMax encoding with adversarially optimized graph augmentation to reduce redundant information without downstream labels. Its theory bounds task-irrelevant information and preserves label-relevant information, while a learnable edge-dropping instantiation operationalizes the principle.

  • 3.1 Theoretical Motivation and Formulation of AD-GCL: Traditional InfoMax can capture redundant graph information that identifies graphs but is irrelevant to downstream tasks, producing brittle representations.The paper contrasts this with information bottleneck learning, which seeks minimal sufficient task information but requires downstream labels.
  • 3.1 Theoretical Motivation and Formulation of AD-GCL: Unlike predefined augmentations selected from domain knowledge or downstream validation, AD-GCL automatically searches a parameterized augmentation family.The original graph serves as the anchor while its perturbation is pushed away within the augmentation family.
  • 3.1 Theoretical Motivation and Formulation of AD-GCL: Theorem 1 bounds task-irrelevant information in AD-GCL representations by a quantity involving the best augmentation family and the augmentation's label information.This upper bound is related to the graph information bottleneck objective, although GIB can be better because it accesses downstream labels.
  • 3.1 Theoretical Motivation and Formulation of AD-GCL: Theorem 1 also lower-bounds mutual information between learned representations and downstream labels when the augmentation family is well controlled.The edge-dropping instantiation regularizes the ratio of dropped edges per graph to control this family.
  • 3.1 Theoretical Motivation and Formulation of AD-GCL: AD-GCL pairs an InfoMax encoder with a GNN-based augmenter that adversarially optimizes graph perturbations.The encoder maximizes correspondence between original and augmented graphs, while the augmenter seeks perturbations that minimize that mutual information.
  • 3.2 Instantiation of AD-GCL via Learnable Edge Perturbation: The practical instantiation learns edge-dropping augmentations, while the principle can also pair with node dropping, edge adding, or feature masking.A GNN augmenter parameterizes edge-dropping probabilities, and continuous relaxation enables end-to-end training.

4 Related Work

Related work positions GCL as contrastive learning for GNNs, emphasizing that graph irregularity makes augmentation design more difficult than image-view construction. AD-GCL is distinguished from InfoMin because InfoMin theoretically requires downstream-task knowledge.

  • 4 Related Work: GCL transfers contrastive learning from CNN-based image representation learning to graph neural networks.Unlike images with multiple natural views, graphs require contrastive pairs that manipulate irregular structures without discarding crucial information.
  • 4 Related Work: GNN-based graph representation learning differs fundamentally from CNN-based image representation learning because graph structures and their contrastive views are more abstract and irregular.This difference motivates specialized GCL augmentation strategies rather than directly importing image-view construction.
  • 4 Related Work: Graph contrastive pairs have been built from nodes, whole graphs, subgraphs, and other graph components.These approaches reflect the challenge of constructing useful views for irregular graph structures.
  • 4 Related Work: InfoMin shares ideas with AD-GCL but requires downstream-task supervision to identify an augmentation preserving label information while removing graph information conditional on labels.The paper argues these requirements are unavailable without downstream-task knowledge.

5 Experiments and Analysis

The experiments evaluate AD-GCL with trainable or fixed edge-dropping augmentation across unsupervised, transfer, and semi-supervised graph-learning settings. AD-GCL generally outperforms supervised-learning baselines, non-adversarial augmentation, and established GCL methods, while remaining effective across regularization choices.

  • Experimental setup: AD-GCL is evaluated on graph-level classification and regression using OGB, TU, and ZINC datasets, with GIN encoders and multiple self-supervised baselines.The study compares AD-GCL-FIX and validation-tuned AD-GCL-OPT, alongside RU-GIN, InfoGraph, GraphCL, NAD-GCL, and fully supervised GIN.
  • Unsupervised learning: AD-GCL produces large performance gains over baselines across unsupervised biochemical, social-network, and chemical-molecule tasks.GraphCL selects augmentations through extensive evaluation, whereas AD-GCL learns input-graph-dependent augmentation using edge dropping.
  • Unsupervised learning: AD-GCL improves over non-adversarial edge dropping across all evaluated datasets, while AD-GCL-FIX and AD-GCL-OPT achieve comparable results.The authors attribute the benefit to input-graph-dependent augmentation and report that weak downstream-task information can further improve search-space control.
  • Regularization analysis: AD-GCL remains robust across regularization strengths: λreg from 0.1 to 10.0 corresponds to dropping about 80% of edges to less than 10%.For classification, performance is reported as extremely robust despite substantially different convergent edge-drop ratios; AD-GCL still outperforms naive GCL at extreme ratios.
  • Transfer learning: AD-GCL-FIX significantly outperforms baselines on 3 of 9 transfer datasets and achieves a mean rank of 2.4, best among the compared methods.Its performance is sometimes close to methods using domain knowledge or extensive downstream evaluation, despite using only edge-dropping augmentation.
  • Semi-supervised learning: AD-GCL-FIX significantly outperforms baselines on 3 of 6 semi-supervised datasets and achieves a mean rank of 1.5 with 10% labels.This evaluation uses GCN encoders and compares against models trained from scratch, predictive pre-training, and established GCL methods.

6 Conclusions

AD-GCL extends InfoMax by adversarially optimizing graph augmentations to encourage minimal sufficient graph representations. Its learnable edge-dropping instantiation was evaluated across unsupervised, transfer, and semi-supervised graph property prediction settings.

  • AD-GCL goes beyond conventional InfoMax by optimizing graph augmentations adversarially.The principle targets minimal sufficient information for identifying graphs rather than redundant information.
  • The practical instantiation uses learnable edge dropping as its augmentation strategy.
  • AD-GCL was demonstrated on real-world graph property prediction datasets in unsupervised, transfer, and semi-supervised settings.

B Proof of Theorem 1

The proof analyzes AD-GCL under countable graph spaces and an encoder at least as powerful as the 1-WL test. It establishes upper and lower mutual-information bounds linking optimal augmentation, graph representations, and task labels.

  • Theorem assumptions: Theorem 2 assumes a GNN encoder as powerful as the 1-WL test and countable graph spaces.Under these assumptions, the optimal AD-GCL solution satisfies two stated information bounds.
  • Upper bound: The upper bound constrains I(f*(t*(G)); G | Y) by an augmentation-dependent mutual-information difference.
  • Lower bound: The lower bound states I(f*(G); Y) ≥ I(f*(t′*(G′)); Y) = I(t′*(G′); Y).
  • Proof strategy: The proof uses injectivity of the optimal encoder on relevant augmented graphs and the data processing inequality.
  • 1-WL background: The 1-WL test efficiently generates canonical graph forms but cannot distinguish some non-isomorphic regular graphs with matching sizes and degrees.
  • 1-WL background: GNNs are at most as powerful as 1-WL, while suitable designs may attain 1-WL power, motivating the theorem’s encoder assumption.

D The Training Algorithm for the Instantiation of AD-GCL

The training algorithm implements AD-GCL with a GNN encoder, a GNN-based augmenter, and a projection head under learnable edge-dropping augmentation.

  • The augmenter applies a GNN followed by an MLP to produce Bernoulli edge-dropping weights.

E Summary of Datasets

The evaluation spans molecular, biochemical, social-network, biological-interaction, and chemical-molecule datasets across unsupervised, semi-supervised, and transfer settings. Regression results suggest that edge dropping may be less suitable when node-level information is especially important.

  • Dataset coverage: The experiments use diverse datasets from multiple domains for graph property prediction tasks.
  • Unsupervised learning: Molecular datasets from OGB and ZINC-10K support unsupervised classification and regression experiments.
  • Unsupervised and semi-supervised learning: Biochemical and social-network datasets from the TU Benchmark support unsupervised and semi-supervised graph classification.
  • Transfer learning: Biological-interaction and chemical-molecule datasets support transfer-learning graph classification.
  • Hyperparameter selection: AD-GCL’s main hyperparameter is the edge-dropping regularization strength λreg, selected using validation performance.
  • Regression-task behavior: Regression tasks perform better with small λreg and larger expected edge-drop ratios, but AD-GCL remains uniformly better than random edge dropping.
  • Regression-task behavior: The authors identify alternative learnable augmentation families as future work because edge dropping may not best fit regression tasks sensitive to node-level information.

F.3 Effects of regularization on edge-drop ratio as complete results in Figure 3 setting.

AD-GCL uses λreg to control the expected edge-drop ratio, with stronger regularization producing fewer dropped edges. Validation behavior is evaluated across graph classification and regression settings against non-adversarial random edge dropping.

  • Regularization and edge-drop ratio: λreg and the expected edge-drop ratio are inversely related across multiple datasets.This relationship provides a way to control the augmentation space for learnable edge dropping.
  • Regularization and edge-drop ratio: Figure 4 measures expected edge-drop ratio at the saddle point of Eq.8 as λreg varies.
  • Regression validation: Figure 5 compares AD-GCL with non-adversarial random edge dropping on graph-regression validation performance across edge-drop ratios.Lower validation metric is better, and markers on AD-GCL curves indicate the λreg values used.
  • Classification validation: Figure 6 compares AD-GCL with non-adversarial random edge dropping on graph-classification validation performance across edge-drop ratios.Higher validation metric is better.
  • Experimental selection: The selected λreg values are reported as validation-set optima for the test results in Tables 1.

G.1 Motivating Experiment (Figure 2)

The motivating experiment tests whether mutual-information maximization alone guarantees useful downstream graph representations. It compares self-supervised GNN encoders under controlled evaluation and documents practical training and evaluation choices.

  • Motivating experiment: Mutual-information maximization can preserve graph identification while still allowing redundant information that may harm downstream usefulness.
  • Motivating experiment: Both GNN encoders produce distinct representations for every pair of graphs, consistent with one-to-one graph–representation correspondence.
  • Evaluation protocol: Downstream comparisons use identical logistic-classifier architecture, hyperparameters, and initialization for the two encoders.The classifier uses L2 regularization tuned on the validation set.
  • Evaluation protocol: The evaluation protocol trains self-supervised encoders first, then applies linear Ridge or Logistic models to graph representations.
  • Experimental controls: All experiments use the same fixed GIN encoder design across methods to attribute performance differences to self-supervised objectives.
  • Optimization: AD-GCL uses Adam with a learning rate of 0.001 for stability, while training epochs are selected using validation performance.

H.1 Experimental Comparison

AD-GCL is compared with JOAO and JOAOv2 under unsupervised, transfer, and semi-supervised evaluation settings. The reported comparisons favor AD-GCL across most benchmark datasets, with the strongest coverage in semi-supervised learning.

  • Unsupervised comparison: AD-GCL outperforms JOAO on 6 of 8 TU benchmark datasets in the unsupervised linear-evaluation comparison.
  • Transfer learning: In transfer learning, AD-GCL outperforms JOAO on 7 of 9 datasets and JOAOv2 on 5 of 9 datasets.
  • Semi-supervised learning: In semi-supervised learning, AD-GCL outperforms both JOAO and JOAOv2 on all 6 datasets.
  • Evaluation settings: Transfer results use mean ROC-AUC ± standard deviation with pre-training on ZINC 2M or PPI-306K.
  • Evaluation settings: Semi-supervised results use a 10% label rate and report 10-fold Accuracy (%).

I Limitations and Broader Impact

The paper identifies limitations of self-supervised graph representation learning and discusses broader implications. Without downstream-task information, augmentation design can introduce biases that affect learned representations and downstream performance.

  • Limitations: Self-supervised methods lack access to downstream-task information, creating a fundamental limitation for representation learning.
  • Limitations: Contrastive augmentation design guides which graph invariances encoders capture, so augmentation biases can produce sub-optimal or non-robust representations.
  • Broader impact: Graph representation learning has implications for fairness, robustness, and privacy as adoption expands across fields.
  • Experimental resources: The experiments were run on a compute cluster using Nvidia GeForce GTX 1080 Ti, GTX TITAN X, and TITAN Xp GPUs.
Loading 2106.05819v4…