PaperlayerView exact revision on arXiv

Source-linked AI summary

Graph Contrastive Learning with Augmentations

Yuning You, Tianlong Chen, Yongduo Sui, Ting Chen, Zhangyang Wang, Yang Shen

arXiv:2010.13902v3cs.LGcs.AI

TL;DR

Generalizable, transferable, and robust representation learning remains challenging for GNNs because self-supervised pre-training is less explored than for CNNs. GraphCL designs graph augmentations and contrastive pre-training, finding similar or better generalizability, transferability, and robustness than state-of-the-art methods. The study also reports further preliminary gains from parameterized augmentation extents and patterns.

  • Problem

    Self-supervised learning and pre-training are less explored for GNNs, while generalizable, transferable, and robust graph representations remain challenging.

  • Method

    GraphCL designs four types of graph augmentations and uses contrastive learning to pre-train GNNs on representations invariant to specialized perturbations.

  • Results

    GraphCL produces representations with similar or better generalizability, transferability, and robustness than state-of-the-art methods across semi-supervised, unsupervised, transfer-learning, and adversarial-attack settings.

  • Takeaways & Limitations

    Appropriate augmentation combinations can improve downstream performance, with reported single-best augmentation gains of 1.62% on NCI1, 3.15% on PROTEINS, 6.27% on COLLAB, and 1.66% on RDT-B.

  • Takeaways & Limitations

    Graph augmentations may not be universally appropriate because graph datasets come from diverse fields, and effective augmentation choices can differ across dataset categories.

Abstract

from arXiv · show

Generalizable, transferrable, and robust representation learning on graph-structured data remains a challenge for current graph neural networks (GNNs). Unlike what has been developed for convolutional neural networks (CNNs) for image data, self-supervised learning and pre-training are less explored for GNNs. In this paper, we propose a graph contrastive learning (GraphCL) framework for learning unsupervised representations of graph data. We first design four types of graph augmentations to incorporate various priors. We then systematically study the impact of various combinations of graph augmentations on multiple datasets, in four different settings: semi-supervised, unsupervised, and transfer learning as well as adversarial attacks. The results show that, even without tuning augmentation extents nor using sophisticated GNN architectures, our GraphCL framework can produce graph representations of similar or better generalizability, transferrability, and robustness compared to state-of-the-art methods. We also investigate the impact of parameterized graph augmentation extents and patterns, and observe further performance gains in preliminary experiments. Our codes are available at https://github.com/Shen-Lab/GraphCL.

1 Introduction

GNNs have achieved strong performance on graph tasks, but self-supervised pre-training remains little explored despite scarce task-specific labels. Graph pre-training is difficult because graph data are heterogeneous and naïve adjacency reconstruction can overemphasize proximity.

  • GNNs commonly use end-to-end supervised training for graph-level tasks, while self-supervised pre-training remains comparatively unexplored.
  • Task-specific graph labels can be scarce because biological and chemical labeling often requires resource- and time-intensive wet-lab experiments.The paper argues that pre-training could help mitigate this scarcity, as it does for CNNs.
  • Graph pre-training is challenging because graphs contain rich structured information across diverse contexts, making generic task-beneficial schemes difficult to design.
  • Adjacency reconstruction is limited because it over-emphasizes proximity, which is not always the relevant information for graph-level tasks.
  • Contrastive learning seeks feature consistency across differently augmented views, potentially improving the generality of graph pre-training beyond handcrafted pretext tasks.
  • GraphCL uses four parameterized graph augmentations to create correlated views and learn representations invariant to specialized perturbations.The framework is presented as contrastive learning with augmentations for GNN pre-training and is connected to mutual-information maximization.

2 Related Work

GNNs learn graph representations through iterative neighborhood aggregation, graph-level readout, and downstream prediction. Related work covers graph augmentation, GNN pre-training, and contrastive learning, with adjacency reconstruction criticized for emphasizing local proximity over structural information.

  • Graph neural networks: GNNs iteratively aggregate information from neighboring nodes to capture local graph structure.The kth layer combines aggregated neighbor information with the node’s previous representation.
  • Graph neural networks: After K-layer propagation, a READOUT function summarizes layer embeddings into a graph representation, followed by an MLP for graph-level classification or regression.
  • Graph data augmentation: Graph data augmentation remains under-explored, with some existing approaches requiring prohibitive additional computation or generating adversarial perturbations.
  • Pre-training GNNs: GNN pre-training is rarely explored compared with CNN pre-training, and transfer can require substantial domain knowledge to avoid negative transfer.
  • Contrastive learning: Contrastive learning encourages representations to agree under appropriate transformations, whereas adjacency reconstruction can over-emphasize vertex proximity at the expense of structural information.

3 Methodology

GraphCL creates correlated graph views through domain-sensitive augmentations, encodes them with a shared GNN and projection head, and trains representations by contrastive agreement. Its formulation connects augmentation-based graph contrastive learning to mutual-information maximization and a broader family of contrastive methods.

  • Data Augmentation for Graphs: Graph augmentation transforms an input graph into a realistically rational view while preserving its semantic label, but suitable transformations vary across graph categories.The paper focuses on biochemical molecules, social networks, and image super-pixel graphs.
  • Data Augmentation for Graphs: GraphCL uses node dropping, edge perturbation, attribute masking, and random-walk subgraph sampling as four graph augmentations.Their priors assume that semantics tolerate missing vertices, connectivity changes, partial attributes, or partial local structure.
  • Graph Contrastive Learning: Two augmentations sampled from an augmentation pool produce correlated views of the same graph as a positive pair.The default augmentation ratio for dropping, perturbation, masking, and subgraph sampling is 0.2.
  • Graph Contrastive Learning: A shared GNN encoder maps the augmented graphs to graph-level representations, and a nonlinear projection head maps them into the latent space used for contrastive loss.The framework does not impose a constraint on the GNN architecture.
  • Graph Contrastive Learning: The NT-Xent loss maximizes consistency between positive representations relative to negatives generated from the other augmented graphs in the minibatch.A minibatch of N graphs yields 2N augmented graphs, with τ as the temperature parameter.
  • Graph Contrastive Learning: GraphCL can be viewed as mutual-information maximization and rewritten as a general framework encompassing multiple graph contrastive learning methods.The paper’s implementation uses the same encoder for both views and generates them through data augmentation.

4 The Role of Data Augmentation in Graph Contrastive Learning

GraphCL experiments show that suitable graph augmentations are essential, and that augmentation combinations, types, strengths, and patterns affect downstream performance in dataset-dependent ways. Different augmentation choices can improve or harm results depending on graph structure and domain.

  • Data Augmentations are Crucial. Composing Augmentations Benefits.: Appropriate augmentations improved accuracy over training from scratch by 1.62% on NCI1, 3.15% on PROTEINS, 6.27% on COLLAB, and 1.66% on RDT-B.Without augmentation, graph contrastive learning was often worse than training from scratch.
  • Data Augmentations are Crucial. Composing Augmentations Benefits.: Augmentation pairs achieved maximum accuracy gains of 2.10% for NCI1, 3.15% for PROTEINS, 7.11% for COLLAB, and 1.85% for RDT-B.Different augmentation types generally outperformed same-type pairs, which the authors conjecture avoids shortcut overfitting.
  • Data Augmentations are Crucial. Composing Augmentations Benefits.: Different augmentation types produced slower contrastive-loss descent than same-type pairs, indicating a harder contrastive task.The comparison used augmentation pairs involving attribute masking or edge perturbation on NCI1 and PROTEINS.
  • The Types, the Extent, and the Patterns of Effective Graph Augmentations: Edge perturbation improved COLLAB, RDT-B, and PROTEINS but hurt NCI1, where changing an edge can alter a covalent bond and compound identity.The authors relate this contrast to greater tolerance of individual edge changes in social networks.
  • The Types, the Extent, and the Patterns of Effective Graph Augmentations: Attribute masking improved COLLAB by 5.12% but RDT-B by only 0.17%, with stronger masking helping dense COLLAB but not sparse RDT-B.The same density-related pattern was observed between PROTEINS and NCI1.
  • The Types, the Extent, and the Patterns of Effective Graph Augmentations: Masking high-degree nodes showed an upward performance tendency on dense COLLAB, while non-uniform node dropping did not necessarily help less-dense PROTEINS.For dense COLLAB, dropping hub nodes more often produced more GraphCL improvements.
  • The Types, the Extent, and the Patterns of Effective Graph Augmentations: Node dropping and subgraph augmentations were generally beneficial across the studied datasets, especially subgraph augmentation.The authors associate node dropping with the prior that removing certain vertices does not alter graph semantics.
  • The Types, the Extent, and the Patterns of Effective Graph Augmentations: Larger dropping or masking ratios and higher control factors produced better COLLAB performance within the tested range by creating harder contrastive tasks.Very small ratios or negative control factors corresponded to overly simple tasks.

5 Comparison with the State-of-the-art Methods

GraphCL is evaluated against state-of-the-art and conventional pre-training methods across semi-supervised, unsupervised, transfer-learning, and adversarial-robustness settings. It generally performs strongly, while remaining less consistently beneficial for small graphs and out-of-distribution transfer.

  • Semi-supervised learning: GraphCL outperforms traditional pre-training schemes in semi-supervised graph classification.The comparison includes adjacency reconstruction, local-global representation consistency, training from scratch, and augmentation without contrasting.
  • Semi-supervised learning: Table 3 compares semi-supervised pre-training and finetuning at 1% or 10% label rates, distinguishing baseline from augmentation-only training.Red values mark the best performance, including values overlapping the best result’s standard deviation as comparable.
  • Unsupervised representation learning: GraphCL outperforms most compared methods in unsupervised representation learning, except on datasets with small graph size.The comparison covers graph kernels and unsupervised representation-learning methods, including node2vec, sub2vec, graph2vec, and InfoGraph.
  • Adversarial robustness: GraphCL boosts GNN robustness compared with training from scratch under three evasion attacks.The attacks are RandSampling, GradArgmax, and RL-S2V, evaluated on synthetic graphs for component-number classification.

6 Conclusion

The paper studies contrastive learning for GNN pre-training under graph-specific challenges. It introduces augmentations and GraphCL, analyzes augmentation choices, and reports state-of-the-art generalizability and robustness.

  • Conclusion: The paper proposes graph augmentations that introduce human priors about graph-data distributions.It studies the influence of these augmentations to guide their selection within GraphCL.
  • Conclusion: GraphCL facilitates invariant representation learning through contrastive GNN pre-training.The framework is accompanied by theoretical analysis and systematic augmentation studies.
  • Conclusion: Experiments verify state-of-the-art performance in both generalizability and robustness.This conclusion summarizes the paper’s reported empirical findings.

Broader Impact

GraphCL is presented as a general framework for pre-training graph neural networks. The paper connects it to applications involving reasoning and prediction over graph-structured data.

  • Broader Impact: GraphCL can potentially improve the effectiveness and efficiency of graph neural networks through model pre-training.The paper identifies recommendation systems, neural architecture search, and drug discovery as application areas for graph-structured reasoning and prediction.
  • Broader Impact: The numerical results and analyses can inspire the design of proper graph augmentations.The stated broader-impact claim remains within the paper’s proposed framework and reported analyses.
Loading 2010.13902v3…