PaperlayerView exact revision on arXiv

Source-linked AI summary

Let Invariant Rationale Discovery Inspire Graph Contrastive Learning

Sihang Li, Xiang Wang, An zhang, Yingxin Wu, Xiangnan He, Tat-Seng Chua

arXiv:2206.07869v1cs.LG

TL;DR

Existing GCL augmentations may lose graph semantics or rely on domain knowledge that limits generalization. RGCL discovers instance-discriminative rationales and uses them to create contrastive views; it surpasses current state-of-the-art GCL methods on biochemical molecule and social network benchmarks.

  • Problem

    Random graph corruption may lose discriminative semantics, while domain-knowledge-guided augmentation can undermine generalization to unseen domains.

    Source: §1 · p. 2

  • Method

    RGCL uses a rationale generator to discover instance-discriminative graph information and creates rationale-aware views for contrastive learning.

    Source: §1 · p. 2, §3.3 · p. 5

  • Results

    RGCL surpasses current state-of-the-art GCL methods on biochemical molecule and social network benchmark datasets.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Visual inspections on MNIST-Superpixel rationales show the generator can discover discriminative nodes in graph instances.

    Source: §4.1 · p. 6, §5 · p. 9

    The rationale generator struggles to generalize from ZINC15 pre-training to HIV, where it fails to capture discriminative features.

    Source: §4.2 · p. 8

Abstract

from arXiv · show

Leading graph contrastive learning (GCL) methods perform graph augmentations in two fashions: (1) randomly corrupting the anchor graph, which could cause the loss of semantic information, or (2) using domain knowledge to maintain salient features, which undermines the generalization to other domains. Taking an invariance look at GCL, we argue that a high-performing augmentation should preserve the salient semantics of anchor graphs regarding instance-discrimination. To this end, we relate GCL with invariant rationale discovery, and propose a new framework, Rationale-aware Graph Contrastive Learning (RGCL). Specifically, without supervision signals, RGCL uses a rationale generator to reveal salient features about graph instance-discrimination as the rationale, and then creates rationale-aware views for contrastive learning. This rationale-aware pre-training scheme endows the backbone model with the powerful representation ability, further facilitating the fine-tuning on downstream tasks. On MNIST-Superpixel and MUTAG datasets, visual inspections on the discovered rationales showcase that the rationale generator successfully captures the salient features (i.e. distinguishing semantic nodes in graphs). On biochemical molecule and social network benchmark datasets, the state-of-the-art performance of RGCL demonstrates the effectiveness of rationale-aware views for contrastive learning. Our codes are available at https://github.com/lsh0520/RGCL.

1. Introduction

RGCL addresses the trade-off between random augmentations that may lose graph semantics and knowledge-guided augmentations that can undermine generalization. It uses invariant rationale discovery to identify instance-discriminative information and create rationale-aware views for contrastive learning.

  • Random corruption may remove discriminative semantics, while domain-knowledge-guided augmentation can undermine generalization to unseen domains.Source: §1 · pp. 1–2
  • RGCL relates GCL to invariant rationale discovery, using rationale-aware augmentations that preserve instance-discriminative information.Its rationale generator reveals and conceals parts of the anchor graph, and a contrastive learner uses the resulting views for instance discrimination.Source: §1 · p. 2
  • RGCL surpasses current state-of-the-art GCL methods on biochemical molecule and social network benchmark datasets.Source: §1 · p. 2

2. Invariance Look at GCL

The section frames GCL and invariant rationale discovery through representation invariance: contrastive learning seeks agreement across augmented views, while rationale discovery identifies information preserved for prediction. It connects these ideas by proposing rationale-aware augmentations that retain instance-discriminative information.

2.1. Graph Contrastive Learning (GCL)

  • GCL creates augmented graph views and trains representations to distinguish instances, using random corruption or domain knowledge to guide augmentation.Random methods risk semantic information loss, while reliance on domain knowledge can limit generalization across domains.Source: §2.1 · pp. 2–3

2.2. Invariant Rationale Discovery (IRD)

  • IRD extracts a supporting substructure as a rationale; sufficiency preserves label-relevant information, while independence makes classification insensitive to the rationale’s complement.Source: §2.2 · p. 3

2.3. Connecting GCL and IRD

  • RGCL applies rationale discovery to GCL by revealing instance-discriminative information in augmented views while concealing the rationale’s complement.Source: §2.3 · p. 4

3. Methodology

RGCL discovers instance-discriminative rationales and uses rationale-aware views for contrastive pre-training. Its pipeline probabilistically samples graph views, converts them into weighted representations, and trains them with sufficiency and independence objectives.

3. Methodology

  • RGCL discovers rationales for instance-discriminative semantics, then performs contrastive learning on rationale-aware views.The paper presents this as a pre-training scheme.Source: §3 · p. 4

3.1. Rationale-aware Graph Augmentation

  • RGCL probabilistically scores node salience and samples rationale and complement views, retaining edges among sampled nodes.The rationale generator parameterizes node attribution probabilities; probabilistic sampling is used to maintain view diversity.Source: §3.1 · pp. 4–5

3.2. Rationale-aware Representation Learning

  • The backbone weights rationale-node embeddings by their attribution scores and pools them into a rationale representation; fine-tuning disables rationale discovery and discards those scores.The rationale representation is d′-dimensional.Source: §3.2 · p. 5

3.3. Rationale-aware Contrastive Learning

  • Two sampled rationales from one anchor form a positive pair, while rationales from other anchors serve as negative views in the sufficiency loss.This loss encourages agreement between same-anchor views and divergence between different anchors.Source: §3.3 · p. 5
  • The independence loss pushes complement representations away from rationale representations, and λ balances it with the sufficiency loss in a backbone-agnostic framework.The rationale generator and target backbone optimize the combined objective cooperatively.Source: §3.3 · pp. 5–6

4. Experiments

The experiments assess whether RGCL discovers semantically important graph nodes and whether rationale-aware views improve downstream representation learning. They also examine component ablations and sensitivity to three hyperparameters.

4.1. Effectiveness of Rationale Generator (RQ1)

  • RGCL rationales achieved 78.9% precision against expert labels on MUTAG and captured semantic MNIST-Superpixel nodes for rationale-augmented views.For MNIST-Superpixel, pre-training omitted train labels and fine-tuning used 1% or 10% labeled samples.Source: §4.1 · p. 6

4.2. Performance on Downstream Tasks (RQ2)

  • Across transfer and unsupervised benchmarks, RGCL led overall: best on 4/8 transfer datasets, highest average gain, and best average TU accuracy.It outperformed GraphCL on 7/8 transfer datasets, with average performance gain rising from 3.77 to 6.16; it also led on 5/8 unsupervised cases.Source: Table 2 · p. 7, §4.2 · pp. 7–8, Table 3 · p. 8

4.3. Ablation Study

  • Ablations show sharp performance drops without rationale-aware views and lower performance without the independence term.The study evaluates these variants on downstream transfer-learning datasets.Source: Table 4 · p. 8, §4.3 · p. 8

4.4. Hyperparameter sensitivity

  • Sensitivity tests suggest tuning τ near 0.1 and ρ near 80%, while λ = 0.1 prevents degeneration.Too-small τ or τ above 0.2 hurts performance; overemphasizing the independence term also lowers it, and too-small ρ increases the gap between augmented and anchor graphs.Source: Figure 4 · p. 8, §4.4 · pp. 8–9

5. Conclusion

RGCL uses invariant rationale discovery to generate semantic-information-preserving graph views for GNN pre-training, with visual and benchmark evaluations supporting its rationale discovery and downstream transferability. The paper identifies further rationale-generator fine-tuning and retrospective and introspective rationale learning as future directions.

5. Conclusion

  • RGCL uses a GNN-MLP rationale generator without domain knowledge or manually designed features to discover semantic nodes and generate views for contrastive learning.The paper reports visual inspections on MNIST-Superpixel and state-of-the-art benchmark performance on biochemical molecule and social network datasets.Source: §5 · p. 9
  • Visual inspections on MNIST-Superpixel show the rationale generator can discover discriminative nodes, while benchmark experiments report improved transferability and generalization with state-of-the-art performance.Source: §5 · p. 9
  • Future work will fine-tune rationale generators on downstream tasks and explore retrospective and introspective rationale discovery to guide discrimination tasks.The paper also proposes probing GCL explainability.Source: §5 · p. 9

A. Rationale-aware Graph Contrastive Learning (RGCL) algorithm

The RGCL algorithm samples rationale-aware and complementary views from each graph using a rationale generator, encodes and projects them, then updates the encoder, generator, and projector to minimize the RGCL loss.

A. Rationale-aware Graph Contrastive Learning (RGCL) algorithm

B. MNIST-superpixel Settings

The MNIST-superpixel settings section summarizes dataset statistics and lists the comparison baselines for semi-supervised experiments.

B. MNIST-superpixel Settings

C. Transfer Learning Settings

The transfer-learning settings use MoleculeNet downstream tasks and compare RGCL with several self-supervised pre-training frameworks.

C. Transfer Learning Settings

D. Unsupervised Learning Settings

The section identifies the TU datasets used for unsupervised learning and names the baselines used to evaluate RGCL.

D. Unsupervised Learning Settings

E. Model Structure and Hyperparameters

The model follows You et al. (2020) backbone settings and uses GCN and GIN architectures; Table 8 summarizes architectures and hyperparameters.

E. Model Structure and Hyperparameters

F. Complexity Analysis

RGCL adds a rationale generator and complement computation, with a modestly longer reported pre-training time than GraphCL on the stated platform.

F. Complexity Analysis

  • The rationale generator does not significantly increase space complexity because it is much smaller than the backbone encoder.Its scale can be adjusted if necessary.Source: F. Complexity Analysis · p. 13
  • The time-complexity analysis compares RGCL and GraphCL for a minibatch using graph-size, batch, sampling, layer-count, and latent-dimension terms.The comparison is presented in Table 9; the supplied passage does not give its complexity expressions.Source: F. Complexity Analysis · p. 14, Table 9 · p. 14
  • 20 hours for RGCL versus 15 hours for GraphCL were reported for pre-training on the stated platform.The platform used a GeForce RTX 2080 Ti and Intel(R) Core(TM) i9-9900X.Source: F. Complexity Analysis · p. 14
  • RGCL's three-tower structure adds calculation of rationale complements to regulate information outside the rationale.GraphCL is described as having a two-tower structure.Source: F. Complexity Analysis · p. 14

Related papers

Loading 2206.07869v1…