Source-linked AI summary
InfoGCL: Information-Aware Graph Contrastive Learning
Dongkuan Xu, Wei Cheng, Dongsheng Luo, Haifeng Chen, Xiang Zhang
TL;DR
Existing graph contrastive learning models require customized choices for particular tasks and datasets, leaving guiding principles for selecting their components open. InfoGCL applies Information Bottleneck ideas across view augmentation, encoding, and contrasting to preserve task-relevant information while reducing unnecessary shared information. It unifies recent graph contrastive methods and reports up to 5.2% relative accuracy improvement on graph classification with competitive node-classification results.
Problem
Existing graph contrastive models differ across augmentations, architectures, and objectives, leaving it unclear how to design them for specific graph tasks and datasets.
Method
InfoGCL decouples graph contrastive learning into three modules and uses Information Bottleneck principles to reduce shared information while preserving task-relevant information at module and framework levels.
Results
Up to 5.2% relative improvement in accuracy is reported for graph classification, with competitive node-classification results against state-of-the-art unsupervised methods.
Takeaways & Limitations
The framework provides practically feasible principles for selecting graph contrastive modules and unifies recent graph contrastive learning methods.
Takeaways & Limitations
The hypothesis that negative samples benefit graph modeling is specifically tied to datasets with extremely sparse network topology or node features.
Abstract
from arXiv · showhide
Various graph contrastive learning models have been proposed to improve the performance of learning tasks on graph datasets in recent years. While effective and prevalent, these models are usually carefully customized. In particular, although all recent researches create two contrastive views, they differ greatly in view augmentations, architectures, and objectives. It remains an open question how to build your graph contrastive learning model from scratch for particular graph learning tasks and datasets. In this work, we aim to fill this gap by studying how graph information is transformed and transferred during the contrastive learning process and proposing an information-aware graph contrastive learning framework called InfoGCL. The key point of this framework is to follow the Information Bottleneck principle to reduce the mutual information between contrastive parts while keeping task-relevant information intact at both the levels of the individual module and the entire framework so that the information loss during graph representation learning can be minimized. We show for the first time that all recent graph contrastive learning methods can be unified by our framework. We empirically validate our theoretical analysis on both node and graph classification benchmark datasets, and demonstrate that our algorithm significantly outperforms the state-of-the-arts.
1 Introduction
Graph contrastive learning typically creates two augmented graph views and maximizes their representation consistency, but existing models require task- and dataset-specific choices. InfoGCL uses Information Bottleneck principles to guide these choices while preserving task-relevant information, and achieves strong node- and graph-classification results.
- Motivation: Existing graph contrastive models differ in view augmentation, encoding architecture, and contrastive objective, making task-specific customization dependent on substantial domain expertise.The paper highlights differing choices such as graph encoders, diffusion convolution, subgraph sampling, and graph perturbation.
- Research question: The paper asks how to perform contrastive learning for specific graph tasks and datasets when multiple component choices make robust and efficient model design difficult.The challenge also reflects graph data’s rich structural information and diverse distributions.
- Approach: InfoGCL decouples graph contrastive learning into view augmentation, view encoding, and representation contrasting, then formalizes optimality for the three modules.Its analysis studies how information is transformed and transferred during contrastive learning.
- Approach: The proposed principles seek views with high task-relevant information and low shared information, simple task-relevant encoders, and contrastive modes that preserve task-relevant information.The paper also argues that negative samples are not necessarily required, especially when graph data is not extremely sparse.
- Results: Up to 5.2% relative accuracy improvement is reported for graph classification, alongside competitive node-classification results against state-of-the-art unsupervised methods.The evaluation covers benchmark datasets for both node-level and graph-level tasks.
2 Related Work
The related work situates InfoGCL at the intersection of graph contrastive learning and Information Bottleneck theory. Unlike prior graph representation studies and vision-domain view theories, it addresses information trade-offs across two graph views and three graph-specific stages.
- Graph Contrastive Learning: Graph contrastive learning methods transfer contrastive learning from vision and language to graphs by creating two views and maximizing feature disagreement or consistency between representations.Examples include DGI, InfoGraph, and other graph contrastive approaches with differing designs.
- Information Bottleneck: Information Bottleneck balances representation accuracy and complexity through mutual information between input, latent representation, and task label.It favors representations expressive about the task while minimally expressive about the input.
- Positioning: InfoGCL applies Information Bottleneck to graph contrastive learning, where two augmented views create a distinct information trade-off.The framework also considers graph-specific properties through three stages, differing from prior vision-domain minimal-sufficient-statistics work.
3 Preliminaries and Notations
Graph representation learning generates node- or graph-level latent representations from graph structure and attributes, while graph contrastive learning trains representations through two augmented views. Its pipeline comprises view augmentation, view encoding, and representation contrasting with a contrastive loss.
- Graph Representation Learning: A graph G=(A,X) consists of an adjacency matrix A and node attribute matrix X; node- and graph-level tasks learn representations used to predict corresponding labels.Graph neural networks typically generate graph representations from the input graph.
- Graph Contrastive Learning: Graph contrastive learning maximizes feature consistency between representations of two augmented views through a latent-space contrastive loss.The framework is applicable to graph-level and node-level representation learning.
- View augmentation: View augmentation transforms graph G into two views vi and vj using methods such as node dropping, edge perturbation, or subgraph sampling.Each view is represented by graph data including an adjacency matrix and node attribute matrix.
- View encoding: View encoding applies GNN-based encoder networks, optionally with projection MLPs and shared parameters, to obtain latent representations zi and zj.Encoders may share parameters when the views come from the same domain.
- Representation contrasting: Representation contrasting optimizes a loss that scores positive representation pairs higher than negative pairs, commonly using negatives from other graphs in the minibatch.InfoNCE is a popular choice, and minimizing it maximizes a lower bound on mutual information between positive-pair views.
4 Information-Aware Graph Contrastive Learning
InfoGCL characterizes graph contrastive learning through optimal views, view encoders, and contrastive modes for a task and dataset. It converts these information-theoretic conditions into practical module-selection principles and considers several graph-specific contrastive modes.
- 4.1 View Augmentation: Optimal augmented views maximize task-relevant information while minimizing information shared between views.The formal objective minimizes I(v_i; v_j) subject to preserving equal task information in both views.
- 4.2 View Encoding: Optimal view encoders retain the information shared by the two views while keeping the retained information task-relevant.View encoding extracts node or graph representations using flexible graph encoders such as GCN, GAT, and GIN.
- 4.3 Representation Contrasting: The optimal contrastive mode preserves the most task-relevant information after aggregating latent representations.The framework considers global-global, local-local, multi-scale, and hybrid modes, with hybrid combining global-global and local-global contrasting.
- 4.4 InfoGCL Principle: Because exact optimality is difficult under data noise and limited model capability, InfoGCL optimizes each module independently as a practical approximation.The proposed principles select augmentations by task-relevant and shared information, encoders by mutual information, and contrastive modes by task-relevant mutual information.
- 4.5 Role of Negative Samples: The framework also studies negative samples and revises the loss following SimSiam to evaluate graph contrastive learning without them.This analysis covers both node and graph classification tasks.
5 Experiments
Experiments evaluate InfoGCL on graph- and node-classification benchmarks using established protocols and comparisons. Results also examine the InfoGCL principles and the role of negative samples.
- Experimental setup: InfoGCL is evaluated on widely used graph- and node-classification benchmark datasets.The experiments describe datasets, evaluation protocols, baselines, and ablation analyses.
- Graph classification: InfoGCL achieves the best results among unsupervised approaches on graph classification and ranks among the top methods on all six datasets against supervised baselines.It is best on 2 of 6 datasets and among the top on the other 4.
- Node classification: InfoGCL achieves state-of-the-art node-classification results, competes with the best unsupervised approaches, and outperforms all supervised baselines.The comparison includes both supervised and unsupervised methods.
- Evaluation of InfoGCL principle: Existing graph contrastive learning methods can be unified as three stages that implicitly follow the InfoGCL principle despite differing architectures and optimization strategies.The stages are analyzed through the InfoGCL perspective.
- Evaluation of InfoGCL principle: Different augmentations reduce mutual information between views, while node dropping and subgraph sampling generally preserve more task-relevant information than attribute masking and edge perturbation.Edge perturbation benefits social networks but can hurt some biochemical molecules because local connection patterns determine their semantic meaning.
- Evaluation of InfoGCL principle: Node-graph contrastive modes generally extract more graph-structure information and consistently perform better across benchmarks than other contrastive modes.This comparison is linked to improved prediction of the task label y.
- Effect of negative samples: Negative samples have little influence on three graph-task datasets but reduce performance on three node-task datasets, especially Cora.The study uses a SimSiam-style objective to compare the two settings.
6 Conclusion and Limitations
The paper concludes that InfoGCL provides an information-aware framework for graph contrastive learning and demonstrates effectiveness on node- and graph-classification benchmarks. It also identifies better practical approximations to theoretical optimality as a limitation and future improvement direction.
- Conclusion: InfoGCL decouples graph contrastive learning into three sequential modules and provides theoretical analysis for their optimality.The framework is presented as a way to address task- and dataset-specific graph contrastive learning design.
- Conclusion: Experiments on node- and graph-classification benchmark datasets demonstrate the effectiveness of InfoGCL.The paper also reports that negative samples are not necessarily required.
- Limitations: A stated limitation is that better practical approximations to the theoretical optimality of graph contrastive learning could further improve InfoGCL.The paper identifies this as a direction for future improvement.