GCC: Graph Contrastive Coding for Graph Neural Network Pre-Training

Jiezhong Qiu; Qibin Chen; Yuxiao Dong; Jing Zhang; Hongxia Yang; Ming Ding; Kuansan Wang; Jie Tang

Source-linked AI summary

GCC: Graph Contrastive Coding for Graph Neural Network Pre-Training

Jiezhong Qiu, Qibin Chen, Yuxiao Dong, Jing Zhang, Hongxia Yang, Ming Ding, Kuansan Wang, Jie Tang

arXiv:2006.09963v3Jun 17, 2020cs.LGcs.SIstat.ML

TL;DR

Graph representation learning has largely produced graph-specific models with limited transfer to out-of-domain data. GCC pre-trains GNNs across diverse graphs using self-supervised contrastive subgraph discrimination, and its representations achieve competitive or better performance across downstream tasks and datasets. The findings support the potential of graph pre-training and fine-tuning for transferable structural representation learning.

Problem
Most graph representation models learn network-specific patterns for individual datasets, leaving limited transfer to out-of-domain data and tasks.
Method
GCC pre-trains a GNN across diverse graphs using self-supervised contrastive subgraph instance discrimination to learn structural representations.
Results
Across three graph learning tasks and ten graph datasets, GCC achieves competitive or better performance than task-specific or trained-from-scratch graph models.
Takeaways & Limitations
The results suggest that pre-training and fine-tuning can support transferable graph representation learning across networks and tasks.
Takeaways & Limitations
The study focuses on social and information networks and identifies broader benchmarking on more diverse datasets, such as protein-protein association networks, as future work.

Abstract

from arXiv · show

Graph representation learning has emerged as a powerful technique for addressing real-world problems. Various downstream graph learning tasks have benefited from its recent developments, such as node classification, similarity search, and graph classification. However, prior arts on graph representation learning focus on domain specific problems and train a dedicated model for each graph dataset, which is usually non-transferable to out-of-domain data. Inspired by the recent advances in pre-training from natural language processing and computer vision, we design Graph Contrastive Coding (GCC) -- a self-supervised graph neural network pre-training framework -- to capture the universal network topological properties across multiple networks. We design GCC's pre-training task as subgraph instance discrimination in and across networks and leverage contrastive learning to empower graph neural networks to learn the intrinsic and transferable structural representations. We conduct extensive experiments on three graph learning tasks and ten graph datasets. The results show that GCC pre-trained on a collection of diverse datasets can achieve competitive or better performance to its task-specific and trained-from-scratch counterparts. This suggests that the pre-training and fine-tuning paradigm presents great potential for graph representation learning.

1 INTRODUCTION

GCC addresses the limited transferability of graph representations by pre-training GNNs to capture structural patterns shared across diverse networks. It uses contrastive subgraph instance discrimination and achieves competitive or better results on out-of-domain graph tasks.

Motivation: The paper hypothesizes that representative graph structural patterns are universal and transferable across networks.This motivates learning graph embeddings across diverse networks rather than dedicating each model to one dataset.
Motivation: Most graph representation models learn network-specific patterns for individual datasets, limiting transfer to out-of-domain data and tasks.DeepWalk trained on Facebook, for example, cannot be applied to other graphs.
Approach: GCC frames GNN pre-training as self-supervised learning across diverse input graphs, followed by fine-tuning on different graphs and tasks.The motivating example pre-trains on Facebook, IMDB, and DBLP, then applies the model to node classification on US-Airport.
Approach: GCC uses contrastive subgraph instance discrimination: subgraphs sampled from each vertex’s multi-hop ego network are distinguished from subgraphs sampled from other vertices.A GIN encoder maps the sampled subgraphs’ structural patterns to latent representations.
Results: Across three downstream tasks and ten new graph datasets, GCC achieves competitive or better performance than task-specific models trained from scratch.On US-Airport node classification, GCC pre-trained on Facebook, IMDB, and DBLP outperforms GraphWave, ProNE, and Struc2vec.
Contributions: The contributions formalize cross-graph GNN pre-training, introduce subgraph instance discrimination, and apply contrastive learning to structural representation learning.The framework targets structural representations without requiring graph-specific supervised training.

2 RELATED WORK

Related work spans vertex similarity, contrastive learning, and graph pre-training. GCC differs by targeting transferable structural representations across graphs rather than representations tied to a single graph or specialized domain.

Vertex Similarity: Vertex similarity research distinguishes neighborhood, structural, and attribute similarity, with structural similarity based on similar local structures even without connectivity.Neighborhood methods rely on connectivity, while attribute-based GNNs use side information or supervised signals.
Contrastive Learning: Contrastive learning captures similarity by bringing related views or instances together, and GCC adopts InfoNCE loss with instance discrimination.The paper connects this approach to self-supervised representation learning in language and vision.
Graph Pre-Training: Skip-gram network embedding models such as LINE, DeepWalk, and node2vec learn graph-tied representations and cannot handle out-of-sample problems.These methods generally follow the neighborhood similarity assumption.
Graph Pre-Training: Recent GNN pre-training methods target labeled molecular graphs or synthetic graph-learning tasks, whereas GCC focuses on structural pre-training across multiple graphs.Molecular pre-training recovers masked atom and bond types; another approach pre-trains GCNs on synthetic tasks.

3 GRAPH CONTRASTIVE CODING (GCC)

GCC formalizes graph neural network pre-training to learn structural patterns that are similar across graphs and transferable to unseen datasets and tasks. It uses subgraph instance discrimination with contrastive learning, graph sampling, structural positional features, and GNN encoders.

Problem formulation: GCC pre-trains a GNN on collections of graphs to capture common structural patterns for downstream tasks on different datasets.The target embedding function should preserve structural similarity and transferability to unseen vertices and graphs.
Problem formulation: The framework focuses on structural representation learning without node attributes or labels and applies the resulting model to unseen graphs.This distinguishes GCC from attributed-graph pre-training within a specific domain and from traditional network embeddings.
Contrastive objective and encoders: GCC uses subgraph instance discrimination with InfoNCE, encoding query and key subgraphs into d-dimensional representations and contrasting one matched key against a dictionary of negatives.The encoders are implemented with GIN, while graph structure initializes vertex features through generalized positional embeddings based on normalized graph Laplacian eigenvectors.
Cross-graph pre-training: The framework is not restricted to matched instances from the same graph, enabling discrimination between subgraph instances in and across graphs.GCC identifies instance definitions, instance-pair definitions, and graph encoders as its three central design questions.
Subgraph instances: GCC defines each vertex’s r-ego network as a subgraph instance induced by vertices within shortest-path distance r.Each r-ego network is treated as a distinct class for contrastive discrimination.
Instance pairs: Two graph-sampling augmentations of the same r-ego network form a similar pair, while augmentations from different r-ego networks form dissimilar pairs.The sampling procedure uses random walk with restart, subgraph induction, and anonymization; GCC uses restart probability 0.8 in this work.

4 EXPERIMENTS

GCC is evaluated across three graph learning tasks using diverse pre-training graphs and new downstream datasets. Across these evaluations, pre-trained GCC is competitive with or better than task-specific and from-scratch baselines, while ablations examine pre-training, contrastive mechanisms, momentum, and dataset diversity.

Experimental setup: GCC is evaluated on node classification, graph classification, and similarity search, using six datasets for self-supervised pre-training.The pre-training datasets comprise academic and social graphs.
Node classification: Under freezing, GCC performs competitively on US-Airport, achieves the best H-index performance, and improves further with full fine-tuning.The frozen encoder was not trained on either target dataset, unlike the compared baselines.
Graph classification: GCC (MoCo, freeze) is competitive with InfoGraph on IMDB-B and IMDB-M, best on the other graph-classification datasets, and better than DGCNN while comparable to GIN.GCC uses the same pre-training and fine-tuning hyperparameters across all datasets, whereas GIN is sensitive to hyperparameter choices.
Similarity search: Simply applying pre-trained GCC can be competitive with Panther++ and GraphWave on top-k similarity search without fine-tuning.The task evaluates similarity across conference co-author graphs.
Ablation studies: In all datasets except IMDB-B, GCC outperforms its randomly initialized counterpart, indicating that pre-training generally provides a better starting point for fine-tuning.The authors attribute the IMDB-B exception to domain shift between pre-training and downstream data.
Ablation studies: Adding one pre-training dataset is associated with average gains of 0.43 accuracy (%) on US-Airport and 0.81 accuracy (%) on COLLAB.The reported trend is toward higher accuracy with more pre-training datasets.

5 CONCLUSION

The paper presents GCC for pre-training GNNs across multiple graph datasets to characterize and transfer structural representations. Across three graph-learning tasks and ten datasets, GCC achieves competitive performance to supervised models trained from scratch.

GCC pre-trains graph neural networks from multiple graph datasets to characterize and transfer structural representations.
GCC achieves competitive performance to supervised trained-from-scratch counterparts across three graph learning tasks and ten graph datasets.
The authors plan to benchmark additional graph learning tasks and more diverse datasets, including protein-protein association networks.

A APPENDIX

The appendix documents pre-training hyper-parameters for E2E and MoCo and reports pre-training time for different contrastive loss mechanisms and dictionary sizes.

Table 6 lists pre-training hyper-parameters for E2E and MoCo.
The appendix places optimization settings alongside computational comparisons of contrastive mechanisms.
Figure 6 compares pre-training time across different contrastive loss mechanisms and dictionary sizes K.

A.1.1 Hardware Configuration.

The experiments run on Linux servers equipped with an Intel Xeon CPU, 256GB RAM, and eight NVIDIA 2080Ti GPUs.

The experiments use Linux servers with an Intel Xeon E5-2680 v4 CPU at 2.40GHz and 256GB RAM.
The hardware configuration includes 8 NVIDIA 2080Ti GPUs.
The reported experimental environment combines one specified CPU, large system memory, and eight GPUs.

A.1.2 Software Configuration.

The software configuration uses PyTorch, DGL, CUDA, scikit-learn, and Python, with code and datasets planned for release.

The models are implemented with PyTorch 1.3.1 and DGL 0.4.1.
The environment uses CUDA 10.1, scikit-learn 0.20.3, and Python 3.6.
The authors state that their code and datasets will be available.

A.1.3 Pre-training.

GCC variant training times are reported, with MoCo scaling more economically than E2E as dictionary size increases. The elapsed-time measurements may be affected by other programs on the server.

GCC (E2E) training time grows sharply with dictionary size K, while GCC (MoCo) remains roughly unchanged.This indicates that MoCo is more economical and easier to scale with larger dictionaries.
Figure 6 lists training times for GCC variants.
Elapsed real-time measurements for pre-training may be affected by other programs running on the server.
Table 7 reports GIN performance under various hyper-parameter configurations.

A.2.1 Node Classification.

The node-classification evaluation compares GCC with several graph representation baselines and documents their implementation settings, datasets, and scalability constraints. H-index construction uses smaller subgraphs extracted from the much larger OAG co-authorship graph.

Baseline Implementation: GraphWave embeddings were computed with 64 dimensions after converting datasets to networkx format and automatically selecting the scale range.
Baseline Implementation: Struc2vec failed to finish on H-index within 24 hours despite increasing the worker count to 48.The sampling strategy was observed to take most of the training time.
Datasets: The evaluation also uses US-Airport, H-index, COLLAB, IMDB-BINARY, IMDB-MULTI, REDDITBINARY, and REDDIT-MULTI5K datasets.Paired conference co-author datasets include KDD-ICDM, SIGIR-CIKM, and SIGMOD-ICDE.
Datasets: H-index is a co-authorship graph extracted from OAG, whose millions of nodes make it too large as a node-classification benchmark.
Datasets: The H-index subgraph procedure selects seed vertices, runs breadth-first search until N nodes are visited, and returns the induced subgraph.The procedure sets N = 5,000 and randomly selects 20 seeds from the 200 highest-degree nodes.