PaperlayerView exact revision on arXiv

Source-linked AI summary

Contrastive Multi-View Representation Learning on Graphs

Kaveh Hassani, Amir Hosein Khasahmadi

arXiv:2006.05582v1cs.LGstat.ML

TL;DR

Graph representation learning often depends on costly task-specific labels, motivating self-supervised methods that can learn from graph structure alone. The paper contrasts node and graph representations across two structural views—first-order neighbors and a graph diffusion—and finds that this design achieves new self-supervised state-of-the-art results on all eight evaluated benchmarks. It also reports performance on par with or better than strong supervised baselines on four of eight benchmarks.

  • Problem

    Graph neural networks often require task-dependent labels, but graph annotation is challenging and costly in specialized domains.

  • Method

    The method learns node and graph representations by maximizing mutual information between cross-view node and graph encodings from first-order neighbors and a graph diffusion.

  • Results

    New self-supervised state-of-the-art results are achieved on 8 out of 8 node and graph classification benchmarks under linear evaluation, with performance on par with or better than supervised baselines on 4 out of 8.

  • Takeaways & Limitations

    Increasing views beyond two and contrasting multi-scale encodings do not improve performance, while first-order-neighbor and graph-diffusion views provide the strongest reported design.

Abstract

from arXiv · show

We introduce a self-supervised approach for learning node and graph level representations by contrasting structural views of graphs. We show that unlike visual representation learning, increasing the number of views to more than two or contrasting multi-scale encodings do not improve performance, and the best performance is achieved by contrasting encodings from first-order neighbors and a graph diffusion. We achieve new state-of-the-art results in self-supervised learning on 8 out of 8 node and graph classification benchmarks under the linear evaluation protocol. For example, on Cora (node) and Reddit-Binary (graph) classification benchmarks, we achieve 86.8% and 84.5% accuracy, which are 5.5% and 2.4% relative improvements over previous state-of-the-art. When compared to supervised baselines, our approach outperforms them in 4 out of 8 benchmarks. Source code is released at: https://github.com/kavehhassani/mvgrl

1. Introduction

The paper addresses the difficulty of learning useful graph representations without costly task-dependent labels by introducing self-supervised contrastive learning across structural graph views. Its systematic findings identify effective view and encoding choices, yielding state-of-the-art self-supervised results across node and graph classification benchmarks.

  • Motivation: Graph labeling is challenging and costly because graphs often represent specialized-domain concepts requiring domain knowledge.This motivates unsupervised representation learning for transfer to downstream tasks.
  • Contribution: The approach trains graph encoders by maximizing mutual information between representations encoded from different structural views without specialized architectures.The paper applies contrastive learning to graph data by adapting the multi-view idea from visual representation learning.
  • Design findings: More than two views do not improve performance, and the best results come from contrasting first-order neighbors with a general graph diffusion.These findings differ from the pattern reported for visual contrastive learning.
  • Design findings: Cross-view node-graph contrasting outperforms graph-graph and multi-scale encodings on both node and graph classification tasks.A simple graph readout also outperforms hierarchical pooling, while normalization and regularization generally degrade downstream performance except early stopping.
  • Results: 8 out of 8 node and graph classification benchmarks achieve new self-supervised state-of-the-art results under linear evaluation.Representative results are 86.8% accuracy on Cora, a 5.5% relative improvement, and 84.5% on Reddit-Binary, a 2.4% relative improvement.
  • Results: The method performs on par with or better than strong supervised baselines on 4 out of 8 benchmarks.The paper reports relative improvements over GAT of 4.5% on Cora and 5.3% on IMDB-Binary.

2. Related Work

Related work includes graph representation methods based on random walks, graph kernels, contrastive learning, graph diffusion, and the InfoMax principle. The paper builds on these lines by combining structural graph views with contrastive node-graph representation learning.

  • Existing graph representation methods: Random-walk methods flatten graphs into sequences but over-emphasize proximity, remain limited to transductive settings, and cannot use node features.These limitations distinguish them from graph encoders that operate directly on graph structure and features.
  • Contrastive methods: Contrastive graph methods measure latent-space differences between samples with and without dependencies of interest and currently lead unsupervised node and graph classification.Deep Graph Infomax learns node representations by contrasting node and graph encodings.
  • Graph diffusion: Graph diffusion networks combine spatial message passing with generalized diffusion so messages can pass through higher-order neighborhoods.Graph diffusion convolution replaces the adjacency matrix in graph convolution with a sparsified diffusion matrix.
  • InfoMax foundations: The InfoMax principle motivates learning representations that maximize mutual information between inputs and encoded representations.Self-supervised models estimate lower bounds of this objective by contrasting representations.
  • Positioning of the proposed approach: The proposed model contrasts node representations from one structural view with graph representations from another view, in both directions.Figure 1 depicts graph diffusion and regular views processed by dedicated GNNs, followed by shared MLPs and a discriminator.

3. Method

The method learns node and graph representations by contrasting two structural graph views, typically adjacency and diffusion, using cross-view node–graph mutual-information maximization. Dedicated encoders, readout functions, projection heads, and a discriminator produce representations for downstream tasks.

  • Structural views: The framework transforms each graph’s structure into correlated views, samples identical nodes across views, and leaves initial node features unchanged.The sampler selects corresponding nodes and edges from both views, supporting inductive and transductive settings.
  • Structural views: Adjacency and diffusion matrices serve as congruent structural views that provide local and global graph information, respectively.The authors report that this pairing generally achieves the best results among the evaluated graph augmentations.
  • Encoders: Two dedicated GNN encoders and shared projection heads learn node representations for the two views, using GCNs as the base architecture.The encoders process node features with either normalized adjacency or diffusion structure, producing two view-specific node-representation sets.
  • Encoders: A graph readout aggregates pre-projection node representations from each view into graph representations, with a JK-Net-like function concatenating layer-wise sums.The resulting graph representations are passed through a shared projection head; this readout outperforms DiffPool in the reported comparison.
  • Training: At inference, representations from both structural views are summed at node and graph levels and returned for downstream tasks.This combines the learned view-specific representations after training.
  • Training: Training maximizes mutual information by contrasting node representations from one view with graph representations from the other, in both directions.A discriminator scores cross-view agreement, and the objective uses positive joint-distribution samples and negative product-of-marginals samples.

4.1. Benchmarks

The evaluation uses eight widely used benchmarks: three node-classification citation networks and five graph-classification datasets spanning molecular, social, and image domains.

  • Dataset composition: The benchmark suite contains three node-classification and five graph-classification datasets widely used in prior work.Node classification uses Citeseer, Cora, and Pubmed citation networks, where documents are nodes and citations are edges.

4.2. Evaluation Protocol

The evaluation uses established node and graph classification benchmarks with linear evaluation protocols aligned to prior state-of-the-art studies. Benchmark statistics and task-specific evaluation setups define the comparison basis.

  • Node classification: Node classification reports mean test-node accuracy with standard deviation after 50 runs followed by a linear model.The protocol follows DGI.
  • Graph classification: Graph classification reports mean 10-fold cross-validation accuracy with standard deviation after five runs followed by a linear SVM.The protocol follows InfoGraph.
  • Benchmark reporting: The experiments use benchmark statistics to characterize the datasets and task settings.The supplied table captions identify Table 1 as reporting classification-benchmark statistics.

4.3. Comparison with State-of-the-Art

The proposed approach achieves state-of-the-art results across unsupervised node and graph classification comparisons, while also surpassing several supervised baselines. Its unified design reaches these results without task-specific encoders.

  • Node classification: 86.8% Cora accuracy represents a 5.5% relative improvement over the previous state-of-the-art in unsupervised node classification.The comparison is made under the linear evaluation protocol.
  • Node clustering: The model achieves state-of-the-art results across node clustering benchmarks using NMI and ARI scores.The comparison includes VGAE, MGAE, ARGA, ARVGA, and GALA.
  • Graph classification: 84.5% Reddit-Binary accuracy is a 2.4% relative improvement over the previous state-of-the-art in unsupervised graph classification.The approach also outperforms kernel methods in 4 out of 5 datasets.
  • Supervised comparisons: The model outperforms GCN and GAT in 3 out of 5 graph-classification datasets, including a 5.3% relative improvement over GAT on IMDB-Binary.It also outperforms the best supervised model in one dataset.
  • Unified design: State-of-the-art results on both node and graph classification are obtained with one unified approach rather than specialized encoders for each task.This distinguishes the approach from previous unsupervised models cited by the authors.

4.4. Ablation Study

The ablation study examines mutual-information estimators, contrastive modes, structural views, view counts, batch size, negative sampling, and regularization. It finds that local-global contrast across two complementary structural views is most effective, while additional views and several training modifications hurt or fail to improve performance.

  • MI estimators: Jensen-Shannon estimation performs best across graph classification benchmarks, while NCE performs best on 2 out of 3 node classification datasets.The study compares NCE, JSD, NT-Xent, and DV estimators.
  • Contrastive mode: Contrasting node and graph encodings consistently outperforms alternative contrastive modes across benchmarks.The evaluated modes include local-global, global-global, multi-scale, hybrid, and ensemble.
  • Cross-view comparisons: For graphs, node-graph contrast outperforms global-view contrast, whereas multi-scale contrast has a negative effect on graph representation learning.These findings differ from reported patterns in visual representation learning.
  • Structural views: The framework contrasts adjacency-based local information with global structural information from diffusion or related graph views.The investigated views include adjacency, PPR, heat diffusion, and pair-wise distance matrices, computed during preprocessing.
  • Number of views: Adding a third structural view does not improve downstream performance compared with two views.The three-view setting anchors adjacency and adds two diffusion matrices, which may carry similar information.
  • Batch size: Increasing batch size slightly improves graph classification but has no significant effect on node classification.The authors therefore choose efficient smaller batch sizes.
  • Regularization: Normalization layers and regularization methods degrade downstream performance during pre-training, except for early stopping.The tested methods include BatchNorm, Layer-Norm, Gaussian noise, L2 regularization, and dropout.

5. Conclusion

The paper presents a self-supervised method that contrasts two structural graph views and reports state-of-the-art self-supervised results across all eight node and graph classification benchmarks.

  • The approach learns node- and graph-level representations by contrasting encodings from first-order neighbors and a graph diffusion.
  • Unlike visual representation learning, increasing the number of views or contrasting multi-scale encodings does not improve performance.
  • The method achieves new state-of-the-art self-supervised results on 8 out of 8 node and graph classification benchmarks under linear evaluation.
  • The approach outperforms strong supervised baselines on 4 out of 8 benchmarks.

6. Appendix

Appendix experiments examine implementation choices and ablations involving views, encoders, pooling, batch size, and hyper-parameter selection. Dedicated encoders and sum pooling perform best across benchmarks, while adding views or larger batches provides limited gains.

  • Experimental settings: Hyper-parameters are selected by grid search, with fixed diffusion settings α=0.2 for PPR and t=5 for heat diffusion.The search covers GCN layers, epochs, batch size, and the SVM C parameter for graph classification, with separate settings for node classification.
  • Number of views: Adding a third view with PPR and heat diffusion does not improve downstream performance.The three-view setup anchors the main view on the adjacency matrix and uses two diffusion matrices as additional views.
  • Encoder ablation: Dedicated encoders for each view consistently outperform shared encoders across all benchmarks.Randomly sampling two of four views with a shared encoder degraded performance.
  • Batch size: Larger batch sizes slightly improve graph classification but have negligible effect on node classification.The batch-size experiments treat batch size as the number of negative samples in contrastive learning.
  • Pooling and encoders: Sum pooling and dedicated encoders achieve better results across node and graph classification benchmarks.Table 6 summarizes the comparison with mean pooling and two DiffPool variants.
Loading 2006.05582v1…