PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Graph Contrastive Representation Learning

Yanqiao Zhu, Yichen Xu, Feng Yu, Qiang Liu, Shu Wu, Liang Wang

arXiv:2006.04131v2cs.LGstat.ML

TL;DR

Existing graph neural networks often require abundant labels, while prior unsupervised approaches remain tied to preset graph proximity information. GRACE learns node representations by contrasting two corrupted graph views, and experiments report consistent improvements over existing methods across transductive and inductive tasks.

  • Problem

    Most existing GNN models are supervised and require abundant labeled nodes, while earlier unsupervised methods rely heavily on a preset graph proximity matrix.

  • Method

    GRACE generates two graph views by removing edges and masking node features, then maximizes agreement between corresponding node embeddings with a node-level contrastive loss.

  • Results

    GRACE consistently outperforms existing methods across transductive and inductive tasks and even surpasses supervised counterparts on transductive tasks.

  • Takeaways & Limitations

    The self-supervised framework can alleviate label scarcity and serves as a plug-in for applications such as recommender systems and protein function prediction.

  • Takeaways & Limitations

    DGI's InfoMax implementation requires an injective readout function, while its mean-pooling readout is not guaranteed to preserve distinctive node features.

Abstract

from arXiv · show

Graph representation learning nowadays becomes fundamental in analyzing graph-structured data. Inspired by recent success of contrastive methods, in this paper, we propose a novel framework for unsupervised graph representation learning by leveraging a contrastive objective at the node level. Specifically, we generate two graph views by corruption and learn node representations by maximizing the agreement of node representations in these two views. To provide diverse node contexts for the contrastive objective, we propose a hybrid scheme for generating graph views on both structure and attribute levels. Besides, we provide theoretical justification behind our motivation from two perspectives, mutual information and the classical triplet loss. We perform empirical experiments on both transductive and inductive learning tasks using a variety of real-world datasets. Experimental experiments demonstrate that despite its simplicity, our proposed method consistently outperforms existing state-of-the-art methods by large margins. Moreover, our unsupervised method even surpasses its supervised counterparts on transductive tasks, demonstrating its great potential in real-world applications.

1 Introduction

Existing GNN approaches are largely supervised and depend on abundant labels, while prior unsupervised graph methods have important limitations. GRACE addresses these issues by contrasting node embeddings across two corrupted graph views using topology and attribute perturbations.

  • Most existing GNN models are supervised and require abundant labeled nodes, limiting applicability when labels are unavailable.
  • DGI contrasts node-level embeddings with a global graph embedding, but its injective readout assumption is restrictive and mean pooling may lose distinctive node information.
  • GRACE maximizes agreement between corresponding node embeddings in two graph views without assuming an injective readout function.
  • The two views are generated by removing edges and masking node features, jointly providing diverse structural and attribute contexts.
  • GRACE consistently outperforms existing methods and surpasses supervised counterparts on transductive tasks under a linear evaluation protocol.

2 Related Work

Related work applies contrastive learning to visual and graph representation learning, but graph methods differ in how they construct positive contexts and use graph neural networks. Unsupervised GNN research remains less developed than supervised GNN research.

  • Visual contrastive methods learn representations by contrasting positive and negative samples, often using augmentations, memory banks, or in-batch negatives.
  • Information-maximization objectives can be interpreted as maximizing lower bounds on mutual information, while downstream representation quality may depend on architectural and estimator bias.
  • Traditional unsupervised graph methods use local contrastive patterns, treating nodes in the same random walk as positive co-occurrence pairs.
  • Graph neural networks provide more powerful convolutional encoders, but supervised GNNs require labeled datasets that may be inaccessible in real-world applications.

3 Deep Graph Contrastive Representation Learning

GRACE learns unsupervised node representations by contrasting corresponding nodes across two corrupted graph views, using structure- and attribute-level perturbations. Its objective is connected to mutual-information maximization and triplet loss, while avoiding explicit negative sampling.

  • Preliminaries: GRACE learns low-dimensional node embeddings with a GNN encoder from graph features and structure without given node class information.The learned representations can support downstream tasks such as node classification.
  • Contrastive framework: GRACE encodes two randomly corrupted graph views and maximizes agreement between each node’s cross-view embeddings while distinguishing other nodes.The framework uses a GNN encoder to produce embeddings U and V, then optimizes a symmetric contrastive objective.
  • Contrastive framework: The contrastive objective treats corresponding node embeddings as positive pairs and all other nodes across both views as intra-view or inter-view negatives.Negative samples are obtained naturally from the two views rather than generated explicitly.
  • Theoretical justification: GRACE avoids explicit negative-sample generation because all other nodes in the two views serve as negatives and computations can be performed in parallel.The paper contrasts this optimization with the computational expense associated with triplet loss.
  • Graph view generation: GRACE generates diverse node contexts by corrupting graph topology through edge removal and node attributes through feature masking.The paper identifies view generation as a key component and allows alternative corruption mechanisms.
  • Graph view generation: Unlike Dropout and DropEdge, GRACE uses edge removal and feature masking to create contrastive graph views rather than primarily prevent over-fitting or over-smoothing.The encoder is typically shallow, using only two or three layers.
  • Theoretical justification: Maximizing GRACE’s objective corresponds to maximizing a lower bound on mutual information between input node features and learned representations.The paper also notes that downstream performance depends on encoder design, not solely on optimizing a stricter mutual-information bound.
  • Theoretical justification: Under identity projection, inner-product similarity, and strongly aligned positives, the pairwise objective coincides with maximizing a classical triplet loss.This view emphasizes separating positive samples from negative samples in embedding space.

4 Experiments

The experiments evaluate GRACE for unsupervised node classification across six public datasets and transductive and inductive settings. GRACE consistently outperforms unsupervised baselines and is competitive with supervised models in several settings.

  • Experimental settings: Experiments evaluate node embeddings on six public datasets spanning transductive citation networks, inductive Reddit, and inductive PPI graphs.The evaluation covers both large-scale and multiple-graph inductive settings.
  • Evaluation protocol: Models are trained unsupervisedly, then evaluated with an ℓ2-regularized logistic regression classifier using accuracy for transductive tasks and micro-averaged F1-score for inductive tasks.Each experiment averages results over twenty runs.
  • Results: GRACE consistently outperforms unsupervised baselines by considerable margins across transductive and inductive tasks.The reported comparison covers all six datasets.
  • Results: GRACE is competitive with supervised models on all four transductive datasets and the inductive Reddit dataset.This comparison is reported separately from the broader unsupervised-baseline comparison.
  • Results: On PPI, GRACE improves considerably over DGI, where extreme node-feature sparsity makes feature shuffling ineffective for contrastive discrimination.GRACE’s edge-removing scheme does not rely on node features and is presented as a remedy in this setting.
  • Analysis: Ablations support corruption at both topology and feature levels, while sensitivity analyses find stability under mild perturbations of the view-generation parameters.The paper also reports efficacy for its contrastive-loss design choice.

5 Conclusion

The paper concludes that GRACE learns node representations by contrasting two corrupted graph views at the node level. Across transductive and inductive experiments, it reports large-margin gains over state-of-the-art methods and superiority to supervised counterparts on transductive tasks.

  • 5 Conclusion: GRACE maximizes agreement between embeddings of the same node across two graph views generated by edge removal and feature masking.The framework is designed for unsupervised graph representation learning.
  • 5 Conclusion: Theoretical analysis connects the contrastive objective to mutual information maximization and the classical triplet loss.These connections are presented as justification for the framework’s motivation.
  • 5 Conclusion: Across varied real-world datasets and transductive and inductive settings, GRACE consistently outperforms existing state-of-the-art methods by large margins.The conclusion summarizes the reported empirical results across both task types.
  • 5 Conclusion: GRACE surpasses supervised counterparts on transductive tasks.This is the paper’s stated comparison with supervised methods.

Discussions on Broader Impact

The authors present GRACE as a self-supervised technique that can reduce reliance on labeled data and serve as a plug-in for existing applications. They also caution that biased data can still produce biased outputs.

  • Benefits: GRACE is intended to alleviate label scarcity and reduce human annotation effort in real-world machine-learning applications.The stated impact concerns self-supervised graph representation learning techniques generally.
  • Applications: The framework can be plugged into recommender systems to produce embeddings for users and commodities and address cold-start problems.This is given as an example application.
  • Applications: The authors report significant gains on protein function prediction and describe potential relevance to drug discovery and treatment.The passage frames this as potential rather than an established deployment outcome.
  • Risks: Although the method is described as a plug-in without new ethical concerns, biased data collection and graph construction may still yield biased outputs.Examples include gender and ethnicity bias.

A Dataset Details

The experiments use citation, social, and biological graph datasets for transductive and inductive node-classification tasks. Dataset construction and evaluation visibility differ between the transductive and inductive settings.

  • Transductive datasets: Cora, Citeseer, Pubmed, and DBLP are citation networks whose nodes are articles, edges are citation links, and features are sparse bag-of-words vectors.The transductive task predicts article subject categories.
  • Inductive datasets: Reddit is a large-scale social graph of posts, with edges connecting posts commented on by the same user and features derived from text using GloVe embeddings.The inductive task predicts subreddit community structures.
  • Inductive datasets: PPI contains multiple human-tissue protein-interaction graphs with multilabel gene-ontology annotations for predicting protein cellular functions.The dataset evaluates generalization across multiple graphs.
  • Task settings: During transductive training, all node features are visible while labels are masked; inductive evaluation uses unseen or untrained nodes and graphs.These visibility conditions define the distinction between the two task settings.

B Implementation

The implementation specifies the computing environment, optimization settings, and experiment hyperparameters used for GRACE.

  • Implementation: All models use PyTorch Geometric 1.5.0, PyTorch 1.4.0, and scikit-learn logistic regression with ℓ2 regularization.Experiments run on eight NVIDIA Titan Xp GPUs and fourteen Intel Xeon E5-2660 v4 CPUs.
  • Hyperparameters: Models use Glorot initialization and Adam optimization, with dataset-specific learning rates and a uniform ℓ2 weight decay factor of 10^-5.The initial learning rate is 0.001 except for Cora, 0.0005, and Reddit, 10^-5.
  • Hyperparameters: Training lasts 200, 200, 1500, 1000, 40, and 200 epochs for Cora, Citeseer, Pubmed, DBLP, Reddit, and PPI, respectively.

C.1 Sensitivity Analysis

GRACE remains relatively stable across moderate graph-view corruption probabilities, but excessive corruption undermines the original graph and harms classification.

  • Setup: The two graph views are controlled by p1 = pr,1 = pm,1 and p2 = pr,2 = pm,2, while other parameters remain unchanged.
  • Results: GRACE’s Citeseer Micro-F1 remains relatively stable across moderate values of the view-generation probabilities.The sensitivity analysis varies four probabilities from 0.1 to 0.9 and visualizes them through p1 and p2.
  • Results: The authors conclude that GRACE is generally insensitive to these probabilities under non-extreme settings, indicating robustness to hyperparameter tuning.
  • Boundary: When edge-removal probability pr reaches 0.9, almost every existing edge is removed, producing isolated nodes and heavily undermining the original graph.

C.2 Ablation Studies

Ablation studies test GRACE’s hybrid graph-view corruption and its contrastive objective, showing benefits from combining edge removal with feature masking and from retaining the original objective.

  • Ablation Studies: The hybrid GRACE model significantly outperforms variants using only edge removal or only feature masking.The ablations compare GRACE, GRACE (–RE), and GRACE (–MF) under otherwise identical transductive node-classification settings.
  • Ablation Studies: These results support jointly corrupting graph topology and node attributes when generating contrastive graph views.
  • Comparison with InfoNCE Loss: GRACE-NCE performs worse than the original GRACE model on all four citation datasets in transductive node classification.The comparison uses the InfoNCE objective in the GRACE-NCE variant and is summarized in Table 6.
  • Comparison with InfoNCE Loss: Although InfoNCE is described as a stricter mutual-information estimator, the paper reports that its GRACE variant has inferior downstream performance.

C.4 Robustness to Sparse Features

GRACE remains stronger than DGI under feature masking across four citation networks, while severe masking degrades both methods.

  • Results: GRACE consistently outperforms DGI by large margins across different feature-contamination rates on four citation networks.The experiments mask 50% to 90% of node features and evaluate transductive node-classification accuracy.
  • Interpretation: The paper attributes GRACE’s sparse-feature robustness to edge removal, which constructs different topology contexts without depending on node features.
  • Boundary: At 90% feature masking, both GRACE and DGI perform poorly because overly sparse node features hinder the GNN from extracting useful information.
  • Interpretation: The results support considering graph corruption at both topology and attribute levels.

D.1 Proof of Theorem 1

The proof connects the objective J to InfoNCE and uses the data processing inequality to establish its mutual-information lower-bound interpretation. A separate theorem relates the pairwise objective to triplet loss under identity projection, inner-product similarity, and strong positive-pair alignment.

  • Theorem 1 proof: The objective J is connected to the InfoNCE objective through a critic function based on projected embedding similarity.The critic is defined as θ(x, y) = s(g(x), g(y)), with auxiliary quantities ρr and ρc used to rewrite J.
  • Theorem 1 proof: The symmetric InfoNCE estimator bounds the objective through the relation 2J ≤ I(U; V) + I(V; U) = 2I(U; V).The proof invokes the fact that InfoNCE is a lower bound of true mutual information.
  • Theorem 1 proof: The data processing inequality applies because the two view embeddings are conditionally independent given X, yielding I(U; V) ≤ I(U; X).The proof identifies the Markov relation U ← X → V and notes that X → (U, V) → U also holds.
  • Theorem 2: Under identity projection, inner-product similarity, and much stronger positive-pair alignment, minimizing the pairwise objective coincides with maximizing triplet loss.The proof rearranges the pairwise objective under these assumptions and applies a first-order Taylor expansion.
Loading 2006.04131v2…