Source-linked AI summary
Graph Contrastive Learning with Adaptive Augmentation
Yanqiao Zhu, Yichen Xu, Feng Yu, Qiang Liu, Shu Wu, Liang Wang
TL;DR
Graph contrastive learning depends on graph augmentations, but existing schemes often apply uniform perturbations despite differences in node and edge importance. GCA uses centrality-guided topology and attribute augmentations to generate contrasting views, and its experiments report consistent gains over existing methods, including several supervised counterparts.
Problem
Existing graph contrastive methods rarely explore augmentation design, and uniform perturbations can ignore differences in the importance of nodes, edges, and feature dimensions.
Method
GCA generates two correlated graph views using adaptive edge removal and feature masking guided by centrality measures, then maximizes agreement between their node embeddings.
Results
GCA consistently outperforms existing state-of-the-art methods and surpasses several supervised counterparts on node-classification evaluations.
Takeaways & Limitations
Adaptive augmentation preserves important connective and semantic patterns while supporting unsupervised graph representation learning under limited labels.
Takeaways & Limitations
The relationship between specific augmentation functions and the contrastive objective’s lower bound remains difficult to derive.
Abstract
from arXiv · showhide
Recently, contrastive learning (CL) has emerged as a successful method for unsupervised graph representation learning. Most graph CL methods first perform stochastic augmentation on the input graph to obtain two graph views and maximize the agreement of representations in the two views. Despite the prosperous development of graph CL methods, the design of graph augmentation schemes -- a crucial component in CL -- remains rarely explored. We argue that the data augmentation schemes should preserve intrinsic structures and attributes of graphs, which will force the model to learn representations that are insensitive to perturbation on unimportant nodes and edges. However, most existing methods adopt uniform data augmentation schemes, like uniformly dropping edges and uniformly shuffling features, leading to suboptimal performance. In this paper, we propose a novel graph contrastive representation learning method with adaptive augmentation that incorporates various priors for topological and semantic aspects of the graph. Specifically, on the topology level, we design augmentation schemes based on node centrality measures to highlight important connective structures. On the node attribute level, we corrupt node features by adding more noise to unimportant node features, to enforce the model to recognize underlying semantic information. We perform extensive experiments of node classification on a variety of real-world datasets. Experimental results demonstrate that our proposed method consistently outperforms existing state-of-the-art baselines and even surpasses some supervised counterparts, which validates the effectiveness of the proposed contrastive framework with adaptive augmentation.
1 INTRODUCTION
GCA addresses limitations of uniform graph augmentations by adapting topology and attribute perturbations to preserve important graph patterns. It learns node representations by contrasting two adaptively augmented views and reports strong node-classification performance.
- Motivation: Existing graph contrastive methods rarely explore augmentation design despite its critical role in contrastive learning.Graph augmentations are difficult to define because graphs are non-Euclidean.
- Motivation: Uniformly dropping edges can remove influential connections, while simple feature transformations may fail to generate diverse node contexts.These issues are especially problematic when node features are sparse.
- GCA Framework: GCA generates two correlated graph views using stochastic augmentation adaptive to graph structure and attributes.The views are encoded with a shared GNN and trained using a contrastive objective that maximizes agreement between node embeddings.
- Adaptive Augmentation: Topology augmentation assigns higher edge-removal probabilities to unimportant edges, preserving important connective structures.Centrality measures identify important edges and feature dimensions.
- Adaptive Augmentation: Attribute augmentation adds more noise to unimportant feature dimensions to encourage recognition of underlying semantic information.GCA jointly augments topology and node attributes to provide diverse contexts across views.
- Evaluation: Experiments use five public benchmark datasets under linear evaluation, with GCA consistently outperforming existing methods and surpassing supervised counterparts on several transductive tasks.The code is publicly available for reproducibility.
2 RELATED WORK
Prior graph representation learning methods use local contrastive objectives, mutual-information objectives, or graph-view construction strategies, while GCA uniquely adapts augmentation at both topology and attribute levels. The related-work comparison positions GCA as a node-level contrastive method with joint adaptive augmentation.
- Unsupervised Graph Representation Learning: Traditional unsupervised graph representation methods commonly contrast nodes that co-occur in random walks or other local contexts.DeepWalk and node2vec treat nodes appearing in the same random walk as positive samples.
- Graph Neural Networks: Supervised GNNs are prominent but require labeled datasets, whereas unsupervised GNN methods have received less attention.Representative unsupervised approaches include GraphSAGE, DGI, GMI, MVGRL, and GCC.
- Graph Contrastive Methods: DGI shuffles node features for augmentation, GMI measures mutual information without explicit augmentation, and MVGRL uses node diffusion with graph-summary contrasts.These methods differ in both objective design and graph-view construction.
- Comparison with Related Work: GCA’s adaptive perturbations are designed to preserve important patterns underneath the graph through stochastic perturbation.The comparison focuses on topology- and attribute-level augmentation strategies.
- Comparison with Related Work: GCA is the only compared method that proposes adaptive data augmentation on both topology and attribute levels.It also simplifies prior node–global contrastive schemes by defining the contrastive objective at the node level.
3 THE PROPOSED METHOD
GCA learns node representations by contrasting two stochastically corrupted graph views, using adaptive topology- and attribute-level augmentation to preserve important graph information. Its objective has a mutual-information interpretation, and the framework emphasizes influential structures and features during representation learning.
- Contrastive learning framework: GCA samples two stochastic augmentation functions, generates two graph views, encodes them with a GNN, and maximizes a node-level contrastive objective.The same node across views forms a positive pair, while embeddings of other nodes provide inter-view and intra-view negatives.
- Adaptive augmentation: The framework jointly removes edges and masks node features, assigning higher corruption probabilities to unimportant links or features.Important structures and attributes are more likely to remain unchanged, guiding representations toward fundamental topological and semantic patterns.
- Topology-level augmentation: Edge-removal probabilities reflect edge importance, with edge centrality computed from the centrality of its connected nodes and transformed logarithmically before normalization.A cutoff probability limits excessively large removal probabilities to avoid overly corrupted graph structures.
- Theoretical justification: The contrastive objective is a lower bound on mutual information between input node features and node representations from the two views.Theorem 1 provides the formal lower-bound result, while the accompanying remark connects the objective to InfoNCE-style mutual-information maximization.
- Theoretical justification: Contrasting views that preserve important links and attributes encourages embeddings to encode essential structural and semantic information.The objective is also interpreted as encouraging positive samples to separate from negative samples without explicitly generating negatives.
4 EXPERIMENTS
The experiments evaluate GCA on transductive node classification using multiple real-world datasets and linear evaluation. GCA consistently outperforms unsupervised baselines, while adaptive augmentation improves performance across topology and node-attribute levels.
- Experimental setup: Experiments evaluate GCA on six real-world datasets for transductive node classification using linear evaluation and accuracy averaged over twenty runs.The setup uses unsupervised representation learning followed by an ℓ2-regularized logistic regression classifier.
- RQ1: Performance on node classification: GCA consistently outperforms unsupervised baselines across the evaluated datasets and remains competitive with supervised models.The comparison includes traditional methods, graph autoencoders, and several graph contrastive learning baselines.
- RQ1: Performance on node classification: All three centrality-based GCA variants outperform existing contrastive baselines on all datasets, indicating that the framework is not limited to one centrality choice.The variants use degree, eigenvector, and PageRank centrality.
- RQ2: Ablation studies: Adaptive topology and node-attribute augmentation each improve performance consistently, while combining both provides further gains.The ablation replaces adaptive schemes with uniform counterparts to isolate the contributions of each augmentation level.
- RQ2: Ablation studies: On Amazon-Computers, GCA achieves a 1.5% absolute improvement over the base model without adaptive augmentation.This result is reported for the combined adaptive augmentation framework.
- RQ3: Hyperparameter sensitivity: Node-classification accuracy remains relatively stable for moderate hyperparameter probabilities but declines when probabilities become too large, such as p_e = 0.9.At p_e = 0.9, nearly every existing edge is removed, producing isolated nodes.
5 CONCLUSION
The paper presents GCA, an unsupervised graph contrastive framework that generates views through adaptive augmentation. It reports consistent improvements over state-of-the-art methods and some supervised counterparts across real-world datasets.
- Conclusion: GCA maximizes agreement between node embeddings from views generated by adaptive graph augmentation.The framework identifies important edges and feature dimensions before perturbing the graph.
- Conclusion: Topology augmentation assigns larger removal probabilities to unimportant edges to preserve network connectivity patterns.The scheme uses network centrality measures to identify important structure.
- Conclusion: Attribute augmentation adds more noise to unimportant feature dimensions to emphasize underlying semantic information.This is the node-attribute component of the adaptive augmentation scheme.
- Conclusion: Across various real-world datasets, GCA consistently outperforms existing state-of-the-art methods and surpasses several supervised counterparts.The reported evaluation uses comprehensive experiments.
DISCUSSIONS ON BROADER IMPACT
GCA is presented as a plug-in self-supervised technique that may alleviate label scarcity and support high-quality embeddings, while biased data can still produce biased outputs.
- GCA is intended to alleviate label scarcity in real-world machine-learning deployments by reducing reliance on human annotation.The paper also suggests applications in recommender systems, including user and item embeddings for cold-start problems.
- The framework can be plugged into recommender systems to produce high-quality embeddings for users and items addressing the cold-start problem.
- GCA itself is described as not introducing new ethical concerns because it functions as a plug-in for existing machine-learning models.
- Biased outputs remain possible when the underlying data are biased during collection, graph construction, or related processes.
A.1 Computing Infrastructures
The experiments use implementations based on PyTorch Geometric, PyTorch, and NetworkX, with datasets available through PyTorch Geometric, and run on a specified multi-GPU CPU server.
- The models use PyTorch Geometric 1.6.1, PyTorch 1.6.0, and NetworkX 2.5.
- All experimental datasets are available in the PyTorch Geometric libraries.
- The experiments run on four NVIDIA Tesla V100S GPUs with 32GB memory each and twelve Intel Xeon Silver 4214 CPUs.
A.2 Hyperparameter Specifications
Training uses Glorot initialization and Adam SGD, while augmentation probabilities are restricted to 0.0–0.4 and differ between the two views to avoid excessive corruption and create distinct contexts.
- Model parameters use Glorot initialization and are trained with Adam SGD, with ℓ2 weight decay 10^-5 and zero dropout.
- The edge- and feature-sampling probabilities for both views are selected between 0.0 and 0.4 because larger values overly corrupt the original graph.
- The probabilities p_e,1 and p_e,2 should differ, and the same holds for the corresponding feature probabilities, to generate different node contexts across views.
- Table 5 is identified as specifying the hyperparameters used by the experiments.
B.1 Proof of Theorem 1
The proof connects GCA’s objective to InfoNCE and mutual information, then uses Markov relations and the data processing inequality to establish the stated lower-bound relationship.
- Theorem 1 defines neighborhood inputs for node embeddings and states that objective J lower-bounds mutual information involving the encoder input and representations from two graph views.
- The proof begins by establishing a connection between objective J and the InfoNCE objective.
- The critic function compares transformed representations through θ(x,y) = s(g(x),g(y)), with auxiliary sums defined for the proof’s notation.
- The proof rewrites objective J and the InfoNCE estimator using the auxiliary notation ρ_c and related terms.
- InfoNCE is used as a lower bound on true mutual information, after which data processing inequalities are applied to the variables X, U, and V.
- The proof combines the resulting inequalities to reach the theorem’s final inequality and conclusion.
B.2 Proof of Theorem 2
Under the stated alignment and similarity assumptions, the proof connects minimizing the pairwise objective with maximizing triplet loss through algebraic rearrangement and a first-order Taylor expansion.
- The theorem assumes an identity projection function, inner-product embedding similarity, and substantially greater alignment for positive than negative pairs.
- Under these assumptions, minimizing the pairwise objective for each pair coincides with maximizing the triplet loss.
- The proof proceeds by rearranging the pairwise objective under the theorem’s assumptions.
- A first-order Taylor expansion is then used in the derivation.
- Equation (25) completes the proof of the theorem.