Source-linked AI summary
Augmentation-Free Self-Supervised Learning on Graphs
Namkyeong Lee, Junseok Lee, Chanyoung Park
TL;DR
Graph augmentations can arbitrarily change semantics, making augmentation-based representation quality dependent on carefully tuned hyperparameters. AFGRL avoids augmentations and negative samples by discovering positives through local structure and global semantics, and experiments show strong performance across node-level tasks with hyperparameter stability.
Problem
Graph augmentations may change underlying semantics, leaving augmentation-based graph representation quality highly dependent on augmentation-scheme hyperparameters.
Method
AFGRL generates alternative views by discovering positive nodes using local structural information and global graph semantics, with BYOL eliminating negative samples.
Results
AFGRL outperforms a wide range of state-of-the-art methods on node classification, clustering, and similarity search, while remaining stable over hyperparameters.
Takeaways & Limitations
AFGRL provides a practical graph self-supervised framework that avoids manual augmentation techniques and negative samples.
Abstract
from arXiv · showhide
Inspired by the recent success of self-supervised methods applied on images, self-supervised learning on graph structured data has seen rapid growth especially centered on augmentation-based contrastive methods. However, we argue that without carefully designed augmentation techniques, augmentations on graphs may behave arbitrarily in that the underlying semantics of graphs can drastically change. As a consequence, the performance of existing augmentation-based methods is highly dependent on the choice of augmentation scheme, i.e., hyperparameters associated with augmentations. In this paper, we propose a novel augmentation-free self-supervised learning framework for graphs, named AFGRL. Specifically, we generate an alternative view of a graph by discovering nodes that share the local structural information and the global semantics with the graph. Extensive experiments towards various node-level tasks, i.e., node classification, clustering, and similarity search on various real-world datasets demonstrate the superiority of AFGRL. The source code for AFGRL is available at https://github.com/Namkyeong/AFGRL.
1 Introduction
Self-supervised learning has expanded to graphs, but graph augmentations can alter semantics unpredictably and make performance sensitive to augmentation hyperparameters. AFGRL addresses this by discovering positive samples using local structure and global semantics, without negative samples.
- Self-supervised learning derives supervision signals from data itself and has succeeded across computer vision, signal processing, and natural language processing.
- Graph augmentations such as edge, node, or feature perturbations may change the semantic relationship between an augmented graph and the original.
- Augmentation-based graph representations depend strongly on augmentation schemes and require dataset- and task-specific hyperparameter tuning.
- AFGRL filters k-NN samples as positives when they are adjacent to the target node or belong to its cluster, capturing local structure and global semantics.
- BYOL supplies AFGRL’s backbone, eliminating negative samples and avoiding sampling bias from negatives that may share the query node’s semantics.
- Experiments report that AFGRL outperforms state-of-the-art methods on node classification, clustering, and similarity search while remaining stable over hyperparameters.
2 Related Work
Graph representation learning has adopted contrastive methods and diverse augmentations, while newer approaches address negative-sampling bias. BGRL removes negative samples but still relies on graph augmentations that may cause semantic drift.
- DGI learns node representations by maximizing mutual information between local graph patches and a global graph summary.
- BGRL avoids negative samples by comparing representations from two augmented graph views with online and exponentially averaged target encoders.
- BGRL still depends on graph augmentations, which may lead to semantic drift.
- Graph augmentation techniques include node dropping, edge modification, subgraph extraction, and attribute masking.
3 Problem Statement
The task is to learn compact node embeddings from a graph’s features and adjacency matrix that generalize to downstream tasks without class labels.
- A graph G consists of nodes V and edges E, with feature matrix X and adjacency matrix A encoding node relationships.
- Unsupervised graph representation learning seeks an encoder that maps X and A to D-dimensional node embeddings, where D is much smaller than the feature dimension F.
- The learned embeddings should generalize across downstream tasks without using class information.
4 Preliminary: Bootstrap Your Own Latent
BYOL learns representations by matching outputs from online and target encoders on two views, using an asymmetric predictor and an EMA-updated target network. Its objective minimizes normalized representation discrepancy without negative samples.
- BYOL processes two transformed views through separate online and target encoders and maximizes similarity between their representations.
- Projectors map encoder outputs to smaller representations, and a predictor on the online branch makes the architecture asymmetric.
- The BYOL loss is the squared distance between normalized predicted online and target representations, with a symmetric loss from swapped views.
- Training minimizes the BYOL objective with respect to the online parameters while updating target parameters through an exponential moving average.
- The target-network decay rate controls how closely its parameters remain to the online network.
5 Proposed Method
AFGRL generates augmentation-free alternative graph views by identifying positives that jointly reflect local structure and global semantics. Its BYOL-based training filters noisy nearest-neighbor candidates without negative samples, while experiments and sensitivity analyses support stable performance.
- Augmentation-Free GRL (AFGRL): AFGRL uses online and target encoders to compute node embeddings and discover k-nearest-neighbor positive candidates without manual graph augmentation.The online encoder is optimized by gradient descent, while the target encoder is updated through exponential moving average.
- Augmentation-Free GRL (AFGRL): Figure 3 shows that same-label neighbor ratios are high at small k, such as k = 4, but decrease as k grows, indicating residual noise.The local-positive strategy maintains a high correct ratio as k increases.
- Augmentation-Free GRL (AFGRL): Local positives are the intersection of k-nearest neighbors with adjacent nodes, incorporating relational inductive bias while filtering false positives.Adjacent nodes are motivated by the smoothness assumption that neighboring nodes tend to share labels.
- Augmentation-Free GRL (AFGRL): Global positives are the intersection of k-nearest neighbors with nodes assigned to the query node’s K-means cluster.Multiple clustering runs are used to improve robustness to centroid initialization.
- Augmentation-Free GRL (AFGRL): AFGRL combines local and global positives into real positives and minimizes cosine distance between each query node and its real positives.A predictor projects online embeddings, and the loss is symmetrized following BYOL.
- Augmentation-Free GRL (AFGRL): Unlike augmentation-based methods, AFGRL avoids augmentation hyperparameters and negative samples, aiming for stable training while reducing sampling bias and computational or memory costs.The method introduces k, K, and M instead, with reported performance stability across these hyperparameters.
6 Experiments
AFGRL is evaluated against graph self-supervised baselines on three node-level tasks across five datasets, using GCN encoders and standard evaluation protocols. It generally performs strongly, while ablations and sensitivity analyses support the value of combining local structure with global semantics and avoiding augmentation hyperparameters.
- Experimental Setup: The encoders use GCNs with self-loops, normalized adjacency, nonlinear activation, and trainable layer weights.The experiments grid-search learning rate, decay rate, embedding dimension, and GCN depth.
- Experimental Setup: AFGRL is compared with GRACE, BGRL, and GCA on node classification, clustering, and similarity search across five widely used datasets.The datasets are WikiCS, Amazon-Computers, Amazon-Photo, Coauthor-CS, and Coauthor-Physics.
- Performance Analysis: AFGRL generally performs well on node classification and generally outperforms other methods on node clustering.The compared methods use carefully tuned augmentation hyperparameters, whereas AFGRL avoids graph augmentation.
- Performance Analysis: Considering both local structural and global semantic information gives the best ablation performance, while global semantics is more beneficial than local structure.The k-NN-only variant performs on par with k-NN plus adjacency, whereas the clustering-only variant performs poorly.
- Performance Analysis: AFGRL is stable across its hyperparameters, benefits from high-dimensional embeddings, and captures more fine-grained class information than GCA.Other methods rapidly saturate as embedding dimensions increase, while AFGRL embeddings form tighter, smaller clusters within label groups.
7 Conclusion
AFGRL is a graph self-supervised framework that avoids both manual augmentations and negative samples by discovering positive nodes through local structure and global semantics. Experiments show competitive or superior performance with greater hyperparameter stability than augmentation-sensitive methods.
- 7 Conclusion: AFGRL learns graph representations without augmentation techniques or negative samples.It discovers positive samples using local structural information and global graph semantics.
- 7 Conclusion: AFGRL maintains competitive performance while being stable over hyperparameters, making the framework practical.The conclusion contrasts this stability with state-of-the-art methods sensitive to augmentation hyperparameters.
- 7 Conclusion: Experiments on multiple graphs and downstream tasks empirically show AFGRL is superior to state-of-the-art methods sensitive to augmentation hyperparameters.
9 Appendix
The appendix documents the datasets, comparison methods, implementation choices, hyperparameters, and source-code availability used to evaluate AFGRL. It emphasizes that AFGRL avoids augmentation-specific hyperparameters and remains stable over its newly introduced settings.
- 9.1 Datasets: The evaluation uses five datasets spanning reference, e-commerce, and academic co-authorship networks.WikiCS, Amazon-Computers, Amazon-Photo, Coauthor-CS, and Coauthor-Physics provide node labels and textual or keyword-based features.
- 9.2 Methods Compared: The compared methods include pioneering mutual-information approaches and augmentation-based methods such as GRACE and GCA.Table 6 summarizes the properties of the compared methods.
- 9.3 Implementation details: AFGRL uses a GCN base encoder with batch normalization and nonlinearity, plus a BYOL-style MLP predictor.A single-layer GCN works best for AFGRL, so its encoder has no hidden layer.
- 9.3 Implementation details: AFGRL introduces k, K, and M instead of graph-augmentation hyperparameters, and its performance is stable over these settings.The experiments fix K to 100 and M to 5 and select k from {4, 8}.
- 9.3 Implementation details: The best-performing augmentation hyperparameters for comparison methods vary substantially, ranging from 0.1 to 0.5.This appendix note identifies the augmentation-setting variability underlying the comparison concern.
- 9.4 Source codes: Baseline source code was publicly available except for BGRL, which the authors implemented in PyTorch using their framework.