Source-linked AI summary
Multi-Scale Contrastive Siamese Networks for Self-Supervised Graph Representation Learning
Ming Jin, Yizhen Zheng, Yuan-Fang Li, Chen Gong, Chuan Zhou, Shirui Pan
TL;DR
Graph representation learning often depends on labeled nodes, motivating a self-supervised alternative. MERIT combines Siamese self-distillation with multi-scale cross-view and cross-network contrastiveness, and experiments report superior classification performance across five datasets. Its approach addresses limitations associated with mutual-information estimators and reliance on negative samples, while its ablation results support the value of both contrastive schemes.
Problem
Many graph representation learning methods require labeled nodes, while existing self-supervised methods can require expensive mutual-information estimators or large numbers of negative samples.
Method
MERIT uses Siamese bootstrapping with cross-network and cross-view contrastive objectives, including augmented graph views and multi-scale graph contrastive learning.
Results
MERIT achieves the best classification accuracy on all five datasets, surpassing self-supervised and supervised methods except for a draw with GMI on PubMed.
Takeaways & Limitations
The results support combining Siamese bootstrapping with multiple contrastive routes to enrich self-supervision and regularize graph representation learning.
Takeaways & Limitations
Existing graph contrastive methods may remain sensitive to discriminator choice and dependent on deliberately selected negative samples.
Abstract
from arXiv · showhide
Graph representation learning plays a vital role in processing graph-structured data. However, prior arts on graph representation learning heavily rely on labeling information. To overcome this problem, inspired by the recent success of graph contrastive learning and Siamese networks in visual representation learning, we propose a novel self-supervised approach in this paper to learn node representations by enhancing Siamese self-distillation with multi-scale contrastive learning. Specifically, we first generate two augmented views from the input graph based on local and global perspectives. Then, we employ two objectives called cross-view and cross-network contrastiveness to maximize the agreement between node representations across different views and networks. To demonstrate the effectiveness of our approach, we perform empirical experiments on five real-world datasets. Our method not only achieves new state-of-the-art results but also surpasses some semi-supervised counterparts by large margins. Code is made available at https://github.com/GRAND-Lab/MERIT
1 Introduction
Graph representation learning is useful for downstream graph tasks, but labeling requirements and limitations of existing self-supervised and Siamese approaches motivate MERIT, a multi-scale Siamese contrastive framework for node representations.
- Motivation: Graph representation learning produces low-dimensional node or graph embeddings for downstream tasks such as node and graph classification.Most GNN-based approaches require labeled nodes, limiting adoption where labeling information is unavailable or costly.
- Related approaches: Random-walk and matrix-reconstruction methods emphasize graph proximity while overlooking other relationships within or between subgraphs.Contrastive methods extend mutual-information maximization to compare node representations with raw information or representations from other views.
- Limitations: Existing mutual-information methods require computationally expensive estimators and are sensitive to discriminator choices.Many graph contrastive methods also depend on carefully selected negative samples to avoid trivial representation collapse.
- Research gap: Siamese self-supervised methods have mainly been used in visual representation learning and had not yet been extended to graphs.This gap motivates adapting bootstrapping mechanisms to graph representation learning.
- Proposed framework: MERIT combines Siamese bootstrapping with multi-scale graph contrastive learning through cross-network and cross-view contrastive objectives.Its momentum-driven target network acts as a slowly moving mean teacher, while additional negatives preserve information about graph topology.
- Contributions: The paper claims a novel Siamese framework, two mutually regularizing contrastive objectives, and extensive evaluations against state-of-the-art self-supervised methods.The experiments use various real-world datasets to validate the proposed method.
2 Related Work
Prior graph representation methods include random walks, adjacency reconstruction, and contrastive learning, while Siamese architectures provide a separate self-supervised representation-learning paradigm developed mainly in vision.
- Siamese networks: Siamese networks use two or more identical structures, such as online and target encoders, for prediction or entity comparison.They were traditionally applied to supervised signature verification and face matching tasks.
- Self-supervised learning: Recent self-supervised visual methods use Siamese architectures without negative samples, while predictor and stop-gradient mechanisms help prevent collapse.Graph representation learning has also traditionally used random walks and adjacency-matrix reconstruction.
3 Proposed Method
MERIT learns low-dimensional node representations without labels by combining graph augmentations, Siamese bootstrapping, and multi-scale contrastive objectives. It contrasts representations across networks and views to capture local, attributive, and global graph information.
- Problem definition: MERIT aims to learn low-dimensional node embeddings without labeling information for downstream tasks such as node classification.The graph encoder maps node features and adjacency information into representations with D′ ≪ D.
- Overall framework: The framework generates two augmented graph views and processes them through online and target Siamese networks.The model contains graph augmentations, cross-network contrastive learning, and cross-view contrastive learning components.
- Graph augmentations: MERIT combines subsampling, edge modification, and node feature masking in one view with subsampling, graph diffusion, and feature masking in the other.This design encodes local structural and attributive information alongside global information from diffusion.
- Cross-network contrastive learning: Cross-network contrastiveness matches corresponding node representations across views and networks while using negative samples to separate other representations.The target network is updated by momentum and does not directly receive training gradients, supporting knowledge distillation from historical observations.
- Cross-view contrastive learning: Cross-view contrastiveness regularizes representations within the online network through intra-view and inter-view discrimination across differently augmented views.Inter-view contrastiveness pulls representations of the same nodes together and pushes representations of other nodes apart, incorporating local and global augmentations.
- Model training: The training objective minimizes a weighted combination of cross-view and cross-network contrastive losses, controlled by balance factor β.During inference, representations from the online encoder are aggregated using both graph adjacency and diffusion matrices.
4 Experiment
MERIT is evaluated on five benchmark datasets using linear evaluation, comparisons with supervised and self-supervised baselines, and sensitivity, ablation, and visualization studies. It achieves the best classification accuracy on all five datasets except a tie with GMI on PubMed, while component and parameter analyses support its multi-scale design.
- Experimental Setup: Experiments cover Cora, CiteSeer, PubMed, Amazon Photo, and Coauthor CS, with node representations evaluated using a separate logistic regression classifier.Results are averaged over ten repetitions with standard deviations reported.
- Classification Results: MERIT achieves the best classification accuracy on all five datasets, surpassing supervised and self-supervised methods except for a tie with GMI on PubMed.The comparison includes five supervised baselines and four graph contrastive learning models.
- Classification Results: Contrastive learning models achieve similar or better accuracy than supervised baselines on multiple datasets, but gaps remain on Cora and Coauthor CS.This comparison supports the value of mining multi-scale contrastive relations while identifying datasets where supervised methods remain stronger.
- Parameter Sensitivity Study: β values between 0.4 and 0.6 generally perform best, while m = 1 usually performs poorly and m = 0 can perform better.The authors relate intermediate β values to mutual regularization between the two contrastive losses and discuss predictor and stop-gradient effects for m = 0.
- Parameter Sensitivity Study: Increasing augmentation excessively can distort topological and attributive information, whereas graph diffusion with α = 0.05 performs best and removing it sharply reduces performance.The diffusion result is reported for Figure 4(b).
- Ablation Study: Removing either cross-view or cross-network contrastiveness reduces performance on CiteSeer and Amazon Photo, with improvements from the full model ranging from 0.2% to 0.4%.The reported gains are 0.4% and 0.3% over the cross-view ablation, and 0.2% and 0.4% over the cross-network ablation, respectively.
- Ablation Study: MERIT produces a clearer 2D t-SNE separation of CiteSeer node classes than GCN and DGI.The visualization is presented as evidence that MERIT learns more expressive node representations for downstream tasks.
5 Conclusion
MERIT combines Siamese graph networks with two complementary contrastive objectives for self-supervised node representation learning. Experiments demonstrate the method’s effectiveness and superiority.
- MERIT uses Siamese GNNs to distill knowledge from historical representations and stabilize the online graph encoder.
- Cross-view contrastiveness enriches self-supervision across multiple graph scales and regularizes the bootstrapping scheme.
- Experimental results demonstrate MERIT’s superiority and effectiveness for self-supervised graph representation learning.