Source-linked AI summary
Self-supervised Graph-level Representation Learning with Local and Global Structure
Minghao Xu, Hang Wang, Bingbing Ni, Hongyu Guo, Jian Tang
TL;DR
Self-supervised graph representation learning must address limited labeled data and the gap between local similarity modeling and global semantic structure. GraphLoG combines local-instance alignment with hierarchical prototypes learned by online EM, and experiments on chemistry and biology benchmarks show superior downstream performance, including a 2.1% average ROC-AUC gain on eight chemistry tasks.
Problem
Limited and expensive labeled data make unsupervised or self-supervised whole-graph representation learning important, while existing approaches mainly preserve local similarities without discovering global semantic structure.
Method
GraphLoG aligns correlated graph or subgraph embeddings locally, models global structure with hierarchical prototypes, and learns the GNN parameters and prototypes using an online expectation-maximization algorithm.
Results
GraphLoG outperforms previous self-supervised approaches on six of eight chemistry tasks and gains 2.1% in average ROC-AUC across those eight tasks, with superior performance also reported across chemistry and biology benchmarks.
Takeaways & Limitations
Modeling both local and global structure provides a self-supervised framework that empirically improves downstream graph prediction across chemical and biological benchmarks.
Takeaways & Limitations
The authors identify further improvement of global structure learning, unification of pre-training and fine-tuning, and extension to additional domains as future work.
Abstract
from arXiv · showhide
This paper studies unsupervised/self-supervised whole-graph representation learning, which is critical in many tasks such as molecule properties prediction in drug and material discovery. Existing methods mainly focus on preserving the local similarity structure between different graph instances but fail to discover the global semantic structure of the entire data set. In this paper, we propose a unified framework called Local-instance and Global-semantic Learning (GraphLoG) for self-supervised whole-graph representation learning. Specifically, besides preserving the local similarities, GraphLoG introduces the hierarchical prototypes to capture the global semantic clusters. An efficient online expectation-maximization (EM) algorithm is further developed for learning the model. We evaluate GraphLoG by pre-training it on massive unlabeled graphs followed by fine-tuning on downstream tasks. Extensive experiments on both chemical and biological benchmark data sets demonstrate the effectiveness of the proposed approach.
1. Introduction
Graph-level representation learning is important across scientific domains, but labeled graphs are scarce and existing self-supervised methods largely miss global semantic structure. GraphLoG combines local-instance preservation with hierarchical prototypes and online EM, achieving strong chemistry and biology benchmark results.
- Whole-graph representations support molecule, protein, and circuit property prediction across scientific domains.
- Limited and expensive labels motivate unsupervised or self-supervised learning of graph representations.
- Existing self-supervised graph methods preserve local-instance relationships but fail to discover the data set’s global-semantic structure.
- GraphLoG aligns correlated graph or subgraph embeddings locally and introduces hierarchical prototypes to model global semantics.
- GraphLoG learns its GNN parameters and prototypes with an online expectation-maximization algorithm.
- 2.1% average ROC-AUC gain across eight chemistry tasks accompanied GraphLoG’s best performance on six tasks.
2. Problem Definition and Preliminaries
The paper defines self-supervised graph representation learning as mapping unlabeled graphs to low-dimensional embeddings that preserve both pairwise local similarity and global semantic clustering. It also reviews GNN embeddings and EM optimization for latent-variable models.
- 2.1. Problem Definition: Local-instance structure places similar graph or subgraph embeddings nearby and dissimilar pairs far apart.
- 2.1. Problem Definition: Local structure alone is insufficient to capture semantics underlying the entire data set.
- 2.1. Problem Definition: Global-semantic structure organizes graph embeddings into patterns reflecting the data set’s potentially hierarchical semantic clusters.
- 2.1. Problem Definition: The task maps each unlabeled graph G_m to a low-dimensional vector h_Gm, with embeddings expected to follow both local and global structure.
- 2.2. Preliminaries: GNNs use neighborhood aggregation to learn node embeddings and a graph-level embedding from graph structure and attributes.
- 2.2. Preliminaries: EM estimates latent-variable model parameters by alternating posterior inference in the E-step with expectation maximization in the M-step.
3. GraphLoG: Self-supervised Graph-level Representation Learning with Local and Global Structure
GraphLoG learns graph representations by combining local-instance similarity with global-semantic structure. It uses hierarchical prototypes and online EM to model graph embeddings for downstream tasks.
- GraphLoG framework: GraphLoG discovers local and global graph-embedding structure for downstream graph classification.It aligns correlated graphs or subgraphs locally and models global structure with hierarchical prototypes.
- Learning local-instance structure: Local-structure learning brings correlated graph or subgraph embeddings together while separating negative pairs.Correlated graph views arise by randomly masking node or edge attributes; negative pairs replace a graph or subgraph with another sampled instance.
- Learning global-semantic structure: Hierarchical prototypes represent global semantic clusters that cannot be obtained from explicit semantic labels in self-supervised learning.The prototypes are organized as trees, with each prototype associated with child prototypes at the next layer.
- Online EM optimization: GraphLoG maximizes a latent-variable model’s data likelihood over GNN parameters and hierarchical prototypes using online EM on mini-batches.The E-step samples prototype chains for graphs, while the M-step updates parameters using the expected complete-data likelihood.
- Initialization: Initialization first pre-trains the GNN with the local objective, then uses K-means to initialize prototype layers hierarchically.Cluster centers assigned fewer than two samples are discarded to avoid trivial solutions.
- Online EM optimization: Each EM cycle increases the marginal likelihood unless a local maximum is reached on the mini-batch log-likelihood.This is the paper’s stated guarantee for the proposed parameter updates.
4. Related Work
Related work covers GNN representation learning and self-supervised semantic learning. Prior approaches include propagation, mutual-information maximization, prediction tasks, and clustering-based methods.
- Graph Neural Networks: GNNs learn graph representations through neighborhood aggregation and node- or graph-level embedding vectors.The related-work discussion describes message-passing-based graph representation learning alongside earlier random-walk and matrix-factorization approaches.
- Self-supervised learning for GNNs: Self-supervised GNN research includes embedding propagation, mutual-information maximization, edge prediction, context prediction, and graph partitioning.These methods provide several self-supervised objectives for graph representation learning.
- Self-supervised semantic learning: Clustering-based self-supervised methods learn semantic information and have demonstrated transferability to downstream tasks.The discussion cites DeepCluster and Prototypical Contrastive Learning as examples of semantic-learning approaches.
5. Experiments
Experiments evaluate GraphLoG through pre-training and fine-tuning across chemistry and biology benchmarks, alongside comparisons, ablations, sensitivity analyses, and visualization. GraphLoG generally improves downstream graph representation performance, with global-semantic learning and larger hierarchical structures or batches contributing to the observed results.
- Experimental setup: GraphLoG is evaluated through pre-training followed by fine-tuning on chemistry and biology tasks, with analytical studies of local and global structure learning.The experimental procedure uses unlabeled graphs for pre-training and downstream supervised tasks for fine-tuning.
- Chemistry domain: 2.1% average ROC-AUC gain accompanies the best performance on six of eight molecular property prediction tasks among self-supervised strategies.The chemistry experiments use two million unlabeled ZINC15 molecules for pre-training and eight MoleculeNet downstream classification tasks.
- Biology domain: 1.7% performance gain is reported for GraphLoG on the biological function prediction benchmark, supporting fine-grained classification.The biology setup pre-trains on 395K unlabeled protein ego-networks and predicts 40 functions across eight species.
- Architecture comparison: GraphLoG outperforms existing approaches across four GNN architectures and avoids a performance decrease relative to random initialization on GAT.The architectures are GCN, GraphSAGE, GAT, and GIN.
- Ablation study: Global-semantic learning performs best among individual objectives, while combining objectives forms the full model evaluated in the ablation study.The analysis is conducted on the biology domain using GIN.
- Sensitivity and visualization: Deeper hierarchical prototypes (Lp >=3) and larger batches (N >=256) produce stable or improved performance, while global constraints yield clearer t-SNE feature separation.Larger batches are described as better representing the whole data set for likelihood estimation.
6. Conclusions and Future Work
GraphLoG models both local-instance and global-semantic structure in unlabeled graph data and shows superior performance across chemistry and biology benchmarks. Future work targets improved global learning, unified pre-training and fine-tuning, and broader domain coverage.
- GraphLoG models unlabeled graph structure both locally and globally through hierarchical prototypes over graph embeddings.
- Future work includes improving global structure learning, unifying pre-training with fine-tuning, and extending GraphLoG to sociology, physics, and material science.
A. Theoretical Analysis
The theoretical analysis shows that online EM updates increase GraphLoG’s marginal likelihood under mini-batch optimization, except when a local maximum is reached. The argument uses posterior-based E-steps, likelihood optimization in M-steps, and mini-batch approximations.
- Theorem 1 states that a randomly sampled mini-batch’s expected complete-data log-likelihood is approximately proportional to the full-data expectation.
- In the E-step, the variational distribution is set to the current posterior, making the KL divergence vanish and equating the ELBO with the marginal log-likelihood.
- In the M-step, the posterior-based variational distribution is fixed while model parameters θ and C are optimized to increase the expected mini-batch log-likelihood.
- The increase in ELBO after an M-step implies an increase in mini-batch marginal log-likelihood through the stated likelihood relation.
- The EM cycle increases the complete-data marginal likelihood p(G|θ, C), unless a local maximum is reached on the mini-batch objective.
B. More Implementation Details
Implementation details differ by domain for constructing correlated graph counterparts, while all evaluated GNN architectures use a common five-layer, 300-dimensional setup with mean pooling.
- For chemistry graphs, GraphLoG randomly masks attributes of 30% of nodes, marking masked atom type and chirality features with an extra dimension.
- For biology protein ego-networks, GraphLoG randomly masks attributes of 30% of edges using an extra feature dimension.
- GCN, GraphSAGE, GAT, and GIN use five layers, 300-dimensional hidden units, and mean pooling; GAT additionally uses two attention heads per layer.