Source-linked AI summary
InfoGraph: Unsupervised and Semi-supervised Graph-Level Representation Learning via Mutual Information Maximization
Fan-Yun Sun, Jordan Hoffmann, Vikas Verma, Jian Tang
TL;DR
Graph-level learning needs methods that work when labels are costly or scarce, while existing graph kernels can yield limited representations and generalization. InfoGraph maximizes mutual information across graph and substructure representations, and InfoGraph* transfers unsupervised information into semi-supervised learning. The reported results surpass state-of-the-art performance on unsupervised graph classification and are competitive on semi-supervised molecular property prediction.
Problem
Graph-level representation learning is needed for unlabeled or partially labeled graphs, but labels can be costly and graph-kernel representations may generalize poorly.
Method
InfoGraph maximizes mutual information between graph-level and patch-level representations, while InfoGraph* uses a student-teacher framework combining labeled supervision with unlabeled InfoGraph learning.
Results
InfoGraph surpasses state-of-the-art performance on unsupervised graph classification, while InfoGraph* achieves comparable performance on semi-supervised molecular property prediction.
Takeaways & Limitations
The methods provide competitive graph representation learning across unsupervised graph classification and semi-supervised molecular property prediction tasks.
Takeaways & Limitations
Combining supervised and unsupervised objectives in one encoder may cause negative transfer because the tasks can favor different information or semantic spaces.
Abstract
from arXiv · showhide
This paper studies learning the representations of whole graphs in both unsupervised and semi-supervised scenarios. Graph-level representations are critical in a variety of real-world applications such as predicting the properties of molecules and community analysis in social networks. Traditional graph kernel based methods are simple, yet effective for obtaining fixed-length representations for graphs but they suffer from poor generalization due to hand-crafted designs. There are also some recent methods based on language models (e.g. graph2vec) but they tend to only consider certain substructures (e.g. subtrees) as graph representatives. Inspired by recent progress of unsupervised representation learning, in this paper we proposed a novel method called InfoGraph for learning graph-level representations. We maximize the mutual information between the graph-level representation and the representations of substructures of different scales (e.g., nodes, edges, triangles). By doing so, the graph-level representations encode aspects of the data that are shared across different scales of substructures. Furthermore, we further propose InfoGraph*, an extension of InfoGraph for semi-supervised scenarios. InfoGraph* maximizes the mutual information between unsupervised graph representations learned by InfoGraph and the representations learned by existing supervised methods. As a result, the supervised encoder learns from unlabeled data while preserving the latent semantic space favored by the current supervised task. Experimental results on the tasks of graph classification and molecular property prediction show that InfoGraph is superior to state-of-the-art baselines and InfoGraph* can achieve performance competitive with state-of-the-art semi-supervised models.
1 INTRODUCTION
Graph-level representation learning is important for unlabeled or partially labeled graph data, but existing unsupervised approaches have limitations. InfoGraph uses mutual information maximization, while InfoGraph* extends it to semi-supervised learning and achieves strong reported performance.
- Motivation: Graph-level representations support applications such as molecular property prediction, while supervised graph learning is hindered by costly or unavailable annotations.Chemical labels may require costly Density Functional Theory calculations, motivating methods that use unlabeled data.
- Motivation: Graph kernels often lack explicit graph embeddings and use handcrafted features that can be high-dimensional, sparse, nonsmooth, and prone to poor generalization.These limitations are especially noted for large datasets.
- Approach: InfoGraph* uses a student-teacher framework in which the student uses labeled data and the teacher learns from unlabeled data with InfoGraph.The method maximizes mutual information between intermediate representations of the two models.
- Approach: InfoGraph learns unsupervised graph-level representations by maximizing mutual information between graph-level and patch-level representations.The objective encourages graph representations to encode information shared across substructures.
- Results: InfoGraph surpasses state-of-the-art performance on unsupervised graph classification, while InfoGraph* achieves performance comparable with state-of-the-art methods for semi-supervised molecular property prediction.These outcomes are reported as empirical contributions of the paper.
2 RELATED WORK
Related work covers unsupervised graph representations, mutual-information and contrastive learning, graph kernels, node-level Deep Graph InfoMax, and semi-supervised learning. The paper distinguishes InfoGraph by targeting whole-graph embeddings in unsupervised and semi-supervised settings.
- Unsupervised graph representations: Unsupervised graph-level representation learning has emerged alongside a literature historically focused mainly on supervised graph tasks.The related work identifies recent algorithms that learn graph-level representations without labels.
- Positioning: Information-maximizing graph neural networks target supervised molecule-property prediction, whereas this paper focuses on unsupervised and semi-supervised scenarios.The distinction concerns the learning setting rather than the use of mutual information itself.
- Graph kernels: Graph kernels represent graphs through frequencies of selected substructures, whereas InfoGraph replaces handcrafted substructure similarity with mutual information.Kernel design depends on deciding which substructures are suitable.
- Contrastive methods: Contrastive methods increase scores for statistically dependent representations and decrease scores for negative or unrelated examples.The related-work discussion places Deep Graph InfoMax within this category.
- Contrastive methods: Deep Graph InfoMax maximizes mutual information between node representations and pooled global graph representations, but evaluates node-level rather than whole-graph embeddings.InfoGraph shares a related methodology while pursuing a different representation target.
- Semi-supervised learning: Semi-supervised learning incorporates many unlabeled samples with a small number of labeled samples to improve prediction accuracy under scarce labels.Mean Teacher is discussed as a state-of-the-art method applicable to regression tasks.
3 METHODOLOGY
InfoGraph learns fixed-length graph representations by maximizing mutual information between graph-level representations and patch representations at multiple scales. InfoGraph* extends this framework to semi-supervised prediction by combining supervised learning with unsupervised representation learning while addressing potential negative transfer between objectives.
- Motivation: Graph-level representations support graph classification, regression, clustering, and molecular property prediction, while semi-supervised learning addresses scarce labels in biology and molecular applications.Partially labeled datasets commonly contain many more unlabeled than labeled graphs.
- InfoGraph: InfoGraph forms positive and negative global–patch pairs across all graph instances in a batch for discriminator-based mutual-information estimation.Negative samples use all possible combinations of global and local patch representations across graphs in the batch.
- InfoGraph: InfoGraph uses GNNs to aggregate local neighborhoods into patch representations and a READOUT function to produce fixed-length graph-level representations.Patch representations can incorporate information from multiple neighborhood scales through repeated GNN aggregation.
- InfoGraph: The method maximizes mutual information between each graph representation and patch representations, encouraging global embeddings to encode information shared across graph substructures and scales.A neural discriminator estimates mutual information for global–local representation pairs.
- InfoGraph*: InfoGraph* combines supervised prediction with an unsupervised InfoGraph objective so representations can use labeled and unlabeled data.The unsupervised term acts as a regularizer, but sharing one encoder may cause negative transfer when the objectives favor different information or semantic spaces.
- InfoGraph*: InfoGraph* uses separate supervised and unsupervised encoders and maximizes mutual information between their representations to transfer information while preserving task-relevant representations.The semi-supervised architecture uses two encoders with the same architecture and a discriminator operating on representation pairs.
4 EXPERIMENTS
The experiments evaluate InfoGraph on graph classification and semi-supervised molecular property prediction using established datasets, baselines, and controlled comparison settings.
- InfoGraph is evaluated on six graph-classification benchmark datasets, while semi-supervised experiments use the QM9 molecular dataset.
- Graph classification: Graph-classification comparisons include six graph kernels and three unsupervised graph-level representation-learning methods.
- Semi-supervised learning: Semi-supervised molecular-property experiments compare fully supervised results with Mean Teacher and InfoGraph-based methods.
- Evaluation protocol: Graph-classification performance is reported with 10-fold cross-validation accuracy across five repetitions.
- Evaluation protocol: QM9 experiments use 5,000 labeled samples, 10,000 validation samples, 10,000 test samples, and the remaining molecules as unlabeled training data.
- Model configuration: The unsupervised experiments use GIN, while semi-supervised comparisons use the same underlying enn-s2s architecture for rigorous comparison.
5 RESULTS
InfoGraph performs strongly in unsupervised graph classification, while InfoGraph* improves semi-supervised molecular-property prediction across all evaluated targets.
- Unsupervised graph classification: Graph-kernel baselines perform well on individual datasets but are not competitive across all datasets, while MLG requires more than 24 hours on two larger benchmarks.
- Unsupervised graph classification: InfoGraph outperforms all listed baselines on 4 of 6 graph-classification datasets and remains competitive on the other 2.
- Semi-supervised molecular property prediction: Combining supervised and unsupervised InfoGraph objectives improves performance over purely supervised models on 7 of 12 molecular-property targets.
- Semi-supervised molecular property prediction: The combined InfoGraph objective performs worse on 4 of 12 targets, illustrating a negative-transfer effect when objectives favor different information or latent semantic spaces.
- Semi-supervised molecular property prediction: InfoGraph* improves over the supervised model on all 12 targets and obtains the best result on 11 targets.
6 CONCLUSION AND FUTURE WORK
The paper proposes InfoGraph for unsupervised graph-level representation learning and InfoGraph* for semi-supervised learning, evaluating both on graph and molecular tasks.
- InfoGraph learns unsupervised graph-level representations, while InfoGraph* addresses semi-supervised learning.
- Experiments evaluate the methods on graph classification and molecular property prediction tasks.
- The reported results show that both methods are competitive with state-of-the-art methods.
- Future work: Future work will explore semi-supervised frameworks designed specifically for graph-structured data.
A RELATED WORK
Related work on graph representation learning includes multiple families of graph kernels designed around graph substructures and multiscale structure.
- Graph kernels: Graphlet, random-walk, and shortest-path kernels are established graph-kernel families.
- Graph kernels: The Weisfeiler-Lehman subtree kernel represents another established approach to graph kernels.
- Graph kernels: Deep graph, graph-invariant, optimal-assignment, and multiscale-Laplacian kernels extend graph-kernel design in other directions.
A.2 CONTRASTIVE METHODS
Contrastive methods learn representations by distinguishing inputs that capture desired statistical dependencies from those that do not. The section covers representative semi-supervised objectives and graph extensions of language-model approaches.
- Graph2vec extends Doc2vec to arbitrary graphs by treating graphs as documents and rooted subgraphs as paragraph words.It uses Weisfeiler-Lehman relabelling to enumerate rooted subgraphs up to a specified depth.
- Entropy minimization encourages confident, low-entropy predictions on all unlabeled examples regardless of class.
- Pseudo-labeling uses predictions above a predefined class-probability threshold as targets in the supervised loss.
- The Π-Model encourages consistent outputs across different stochastic passes of unlabeled data.
- Virtual Adversarial Training adds an approximated input perturbation designed to most affect the prediction function.The perturbation requires an extra back-propagation for each optimization step.
- Mean Teacher stabilizes unlabeled-data targets by using predictions from an exponential moving average of parameters from earlier training steps.
C.1 GRAPH CLASSIFICATION DATASETS
The graph-classification datasets cover chemical compounds, movie-collaboration ego-networks, and online discussion threads, with graph-level labels defined by their application domains.
- MUTAG contains 188 mutagenic aromatic and heteroaromatic nitro compounds with 7 discrete labels.
- PTC contains 344 chemical compounds tested for carcinogenicity in male and female rats, with 19 discrete labels.
- IMDB-BINARY and IMDB-MULTI represent actors’ or actresses’ ego-networks, classifying each graph by its movie genre.
- REDDIT-BINARY and REDDIT-MULTI5K represent online discussion threads, classifying graphs by their community or subreddit.
C.2 QM9
QM9 contains roughly 134,000 drug-like molecules with 12 pre-computed chemical properties. The experiments specify model and training settings, and InfoGraph*’s three losses converge after about 150 epochs.
- QM9 contains about 134,000 molecules composed of H, C, O, N, and F, with up to 9 non-hydrogen atoms.The dataset provides 12 pre-computed chemical properties for each molecule.
- The unsupervised experiments use GIN with 4, 8, or 12 layers, learning rates from 10^-2 to 10^-4, and 10, 20, or 100 epochs.The batch size is 128.
- The semi-supervised experiments train for 500 epochs with learning rate 0.001 and batch size 20, using 3 set2set computations.Weight decay and λ are selected from specified candidate sets.
- The discriminator scores global-patch representation pairs using transformed representations and their dot product.The transformations are 3-layer feed-forward networks with jumping connections and ReLU activations.
- Around 150 epochs, all three InfoGraph* loss terms converge during training.
- Figure 3 presents the InfoGraph* convergence plot for QM9 target 7.