Source-linked AI summary
Graph Representation Learning via Graphical Mutual Information Maximization
Zhen Peng, Wenbing Huang, Minnan Luo, Qinghua Zheng, Yu Rong, Tingyang Xu, Junzhou Huang
TL;DR
Graph representation learning needs ways to preserve rich graph information without expensive or unavailable labels. The paper introduces GMI to maximize mutual information between graph inputs and encoder outputs across features and topology, using a tractable decomposition estimated with MINE. Across transductive and inductive classification and link-prediction tasks, the method outperforms state-of-the-art unsupervised baselines and sometimes supervised counterparts.
Problem
Graph data contain abundant content and linkage information, but graph neural network training commonly requires labels that are costly to acquire or unavailable because of privacy constraints.
Method
The paper defines Graphical Mutual Information between input support graphs and node representations, decomposes it into weighted local mutual informations, and maximizes it with MINE.
Results
GMI outperforms state-of-the-art unsupervised baselines across transductive and inductive node classification and link prediction, sometimes exceeding supervised counterparts.
Takeaways & Limitations
GMI provides an unsupervised graph representation objective that preserves node features and topological structure while remaining invariant to graph isomorphism and compatible with MINE.
Takeaways & Limitations
The formulation must handle graph-specific challenges including invariance to node indexing and variable-sized support graphs, while feature sparsity can limit unsupervised performance on PPI.
Abstract
from arXiv · showhide
The richness in the content of various information networks such as social networks and communication networks provides the unprecedented potential for learning high-quality expressive representations without external supervision. This paper investigates how to preserve and extract the abundant information from graph-structured data into embedding space in an unsupervised manner. To this end, we propose a novel concept, Graphical Mutual Information (GMI), to measure the correlation between input graphs and high-level hidden representations. GMI generalizes the idea of conventional mutual information computations from vector space to the graph domain where measuring mutual information from two aspects of node features and topological structure is indispensable. GMI exhibits several benefits: First, it is invariant to the isomorphic transformation of input graphs---an inevitable constraint in many existing graph representation learning algorithms; Besides, it can be efficiently estimated and maximized by current mutual information estimation methods such as MINE; Finally, our theoretical analysis confirms its correctness and rationality. With the aid of GMI, we develop an unsupervised learning model trained by maximizing GMI between the input and output of a graph neural encoder. Considerable experiments on transductive as well as inductive node classification and link prediction demonstrate that our method outperforms state-of-the-art unsupervised counterparts, and even sometimes exceeds the performance of supervised ones.
1 INTRODUCTION
The paper motivates unsupervised graph representation learning by the abundance of graph content and linkage information and the cost or unavailability of labels. It proposes GMI, which directly maximizes mutual information between graph inputs and encoder outputs across node features and topology.
- Motivation: Supervision is costly or sometimes unavailable, making unsupervised graph representation learning useful for large-scale unlabeled data.Labels may require manual annotation or paid access and can be inaccessible because of privacy policies.
- Motivation: Graph networks contain rich node-content and linkage information that can support classification, link prediction, and related mining tasks.Examples include user features and relationships in social networks, where similarity and shared connections provide useful signals.
- GMI: GMI incorporates both node features and topology by decomposing graphical mutual information into a weighted sum of local mutual informations.The weights are assigned using representation-space proximity and are further correlated with input edge features.
- GMI: GMI generalizes mutual information estimation to graphs and directly relates input graphs to high-level node representations without a readout or corruption function.Its formulation compares each node’s support subgraph with that node’s hidden representation.
- Optimization: MINE enables tractable node-level estimation and maximization of GMI, supporting an unsupervised graph encoder objective.The paper presents a calculable decomposition intended for current mutual-information estimation methods.
- Experiments: Experiments on transductive and inductive node classification and link prediction report promising benchmark performance, sometimes exceeding supervised counterparts.The reported evaluation spans multiple graph-mining tasks and unsupervised comparisons.
2 RELATED WORK
Related work covers mutual information estimation and neural graph representation learning. It positions GMI against Deep InfoMax, MINE, supervised and unsupervised GNN methods, and DGI’s coarse-grained graph-level mutual-information objective.
- Mutual information estimation: Mutual-information methods pursue the InfoMax principle, but high-dimensional continuous-variable mutual information is difficult to calculate for deep networks.MINE addresses this difficulty by training a statistics network to distinguish joint-distribution samples from product-of-marginals samples.
- Neural networks for graph representation learning: Graph representation learning has expanded from random-walk and factorization methods to GNN-based approaches with applications in supervised and unsupervised settings.Supervised methods benefit from labels, while unsupervised methods have broader application potential when labels are inaccessible.
- Neural networks for graph representation learning: DGI applies mutual-information maximization to graphs and performs strongly, but its objective is described as coarse-grained and graph- or patch-level.This distinction motivates more direct correlation between hidden representations and original graph inputs.
3 GRAPHICAL MUTUAL INFORMATION: DEFINITION AND MAXIMIZATION
This section defines Graphical Mutual Information for graph encoders and develops a topology-aware, tractable objective that combines node-feature and edge information. The formulation addresses graph-specific invariance and variable-size challenges while enabling mutual-information maximization with estimators such as MINE.
- Graphical mutual information: Graph representation learning encodes each node from its support graph, consisting of neighborhood features and the corresponding adjacency structure.For an l-layer GNN, the support includes neighbors within at most l hops, and may include the node itself through self-loops.
- Graphical mutual information: Defining mutual information between a node representation and its support graph requires node-index invariance and fixed-size inputs for MINE.Isomorphic support graphs should yield the same mutual information, whereas neighborhoods can contain different numbers of nodes.
- Feature mutual information: Feature Mutual Information decomposes global mutual information into a weighted sum of local mutual informations between each hidden vector and neighboring node features.The decomposition assumes a multiplicative conditional probability and makes discriminator inputs fixed-size pairs of hidden vectors and node features.
- Topology-aware mutual information: Graphical Mutual Information extends this decomposition by using representation similarity as weights and adding mutual information between weights and edge features.The resulting objective explicitly incorporates both node features and graph topology at node level.
- Topology-aware mutual information: The topology-aware weighting adapts each local contribution to representation proximity while encouraging consistency with topological relations.For unweighted graphs, the edge-related mutual-information term can be replaced by a negative cross-entropy loss for more efficient computation.
- Maximization of GMI: GMI is invariant to isomorphic input transformations, computationally feasible through MINE, and intended to capture fine-grained information from both node and edge inputs.MINE maximizes a lower bound on mutual information, while other estimators such as JSD and infoNCE can also be used when only maximization is needed.
4 EXPERIMENTS
Experiments evaluate GMI on node classification and link prediction across transductive and inductive settings, comparing variants, objectives, and representation quality. GMI generally outperforms unsupervised alternatives, while residual and dense variants mitigate depth-related degradation.
- Experimental Setup: Experiments cover transductive and inductive node classification, link prediction, embedding visualization, and model-depth analysis using benchmark graph datasets.The classification datasets include Cora, Citeseer, PubMed, Reddit, and PPI; link prediction also uses BlogCatalog and Flickr.
- Node Classification: GMI-mean and GMI-adaptive achieve the best classification accuracy among unsupervised methods on all three transductive datasets.Their results are comparable to supervised GCN and GAT, and exceed them on Citeseer and PubMed.
- Node Classification: GMI-mean and GMI-adaptive outperform competing unsupervised algorithms on Reddit and PPI, although sparse PPI features limit unsupervised performance.The authors report that 42% of PPI nodes have zero feature values, whereas labels provide auxiliary information to supervised models.
- Objective Comparison: Across five classification datasets, GMI objectives outperform DGI and EP-B, while fine-grained input-output MI avoids limitations attributed to coarse MI or output-only constraints.The comparison replaces the objective function while keeping other experimental settings unchanged.
- Link Prediction: Under different edge-removal rates, GMI and FMI outperform DGI on link prediction except FMI at 70.0% edge removal on BlogCatalog.The comparison uses AUC scores on four datasets; the authors associate GMI’s advantage with fine-grained graphical MI and no corruption function.
- Model Depth: Increasing model depth widens the performance gap between models with and without shortcut connections, while dense GMI partly alleviates deterioration.Residual connections improve deep-network training, and dense GMI constrains each hidden layer to preserve input information despite MI decay with depth.
5 CONCLUSION
The paper introduces GMI-based unsupervised graph representation learning, directly maximizing mutual information between graph encoder inputs and outputs across node features and topology. The resulting model performs strongly across classification and link prediction tasks, while future work targets task-oriented and broader graph settings.
- GMI directly maximizes mutual information between graph encoder inputs and outputs using node features and topological structure.
- Theoretical analysis decomposes GMI into a weighted sum that can be efficiently calculated with MINE.
- The resulting unsupervised model outperforms state-of-the-art unsupervised baselines across transductive and inductive classification and link prediction tasks.
- Future work will address task-oriented representation learning and adapt GMI maximization to heterogeneous graphs and hypergraphs.