Source-linked AI summary
Semi-Implicit Graph Variational Auto-Encoders
Arman Hasanzadeh, Ehsan Hajiramezanali, Nick Duffield, Krishna R. Narayanan, Mingyuan Zhou, Xiaoning Qian
TL;DR
SIG-VAE addresses limited posterior and generative flexibility in VGAE for graph data. It combines semi-implicit hierarchical inference, neighborhood sharing, and a Bernoulli-Poisson decoder; experiments report superior performance across graph analytic tasks and more interpretable latent representations.
Problem
VGAE is limited by restrictive posterior assumptions and decoder flexibility when modeling uncertain, dependent graph data.
Method
SIG-VAE uses semi-implicit hierarchical variational inference with neighborhood sharing and a Bernoulli-Poisson link decoder.
Results
SIG-VAE significantly outperforms state-of-the-art methods across several graph analytic tasks and produces more interpretable latent representations than VGAE.
Takeaways & Limitations
The model captures graph posteriors with heavy tails, skewness, multimodality, and rich dependency structures that simpler VGAE constructions may miss.
Abstract
from arXiv · showhide
Semi-implicit graph variational auto-encoder (SIG-VAE) is proposed to expand the flexibility of variational graph auto-encoders (VGAE) to model graph data. SIG-VAE employs a hierarchical variational framework to enable neighboring node sharing for better generative modeling of graph dependency structure, together with a Bernoulli-Poisson link decoder. Not only does this hierarchical construction provide a more flexible generative graph model to better capture real-world graph properties, but also does SIG-VAE naturally lead to semi-implicit hierarchical variational inference that allows faithful modeling of implicit posteriors of given graph data, which may exhibit heavy tails, multiple modes, skewness, and rich dependency structures. Compared to VGAE, the derived graph latent representations by SIG-VAE are more interpretable, due to more expressive generative model and more faithful inference enabled by the flexible semi-implicit construction. Extensive experiments with a variety of graph data show that SIG-VAE significantly outperforms state-of-the-art methods on several different graph analytic tasks.
1 Introduction
Graph representation learning seeks low-dimensional embeddings that preserve graph structure for downstream tasks, while VGAE extends this setting to uncertain node representations. SIG-VAE addresses limitations in posterior and decoder flexibility through semi-implicit inference and a more expressive graph generative model.
- Graph representation learning compresses structural information into low-dimensional vectors for link prediction, clustering, and node classification.
- VGAE models each node as a latent random variable to represent uncertainty, which matters in applications such as physics, biology, and pharmaceutical link prediction.
- Gaussian variational distributions can restrict inference when graph posteriors violate Gaussian assumptions, while inner-product decoders limit generative flexibility.
- SIG-VAE combines hierarchical semi-implicit inference with a Bernoulli-Poisson decoder to model complex posteriors and sparse real-world graph properties.
2 Background
The background formulates graph latent-variable inference as estimating p(Z | X, A), a difficult posterior often approximated by Gaussian VGAE distributions. Semi-implicit inference broadens the variational family by allowing implicit, dependent, and complex multivariate posteriors while retaining tractable optimization machinery.
- The central problem is estimating the latent node distribution p(Z | X, A) from graph structure and node attributes.
- VGAE uses graph-convolutional networks to parameterize independent Gaussian node embeddings and an inner-product decoder, optimizing an evidence lower bound.
- VGAE inference can underestimate posterior variance when its variational family cannot match the true posterior complexity and KL divergence is asymmetric.
- SIVI can capture multimodal or skewed posteriors that vanilla variational inference misses under restricted exponential-family assumptions.
- Semi-implicit distributions make the marginal posterior implicit and can model complex multivariate dependencies without assuming independent latent dimensions.
3 Baselines: Variational Inference with VGAE
The baseline section combines VGAE with normalizing flows or semi-implicit variational inference to obtain more flexible posteriors. These combinations improve posterior expressiveness but do not fully propagate uncertainty across neighboring nodes, motivating SIG-VAE’s neighborhood-sharing design.
- SIVI-VGAE: SIVI-VGAE introduces a reparameterizable implicit prior over node-specific Gaussian parameters, producing flexible posterior mixtures with neural networks and random noise.
- SIVI-VGAE: Naive SIVI-VGAE can make node posterior parameters random, but its separately constructed implicit distributions may not fully capture dependencies between neighboring nodes.
- NF-VGAE: Normalizing-flow baselines require deterministic transformations with easily computed Jacobians, limiting flexibility for complex graph dependency structures.
- NF-VGAE: NF-VGAE successively transforms Gaussian samples through invertible differentiable mappings with tractable Jacobians.
4 Semi-implicit graph variational auto-encoder (SIG-VAE)
SIG-VAE combines hierarchical semi-implicit inference with uncertainty-aware neighborhood sharing and a Bernoulli-Poisson decoder to model graph dependencies and flexible node posteriors. Experiments show more interpretable latent structures and richer distributions than VGAE on the Swiss roll example.
- Inference framework: The hierarchical encoder injects random noise across multiple stochastic layers, while its first layer remains reparameterizable with an analytic density.Subsequent layers are designed to remain computationally efficient to sample from.
- Inference framework: After marginalization, SIG-VAE’s posterior need not be Gaussian and can represent distributions beyond Gaussian, exponential-family, or von Mises–Fisher families.Randomized parameters and noise allow more expressive latent representations.
- Inference framework: SIG-VAE propagates neighboring nodes’ distributions rather than deterministic features, enabling uncertainty sharing that better captures graph dependency structure.Its stochastic GNN layers produce mixing distributions for node posteriors.
- Decoder: The Bernoulli-Poisson link decoder increases generative-model expressiveness beyond the inner-product decoder.The decoder models adjacency through Poisson-generated edge counts thresholded into binary links.
- Empirical illustration: SIG-VAE produces a more interpretable curved-plane Swiss-roll embedding and richer node posterior distributions than VGAE, including multimodal, skewed, nonsymmetric, and sharply changing shapes.The comparison uses the same inner-product decoder and parameter count for both models.
5 Experiments
Experiments evaluate SIG-VAE on interpretability, link prediction, graph generation, and node classification. SIG-VAE achieves strong link-prediction performance, interpretable latent distributions, and competitive results across graph-analysis tasks.
- 5.1 Interpretable latent representations: SIG-VAE produces more interpretable latent structure than VGAE, including multimodal, skewed, nonsymmetric, and sharply varying node posteriors.In the Swiss Roll graph, multimodal posteriors occur for nodes located between different communities.
- 5.2 Accurate link prediction: SIG-VAE significantly outperforms competing methods in link prediction across networks with and without node attributes.The reported gains cover both AUC and AP; on the Power graph, SIG-VAE improves accuracy by 34% compared with VGAE.
- 5.2 Accurate link prediction: SIG-VAE consistently performs well without node attributes, while its decoder choice varies by graph density.The Bernoulli-Poisson decoder performs well on sparser graphs, whereas the inner-product decoder performs better on the denser USAir graph.
- 5.3 Graph generation: Graph generation with SIG-VAE and VGAE using the inner-product decoder yields density 0.1178 and average clustering coefficient 0.49, indicating limitations for sparse graphs.The reported generated-graph statistics are identical for the two models in this experiment.
- 5.4 Node classification & graph clustering: SIG-VAE shows competitive node-classification performance and robustness to randomly removed edges despite not being trained specifically for classification.The classification evaluation uses graph reconstruction and semi-supervised classification terms, with experiments removing 10%, 20%, 50%, and 70% of edges.
6 Conclusion
SIG-VAE combines semi-implicit hierarchical inference with a Bernoulli-Poisson decoder to improve node-posterior flexibility and graph modeling. Its experiments show promising performance across graph-analysis applications and interpretable latent representations.
- 6 Conclusion: SIG-VAE combines semi-implicit hierarchical variational distributions with VGAE and a Bernoulli-Poisson decoder to capture graph structure and node attributes.The hierarchical construction diffuses neighborhood-node distributions through the graph.
- 6 Conclusion: An asymptotically exact surrogate evidence lower bound enables stochastic-gradient inference without sacrificing variational-distribution flexibility.The surrogate objective makes inference computationally amenable while retaining the expressive semi-implicit construction.
- 6 Conclusion: Experiments across graph datasets show promising SIG-VAE performance in multiple graph-analysis applications with interpretable latent representations.The conclusion attributes interpretability to the hierarchical construction that diffuses neighborhood distributions.