PaperlayerView exact revision on arXiv

Source-linked AI summary

NeVAE: A Deep Generative Model for Molecular Graphs

Bidisha Samanta, Abir De, Gourhari Jana, Pratim Kumar Chattaraj, Niloy Ganguly, Manuel Gomez-Rodriguez

arXiv:1802.05283v4cs.LGphysics.soc-phstat.ML

TL;DR

Molecular graph generators must handle non-grid structure, node-label permutation invariance, variable graph sizes, and three-dimensional coordinates. The paper introduces NeVAE, a variational autoencoder with graph-specific encoder and decoder innovations, plus gradient-based optimization for molecular properties and stability. NeVAE discovers plausible, diverse, and novel molecules more effectively than several state-of-the-art methods, with optimized decoding producing property values 121% higher than competing approaches.

  • Problem

    Existing molecular graph generators have limitations including fixed atom counts, lack of permutation invariance, quadratic training complexity, constrained diversity, and missing atom coordinates.

  • Method

    NeVAE is a variational autoencoder for molecular graphs with permutation-invariant, variable-sized graph encoding, specialized edge and coordinate decoding, masking, and gradient-based property optimization.

  • Results

    NeVAE discovers plausible, diverse, and novel molecules more effectively than several state-of-the-art methods, while optimized decoding identifies molecules with property values 121% higher than the best-performing competitor.

  • Takeaways & Limitations

    NeVAE supports molecular graph generation with variable sizes, permutation invariance, three-dimensional atom coordinates, local structural constraints, and property-oriented optimization.

  • Takeaways & Limitations

    The model assumes graphs are static, leaving dynamic graphs as a future extension.

Abstract

from arXiv · show

Deep generative models have been praised for their ability to learn smooth latent representation of images, text, and audio, which can then be used to generate new, plausible data. However, current generative models are unable to work with molecular graphs due to their unique characteristics-their underlying structure is not Euclidean or grid-like, they remain isomorphic under permutation of the nodes labels, and they come with a different number of nodes and edges. In this paper, we first propose a novel variational autoencoder for molecular graphs, whose encoder and decoder are specially designed to account for the above properties by means of several technical innovations. Moreover, in contrast with the state of the art, our decoder is able to provide the spatial coordinates of the atoms of the molecules it generates. Then, we develop a gradient-based algorithm to optimize the decoder of our model so that it learns to generate molecules that maximize the value of certain property of interest and, given a molecule of interest, it is able to optimize the spatial configuration of its atoms for greater stability. Experiments reveal that our variational autoencoder can discover plausible, diverse and novel molecules more effectively than several state of the art models. Moreover, for several properties of interest, our optimized decoder is able to identify molecules with property values 121% higher than those identified by several state of the art methods based on Bayesian optimization and reinforcement learning

1 Introduction

Molecular graph generation must handle variable-sized, permutation-invariant structures and three-dimensional atom coordinates. NeVAE addresses these constraints with a variational autoencoder and gradient-based decoder optimization for property-oriented design.

  • Motivation: Drug design seeks plausible, diverse, and novel molecules with specified properties across a vast, unstructured molecular space.The conventional process remains lengthy, expensive, difficult, and inefficient.
  • Limitations of prior work: Existing molecular graph generators may require fixed atom counts, lack permutation invariance, scale quadratically, limit diversity, and omit atom coordinates.These limitations conflict with molecular graphs having different numbers of atoms and bonds and with molecular properties depending on three-dimensional structure.
  • NeVAE contributions: NeVAE uses a probabilistic encoder with symmetric aggregation to encode variable-sized molecular graphs invariantly to node-label permutations.The encoder aggregates bond features, atoms, and coordinates from different hop distances into a continuous latent space.
  • NeVAE contributions: Its decoder jointly models edges with a single multinomial distribution, enabling permutation-invariant inference with O(l) complexity instead of O(n^2).Here l is the number of true edges and n is the number of nodes.
  • NeVAE contributions: The decoder masks undesirable edges, generates atom coordinates through Gaussian distributions, and supports gradient-based optimization toward desired properties and greater molecular stability.The optimization can target property-oriented generation or spatial configuration of a molecule of interest.
  • Results: 121% higher property values were achieved than by the best-performing competitor in the reported property-oriented generation experiments.The paper also reports stronger validity, novelty, and uniqueness results than several competing methods.

2 Background on Variational Autoencoders

Variational autoencoders combine a probabilistic decoder, latent-variable prior, and approximate probabilistic encoder. Because exact marginal likelihood is typically intractable, training maximizes an evidence lower bound whose quality depends on the encoder’s expressive ability.

  • VAE formulation: A variational autoencoder defines a probabilistic decoder pθ(x|z), latent prior p(z), and approximate inference model qφ(z|x).The decoder maps latent variables to observations, while the encoder maps observations to latent variables.
  • VAE training: Exact marginal log-likelihood requires intractable marginalization over latent variables, motivating optimization of an evidence lower bound.The ELBO provides a variational lower bound on the observed-data log-likelihood.
  • VAE training: The variational lower bound depends on the expressive ability of the approximate inference model qφ(z|x).The encoder is typically modeled as a normal distribution whose mean and variance are parameterized by a neural network.

3 NeVAE: A Variational Autoencoder for Molecular Graphs

NeVAE is a variational autoencoder designed for molecular graphs with variable sizes, permutation invariance, and atom coordinates. Its probabilistic encoder aggregates multi-hop graph information, while its decoder generates node features, edges, bonds, and spatial coordinates.

  • NeVAE represents each molecular graph with atoms, bonds, atom-type features, three-dimensional coordinates, and bond types.
  • Encoder: The encoder assigns each node a latent variable by aggregating information from multiple graph-hop depths through variational inference.The resulting embeddings parameterize the node-wise posterior distribution.
  • Encoder: Symmetric aggregation makes node embeddings invariant to permutations of neighboring node labels.
  • Encoder: A single encoder supports graphs with different numbers of nodes and edges because its weight matrices do not depend on graph size.
  • Decoder: The decoder samples graph size and node latents, then generates atom features, edges, edge weights, and atom coordinates probabilistically.Node count is sampled from a Poisson distribution, while atom coordinates are sampled from Gaussian distributions conditioned on latent representations, neighbors, and bonds.
  • Training objective: Randomized source-node selection for breadth-first edge ordering makes the training objective invariant to permutations of node labels.Theorem 2 states this invariance when the source distribution ζ does not depend on node labels.

4 Property Oriented Molecule Generation

NeVAE adapts its decoder to generate molecules with higher values of a chosen molecular property. It formulates property-oriented decoding through variational inference and approximates the resulting objective with stochastic gradient updates.

  • Property-oriented generation trains the decoder to produce molecules that maximize a selected molecular property, such as water solubility.
  • Objective: The property-oriented decoder balances a loss penalizing low property values against a regularization term involving the decoder log probability.The objective is expressed as S(E, Y, F|Z) = ℓ(E, Y, F) + ρ log p(E, Y, F|Z).
  • Objective: The optimal property-oriented decoder is characterized through a variational-inference optimization problem.
  • Sampling: Rejection sampling can obtain samples from the optimal property-oriented decoder, but may be inefficient when the original decoder assigns low probability to suitable molecules.
  • Optimization algorithm: A stochastic gradient-based algorithm iteratively updates the parameterized decoder using sampled latent variables, generated molecules, and stochastic gradient descent.
  • Optimization algorithm: The log-derivative trick provides an unbiased Monte Carlo estimator for gradients with respect to the property-oriented decoder parameters.

5 Experiments

Experiments on ZINC and QM9 evaluate NeVAE’s generated-molecule quality, latent-space smoothness, and property-oriented decoding. NeVAE produces strong validity, novelty, and uniqueness results, while optimized decoding improves target properties and molecular stability.

  • Experiments use approximately 10,000 drug-like ZINC molecules and approximately 10,000 QM9 molecules, represented as graphs with atom types and bond types.
  • Quality of the generated molecules: NeVAE significantly outperforms GraphVAE, GrammarVAE, CVAE, SDVAE, and ORGAN in validity, while matching JTVAE, CGVAE, and GCPN comparably.
  • Quality of the generated molecules: NeVAE’s latent space supports smooth molecular variation: sampled molecules remain topologically similar, and increasing latent distance produces greater differences from the original.
  • Property oriented molecule generation: 121% higher property values are achieved by NeVAE’s property-oriented decoder than by the best-performing competitor, Bayesian optimization over JTVAE’s latent space.
  • Property oriented molecule generation: For a majority of 100 molecular graphs, the gradient-based method finds spatial configurations with increased stability, meaning decreased potential energy.

6 Conclusion

The conclusion presents NeVAE as a permutation-invariant variational autoencoder for variable-sized molecular graphs with atom coordinates, combined with gradient-based property optimization. It reports stronger discovery of plausible, diverse, and novel molecules and 121% higher property values than several state-of-the-art methods.

  • NeVAE supports permutation-invariant molecular graphs with different numbers of nodes and edges, three-dimensional atom coordinates, and masked local structural and functional properties.
  • The gradient-based algorithm optimizes NeVAE’s decoder to generate molecules maximizing selected properties.
  • NeVAE discovers plausible, diverse, and novel molecules more effectively than several state-of-the-art methods.
  • 121% higher property values are obtained for several properties than those identified by several state-of-the-art methods.
  • Future work includes extending the model from static to dynamic graphs and adapting the methodology to other real-world graphs and molecular-design problems.

A Implementation Details

Implementation details describe a graph variational autoencoder whose learned parameters do not depend on graph size, trained with Adam-based stochastic gradient descent and minibatches grouped by node count.

  • Architecture details: NeVAE’s learned parameters do not depend on the number of graph nodes or edges.
  • Architecture details: The architecture uses linear forms and a sum aggregator, with the aggregator providing a symmetric function.
  • Hyperparameter tuning: Training uses Adam stochastic gradient descent with four tuned hyperparameters: latent dimension D, encoder hop count K, negative samples L, and learning rate lr.
  • Training with minibatch: Training graphs are grouped into minibatches whose graphs have the same number of nodes, allowing a computation graph to be built for each batch size.
  • Hardware and software specifications: Experiments use TensorFlow 1.4.1 on 64-bit Debian with a 16-core Intel Xeon CPU and 512GB RAM.

B Additional details on Bayesian optimization

The Bayesian optimization analysis evaluates latent-space property prediction and molecule discovery under NeVAE and competing models. NeVAE generally performs best, while JTVAE finds a few higher-value molecules for one property but not a sizeable unique set.

  • Bayesian optimization: The analysis trains sparse Gaussian processes on latent representations and property values, then uses expected improvement to discover molecules.The setup uses 100 inducing points sampled from the training set.
  • Evaluation metrics: The evaluation measures SGP log-likelihood and RMSE, plus discovered-molecule property values and fractions of valid and good molecules.Good molecules satisfy y(m) > 0.
  • Results: NeVAE outperforms all baselines in SGP log-likelihood and RMSE, property y1(m), and the fractions of valid and good discovered molecules.For y2(m), NeVAE ranks second after JTVAE.
  • Results: JTVAE finds a few molecules with larger property values than NeVAE, but not a sizeable set of unique molecules with high property values.This comparison is explicitly highlighted for Bayesian optimization.
  • Results: Table 5 reports property-prediction performance using SGPs and property maximization using BO.The table covers LL, RMSE, and Bayesian-optimization outcomes.

C Additional Experiments on Synthetic Graphs

Additional synthetic-graph experiments test whether NeVAE can enforce local topology, learn smooth latent representations, and mimic graph-generating processes.

  • Additional experiments: The experiments evaluate triangle-free generation, interpolation between Kronecker-graph representations, and quantitative imitation of synthetic graph processes.The section includes further tests of graph generation and latent representations.

C.1 Experimental setup

The synthetic-graph setup trains graph VAEs on triangle-free and Kronecker graph datasets containing graphs with up to 1000 nodes.

  • Experimental setup: Two synthetic datasets each contain 100 graphs with up to n = 1000 nodes.One dataset contains triangle-free graphs; the other is a 50%-50% mixture of Kronecker graphs.
  • Experimental setup: The Kronecker mixture uses initiator matrices Θ1 = [0.9, 0.6; 0.3, 0.2] and Θ2 = [0.6, 0.6; 0.6, 0.6].These matrices define the two graph-generating components.
  • Experimental setup: For each dataset, the variational autoencoder is trained by maximizing the evidence lower bound.The trained models are then used to generate three sets of 1000 graphs.

C.2 Quality of the generated graphs

NeVAE generates valid constrained graphs, interpolates between latent graph representations, and quantitatively mimics synthetic graph-generating processes. Its imitation is more accurate for Barabási–Albert graphs, while both graph types achieve correlation and precision above 0.4.

  • C.2 Quality of the generated graphs: 100% validity is achieved when masking is used during both training and testing for triangle-free graph generation.Training-only masking yields 68% validity, while no masking yields 57%; training masking increases training time by 18%.
  • C.2 Quality of the generated graphs: Linear interpolation between latent representations of two Kronecker graphs is used to sample intermediate graphs.Figure 11 shows node latent representations and inferred initiator matrices for the interpolated graphs.
  • C.2 Quality of the generated graphs: The generated graphs are reported as indistinguishable from true Kronecker graphs under the quantitative evaluation in Table 6.The comparison uses the method of Leskovec et al.
  • C.2 Quality of the generated graphs: The evaluation uses rank correlation and precision to assess whether generated graphs plausibly reproduce Barabási–Albert and Kronecker processes.Rank correlation compares likelihood orderings, while precision measures overlap between top and bottom ranked sets.
  • C.2 Quality of the generated graphs: Decoder parameter estimation is evaluated under node-label permutations for different source node distributions.The analysis measures variation in learned decoder weights.
  • C.2 Quality of the generated graphs: NeVAE learns the Barabási–Albert process more accurately than the Kronecker process, while achieving correlation and precision values over 0.4 for both.The authors attribute the difference possibly to the higher complexity of the Kronecker generative process.

C.3 Effect of permuting node labels on decoder parameter estimation

The decoder’s parameter estimation responds to graph structure: degree-based source sampling works best for skewed Barabási–Albert graphs, while uniform sampling suits homogeneous Kronecker graphs. Search depth and runtime also vary by graph family and scale, with the model scaling to approximately 1,000 nodes.

  • Effect of permuting node labels on decoder parameter estimation: The permutation-invariance experiment generates isomorphic relabelings and measures mean differences between estimated decoder parameters across training iterations.It evaluates Barabási–Albert graphs with 1000 nodes and Kronecker graphs with 1024 nodes under three source-node sampling protocols.
  • Effect of permuting node labels on decoder parameter estimation: Degree-based sampling performs best for Barabási–Albert graphs, whereas uniform sampling performs best for Kronecker graphs.The skewed degree distribution of Barabási–Albert graphs repeatedly samples a few high-degree nodes, while Kronecker graphs have a more uniform degree distribution.
  • Effect of K: Low search depths perform better for Barabási–Albert graphs, while higher search depths perform better for Kronecker graphs.The paper relates this contrast to local, sequential generation in Barabási–Albert networks versus global topological features in Kronecker graphs.
  • Scalability: The model scales to approximately 1,000 nodes for both inference and graph sampling.For 1,000-node graphs of average degree 3, one SGD inference iteration takes 67 + 20 seconds with batches of 10, while sampling takes 5 seconds.

C.6 Property oriented graph generation

The property-oriented decoder is optimized to generate graphs satisfying diameter or clustering objectives. Across the evaluated settings, lower values of ρ produce lower diameters, higher clustering coefficients, and higher success rates under a diameter constraint.

  • Property-oriented graph generation: The model is trained on 10,000 Barabási–Albert graphs and optimized for low diameter or high clustering coefficient.The corresponding loss functions are Diameter(G) and 1 − Clustering-coefficient(G).
  • Property-oriented graph generation: Lower values of ρ generate graphs with lower diameters and higher clustering coefficients.Figure 15 compares the distributions of these properties across different ρ values.
  • Property-oriented graph generation: The property-oriented decoder is trained to generate graphs whose diameters remain below a specified upper bound.The loss uses the maximum allowed diameter, and quality is measured by success rate.
  • Property-oriented graph generation: Lower values of ρ yield higher success rates for generating graphs with diameter smaller than D̄.Table 7 reports success rates across different ρ values.
Loading 1802.05283v4…