Deep learning for molecular design - a review of the state of the art

Daniel C. Elton; Zois Boukouvalas; Mark D. Fuge; Peter W. Chung

Source-linked AI summary

Deep learning for molecular design - a review of the state of the art

Daniel C. Elton, Zois Boukouvalas, Mark D. Fuge, Peter W. Chung

arXiv:1903.04388v3Mar 11, 2019cs.LGphysics.chem-phstat.ML

TL;DR

Deep molecular generative-modeling research lacks consistent quantitative comparison and reproducible evaluation. This review synthesizes model architectures and molecular representations, reporting varied generation and optimization results while highlighting unresolved training and laboratory-integration challenges.

Problem
Quantitative comparison of molecular generative models remains difficult amid biased comparisons and reproducibility problems.
Method
The review summarizes the state of the art across molecular representation strategies, model architectures, and reward-based optimization approaches.
Results
98% SMILES validity was achieved by one network, while a DNC generator produced 76% valid SMILES versus 24% for an ORGAN RNN generator.
Takeaways & Limitations
Progress depends on better generative-model metrics, reward-function design, and workflows connecting models with synthesis and characterization data.
Takeaways & Limitations
Molecular generative-model research lacks agreement on how to calculate the FID metric.

Abstract

from arXiv · show

In the space of only a few years, deep generative modeling has revolutionized how we think of artificial creativity, yielding autonomous systems which produce original images, music, and text. Inspired by these successes, researchers are now applying deep generative modeling techniques to the generation and optimization of molecules - in our review we found 45 papers on the subject published in the past two years. These works point to a future where such systems will be used to generate lead molecules, greatly reducing resources spent downstream synthesizing and characterizing bad leads in the lab. In this review we survey the increasingly complex landscape of models and representation schemes that have been proposed. The four classes of techniques we describe are recursive neural networks, autoencoders, generative adversarial networks, and reinforcement learning. After first discussing some of the mathematical fundamentals of each technique, we draw high level connections and comparisons with other techniques and expose the pros and cons of each. Several important high level themes emerge as a result of this work, including the shift away from the SMILES string representation of molecules towards more sophisticated representations such as graph grammars and 3D representations, the importance of reward function design, the need for better standards for benchmarking and testing, and the benefits of adversarial training and reinforcement learning over maximum likelihood based training.

1 Molecular representation

Molecular representations determine how chemical structure is encoded for deep learning, balancing uniqueness, invertibility, dimensionality, and structural fidelity. The review describes movement beyond non-unique SMILES strings toward graph-based, tensor, grammar, and 3D representations, each with distinct validity and reconstruction trade-offs.

Representation properties: Representations encode molecules for model training and should capture essential structural information; uniqueness and invertibility are desirable but not required.Uniqueness maps each structure to one representation, while invertibility maps each representation to one molecule.
Graph, tensor, and 3D representations: Graph representations preserve connectivity but lose bond lengths and 3D conformation, while 3D voxelized representations face high dimensionality, sparsity, and difficult invariance handling.Tensor and descriptor representations offer alternatives, but compact fingerprints and descriptors are generally not invertible without large database lookup.
SMILES and strings: SMILES strings encode molecular graphs as token sequences but are non-unique because traversal choices and chemical conventions permit multiple strings per molecule.Canonicalization can standardize strings, but latent spaces may then reflect SMILES grammar more than underlying chemical structure.
SMILES and strings: InChI produced inferior decoding rates and subjective generated-molecule appearance versus SMILES, whereas translating InChI to SMILES yielded more physically meaningful latent spaces.The translation model must extract chemical structure represented in two different ways.

2 Deep learning architectures

The review introduces the mathematical foundations of several popular deep learning architectures and compares their advantages and disadvantages. It assumes basic familiarity with machine learning concepts.

Architectural overview: The section summarizes mathematical foundations and pros and cons of several popular deep learning architectures.A basic familiarity with machine learning concepts is assumed.

2.1 Recurrent neural networks (RNNs)

RNNs generate molecules autoregressively by predicting tokens in sequence, commonly using LSTM or GRU memory cells. The review discusses training limitations, SMILES validity, and reinforcement-learning methods that optimize molecular rewards.

Sequence modeling: RNN molecular generators use token sequences and autoregressive prediction, commonly implemented with LSTM or GRU units containing memory cells.Tokens are typically represented using one-hot encoding before sequence modeling.
Sequence modeling: Autoregressive generation suffers from exposure bias because small prediction errors compound when the model is run on its own generated sequences.Data as Demonstrator and scheduled sampling are described as approaches that compare recursive outputs with training data.
Training: Maximum-likelihood training is differentiable and uses standard gradient methods, but strict long-term dependencies can prevent valid SMILES generation.Teacher forcing was necessary in the molecular autoencoder VAE to capture such dependencies.
Sampling: Sampling temperature trades diversity against plausibility: low temperature produces small variations on training molecules, whereas high temperature increases diversity and nonsensical outputs.The passage states that high-temperature sampling has a higher probability of nonsensical results.
Optimization with reinforcement learning: Reinforcement learning treats each generated token as a discrete action and optimizes expected return, with one generated molecule corresponding to an episode.Common approaches include REINFORCE and policy-gradient methods; HillClimb-MLE was competitive with and occasionally superior to PPO.
RNN results: 98% SMILES validity was reported for one RNN, with tested property distributions matching those in the training data.Reported properties included synthetic accessibility, molecular weight, LogP, and total polar surface area.
Optimization with reinforcement learning: RNNs fine-tuned with reinforcement learning generated molecules targeted toward biological functions, including JAK2 inhibition, 5-HT2A activity, and drug analog generation.The JAK2 system could maximize or minimize inhibition and independently identified 793 commercially available ZINC compounds.

2.2 Autoencoders

Autoencoders learn latent molecular representations that support reconstruction, sampling, interpolation, and property-guided optimization. VAEs regularize this latent space probabilistically, while supervised and conditional variants incorporate molecular properties into generation.

Variational autoencoders (VAEs): VAEs model molecules with latent variables sampled from a prior and decoded through pθ(x|z), while variational inference makes training tractable.The encoder qφ(z|x) approximates the posterior, and training optimizes a lower bound on log p(x).
Variational autoencoders (VAEs): VAE training balances reconstruction quality against similarity between the encoded latent distribution and the prior through KL-divergence minimization.The reconstruction term maximizes pθ(x|z), while the KL term regularizes qφ(z|x) toward p(z).
Variational autoencoders (VAEs): Probabilistic Gaussian latents smooth the representation, making optimization and interpolation between training molecules easier than in classical autoencoders.Noise acts as regularization and reduces holes corresponding to invalid regions of latent space.
Variational autoencoders (VAEs): Molecular VAEs using one-hot SMILES and RNN decoders can have low validity, ranging from ≈75% near known molecules to 4% at random latent points.A similar architecture reported an average valid decoding rate of 0.7%, whereas grammar-based representations achieved higher validity.
Supervised VAEs/AAEs for property prediction & optimization: Supervised VAEs jointly train reconstruction and property prediction, enabling latent-space optimization of targets such as the HOMO-LUMO gap.Property optimization can use Gaussian-process optimization or gradient ascent, and supervised training can correlate a latent direction with the target property.
Supervised VAEs/AAEs for property prediction & optimization: Type 2 and Type 3 supervised VAEs support conditional generation, with Type 3 extending this framework to semi-supervised settings where properties are missing for some molecules.Type 3 models treat unknown properties as latent variables and predict them for molecules without labels.

2.3 Generative adversarial networks (GANs)

GANs generate molecules by training a generator against a discriminator rather than relying only on maximum likelihood. The review highlights their potential for better distribution matching alongside substantial instability, mode-collapse, and hyperparameter challenges.

Generative adversarial networks (GANs): GANs train a discriminator to distinguish generated molecules from training-data molecules, causing the generator to pursue samples that resemble the data distribution.This adversarial objective differs from direct maximum-likelihood generation.
Generative adversarial networks (GANs): Maximum likelihood minimizes forward KL divergence, whereas GAN objectives incorporate a more balanced treatment of distribution differences through adversarial training.The discriminator estimates the data distribution, helping address deviations where the data density is near zero.
Generative adversarial networks (GANs): The original GAN objective provides weak initial gradients when generated and data distributions have little overlap, motivating alternatives such as WGAN.WGAN uses the Wasserstein metric, and its Lipschitz constraint improves gradient reliability and training stability.
Molecular GAN applications: ORGAN adds an objective-reinforcement term to the generator reward, biasing SMILES generation toward desired properties such as druglikeness, synthesizability, and solubility.The mixing parameter λ controls the balance between fooling the discriminator and maximizing the objective function.
The perfect discriminator problem and training instabilities: GAN optimization is inherently unstable because it is a saddle-point problem, requiring balancing strategies such as generator pretraining, update-ratio tuning, or dynamic discriminator control.Mode collapse is another issue, where the generator produces only a narrow range of molecular samples.
The perfect discriminator problem and training instabilities: 76% versus 24% valid SMILES generation was reported for a DNC generator versus the ORGAN RNN generator, while DNC-generated SMILES were on average twice as long.The DNC’s differentiable memory was introduced to strengthen the generator against discriminator dominance.

3 Metrics and reward functions

The review emphasizes that generative-model evaluation lacks consistent standards, because different metrics capture different properties and reported results can be difficult to reproduce. It also presents reward design as central to generating molecules that are diverse, novel, stable, synthesizable, non-trivial, and useful, with targeted property rewards enabling experimental validation.

Metrics for scoring generative models: Reproducibility problems and inconsistent evaluation standards make quantitative comparison of generative models difficult.Published code can produce results that differ from reported results, and supposedly equivalent reinforcement-learning implementations can disagree.
Metrics for scoring generative models: Log-likelihood, distribution-divergence estimates, and human visual ratings measure different aspects, so success on one does not imply success on another.
Metrics for scoring generative models: Fréchet ChemNet Distance adapts image-generation evaluation ideas to molecular models, while MOSES packages it with internal diversity and pharmaceutical metrics.
Metrics for scoring generative models: FID calculations lack agreement about whether generated molecules should be compared with training data or a held-out test set; test comparisons better measure generalization.
Reward functions: Reward functions should jointly address diversity, novelty, stability, synthesizability, non-triviality, and application-specific properties.
Diversity and novelty: Diversity rewards can counter GAN mode collapse, while uniqueness and novelty rewards reduce repeated or training-set molecules; excessive diversity may also be undesirable.
Diversity and novelty: A similarity reward with a tunable cutoff can generate focused libraries near known active molecules without rewarding excessive similarity.
Rewards for good properties: Selecting six RNN-generated molecules for synthesis yielded the desired absorption profiles in 5/6 cases, illustrating the value of physics-based property rewards.

4 Prospective and future directions

Future molecular generative modeling is moving toward chemically grounded representations, richer reward design, and methods that can work with limited data. The review also highlights empirical competition with genetic algorithms, stronger pure reinforcement learning results, and the need to integrate models with laboratory feedback loops.

Prospective and future directions: Much current work remains empirical, while future modeling of large molecules and complex materials may benefit from hierarchical representations.The review identifies proteins, polymers, metal-organic frameworks, and molecular crystals as potential targets for hierarchical generative modeling.
Prospective and future directions: Genetic algorithm methods can still compete with deep learning approaches for molecular optimization.The review notes that recent genetic algorithms remain competitive, including methods using chemically informed transformation rules.
New representation methods: SMILES-based techniques are being replaced by methods operating directly on chemical graphs and three-dimensional structures.These approaches can avoid requiring generative models to learn SMILES syntax and can better reflect chemical structure.
Better reward functions: Reward functions are expected to combine multiple objectives, with multi-objective reinforcement learning identified as a promising direction.The review emphasizes reward design for balancing properties such as stability, diversity, novelty, and synthesizability.
Pure reinforcement learning approaches: Pure reinforcement learning achieved superior optimization performance against Junction Tree VAE, ORGAN, and Graph Convolutional Policy Network approaches on logP and QED.The comparison is reported for molecular optimization using the logP and QED metrics.
Closing the loop: Closing the loop requires feeding synthesis and characterization data back into models and developing laboratory interfaces, software, and cyberinfrastructure.The review identifies rapid feedback and integration with robotic laboratories as important infrastructure challenges.