Source-linked AI summary
Molecular De Novo Design through Deep Reinforcement Learning
Marcus Olivecrona, Thomas Blaschke, Ola Engkvist, Hongming Chen
TL;DR
The paper addresses molecular de novo design in an enormous search space where desirable compounds must satisfy multiple criteria. It tunes a ChEMBL-pretrained SMILES RNN with policy-based reinforcement learning and augmented episodic likelihood, generating structures with specified properties across several tasks. For DRD2 activity generation, more than 95% of generated structures were predicted active, including experimentally confirmed actives absent from both training models.
Problem
Molecular de novo design must search an enormous chemical space, while inverse-QSAR methods face difficulties mapping suitable descriptors back to molecular structures.
Method
The method fine-tunes a ChEMBL-pretrained SMILES RNN using policy-based reinforcement learning and augmented episodic likelihood combining prior likelihood with a user-defined score.
Results
More than 95% of structures generated toward DRD2 activity were predicted active, including experimentally confirmed test actives absent from both the generative and activity-prediction models.
Takeaways & Limitations
The results indicate that augmented episodic likelihood can tune an RNN for sulfur-free generation, query-structure analogues, and target-activity-guided molecular design.
Takeaways & Limitations
The demonstrated examples use single-parameter scoring functions; applying multiparametric scores for activity, DMPK, and synthetic accessibility requires further research.
Abstract
from arXiv · showhide
This work introduces a method to tune a sequence-based generative model for molecular de novo design that through augmented episodic likelihood can learn to generate structures with certain specified desirable properties. We demonstrate how this model can execute a range of tasks such as generating analogues to a query structure and generating compounds predicted to be active against a biological target. As a proof of principle, the model is first trained to generate molecules that do not contain sulphur. As a second example, the model is trained to generate analogues to the drug Celecoxib, a technique that could be used for scaffold hopping or library expansion starting from a single molecule. Finally, when tuning the model towards generating compounds predicted to be active against the dopamine receptor type 2, the model generates structures of which more than 95% are predicted to be active, including experimentally confirmed actives that have not been included in either the generative model nor the activity prediction model.
1 Introduction
Molecular de novo design addresses an enormous chemical search space by generating structures computationally, but rule-based and inverse-QSAR approaches have important constraints. The study proposes policy-based reinforcement learning with augmented episodic likelihood to tune an RNN toward desirable molecular properties.
- The synthetically feasible molecular space is estimated at 10^60–10^100 compounds, while only a negligible fraction can be synthesized and tested efficiently.
- Rule- and reaction-based methods can generate novel structures but are constrained by the rigidity and scope of predefined transformations.
- Inverse QSAR maps favorable predicted-activity regions to molecular structures, but descriptor selection must support both predictive modeling and translation back to structures.
- RNN generative models learn probability distributions over chemical structures, reducing search to molecules represented as reasonable by the training data.
- The proposed policy-based RL method tunes a ChEMBL-pretrained RNN using augmented episodic likelihood, combining prior likelihood with a user-defined scoring function.
2.1 Recurrent Neural Networks
The model represents molecules as tokenized SMILES sequences and uses an RNN to learn and generate them sequentially. Training maximizes next-token likelihood, while generation samples each token and feeds it back as the next input.
- An RNN maintains information from previous sequence steps, allowing earlier tokens to influence current outputs and future cell states.
- Maximum-likelihood training predicts each target token from preceding tokens and updates network parameters to maximize the correct-token likelihood.
- Back-propagation through time can produce exploding or vanishing gradients, motivating gated architectures such as LSTM and GRU cells.
- After training, generation begins with GO, samples tokens from the learned distribution, feeds each sampled token back as input, and ends at EOS.
- SMILES encodes molecular atoms, rings, and branches as tokens, including multi-character atom types and bracketed environments.
- Different SMILES strings can represent the same molecule, although canonicalization algorithms attempt to assign consistent representations.
2.2 Reinforcement Learning
The study frames SMILES generation as an episodic reinforcement-learning task in which an agent improves its policy to increase expected molecular desirability. It uses policy-based RL because the goal is to learn a stochastic policy starting from a pretrained sequence model.
- In reinforcement learning, an agent selects actions from states according to a policy and receives rewards that determine cumulative return.
- Because molecular desirability is assessed after a complete SMILES, the relevant return is the reward for the completed sequence.
- SMILES generation is an episodic task because it ends when the EOS token is sampled.
- Policy-based RL directly learns a policy, whereas value-based RL first learns expected returns and derives a policy from them.
- The method starts from a prior sequence model and fine-tunes it according to a scoring function, potentially requiring only small policy changes.
- The short, rapidly sampled episodes reduce the impact of variance in gradient estimates.
2.3 The Prior network
The prior network is a maximum-likelihood RNN trained on canonical ChEMBL SMILES to model the syntax and molecular-structure distribution used as the starting policy.
- The prior RNN has 3 layers of 1024 GRUs each and was trained by maximum likelihood on 1.5 million ChEMBL structures.
- Training molecules were restricted to 10–50 heavy atoms and elements from {H, B, C, N, O, F, Si, P, S, Cl, Br, I}.
2.4 The Agent network
The Agent treats SMILES generation as an episodic decision process and fine-tunes a Prior RNN with an augmented likelihood combining prior behavior and a desirability score. Reinforcement learning shifts generation toward desirable structures while retaining the Prior’s learned molecular syntax and distribution.
- Agent formulation: The Agent generates SMILES by choosing sequential characters until an EOS token completes an episode.The sequence of actions represents the generated SMILES, with its probability given by the product of action probabilities.
- Augmented likelihood: The Agent starts from a Prior RNN policy trained on ChEMBL and is optimized using an augmented likelihood based on prior likelihood and a user-defined scoring function.The scoring function rates sequence desirability, while the Prior anchors the updated policy to learned SMILES syntax and molecular structure distributions.
- Optimization: The Agent minimizes L(Θ) = −G to maximize expected return, using on-policy batches of 128 generated sequences and updates after each scored batch.Training used gradient descent with learning rate 0.0005 and gradient clipping to [−3, 3].
- Training effect: Training shifts the Prior’s probability distribution toward structures modulated by desirability.The method uses a policy-based reinforcement-learning formulation and a novel cost function intended to reduce reliance on handwritten rules and overly simple structures.
- Scope of the formulation: Unlike discriminator-based approaches, the investigated tasks keep the scoring function fixed during Agent training.A jointly trained discriminator would instead make the method resemble a Generative Adversarial Network.
2.5 The DRD2 activity model
The DRD2 activity model supplies a predictive scoring model for compounds generated against dopamine receptor type 2 activity. Activity data are split to reduce nearest-neighbor similarity across sets, and an SVM is selected using validation ROC-AUC.
- Activity data: The DRD2 dataset contains 7218 actives with pIC50 > 5 and 343204 inactives with pIC50 < 5.A random subset of 100000 inactive compounds was selected for the modeling procedure.
- Dataset splitting: The activity data are clustered by similarity to centroid molecules before iterative assignment to test, validation, and training sets.The procedure assigns compounds with similarity higher than 0.4 to the same cluster and uses four clusters per iteration.
- Generation process: Figure 5 depicts the conditional probability distribution for the next SMILES token and the character sampled at each generation step.The y-axis shows the probability distribution over the current character, while the x-axis shows the sampled character; E denotes EOS.
- Dataset splitting: The inactive compounds were randomly split using the same set ratios, with less than 0.5% belonging to clusters formed by actives.The cluster-based procedure is used to decrease nearest-neighbor similarity between training and testing sets.
- Activity prediction model: A Gaussian-kernel SVM was trained as the DRD2 activity predictor, with C and Gamma chosen by validation-set ROC-AUC grid search.The model was built in Scikit-learn on the training set.
3 Results and Discussion
The Prior generated mostly valid and novel molecular structures, while Agents were tuned for sulphur avoidance, Celecoxib similarity, and predicted DRD2 activity. Across these tasks, augmented-likelihood training produced targeted structures while retaining diversity and prior-derived chemical patterns.
- 3.2 Learning to avoid sulphur: The Agent was first trained to generate molecules without sulphur and compared with three policy-gradient alternatives.The comparison included episodic and action-basis objectives alongside REINFORCE-based methods.
- 3.3 Similarity guided structure generation: Celecoxib became the predominant generated structure after 200 training steps when the Agent was trained with k = 1 and σ = 15.After 1000 steps, Celecoxib and demethylated Celecoxib each represented about a third of generated structures.
- 3.3 Similarity guided structure generation: Agents based on the canonical Prior converged quickly, whereas reduced-Prior Agents converged more slowly and reached lower average similarity under k = 0.7.With k = 1, similar augmented likelihoods for Celecoxib and its demethylated analogue produced convergence near 0.9 rather than 1.0.
- 3.4 Target activity guided structure generation: The DRD2-predicted active fraction increased from 0.02 for reduced-Prior structures to 0.96 for the corresponding Agent after 3000 training steps.The Agent also recovered experimentally confirmed test actives excluded from both the Prior training set and the activity-model training data.
- 3.4 Target activity guided structure generation: The reduced-Prior Agent generated experimentally confirmed actives unseen by both the generative model and the DRD2 activity model.Recovered test actives varied considerably in structure rather than collapsing to one highly similar predicted-active chemotype.
- 3.4 Target activity guided structure generation: Removing known actives from Prior training reduced overlap with known actives but did not significantly change the fraction of predicted actives.The reduced-Prior Agent generated structures less similar to known actives while remaining equally likely to be predicted active; actual activity depends on the activity model's accuracy.
4 Conclusion
The study extends SMILES-based RNN molecular de novo design with augmented episodic likelihood, demonstrating property-directed generation across several tasks. The authors report promising results for sulphur exclusion, Celecoxib analogue generation, and DRD2 activity targeting, while identifying multi-parameter optimization as future work.
- 4 Conclusion: Augmented episodic likelihood tunes an RNN to generate structures with specified desirable properties without relying on predefined building blocks and rules.The approach combines the Prior likelihood with a user-defined scoring function.
- 4 Conclusion: Sulphur-free generation showed that the Agent could produce solutions reflecting the Prior’s underlying probability distribution, improving over traditional REINFORCE algorithms and previously reported methods.
- 4 Conclusion: The Agent located the intended Celecoxib analogue region even after all Celecoxib analogues were removed from the Prior.
- 4 Conclusion: More than 95% of structures generated for DRD2 were predicted active, including test-set actives absent from both the activity model and the Prior.
- 4 Conclusion: Applying multi-parametric scoring functions that jointly consider properties such as activity, DMPK, and synthetic accessibility requires further research.
Additional file 2 — Generated structures
The additional file provides randomly selected molecular structures generated by the canonical Prior and by Agents trained for sulphur avoidance, Celecoxib analogue design, and DRD2 activity.
- Additional file 2 — Generated structures: The additional file includes randomly selected structures generated by the canonical Prior and different Agents.
Consent for publication
The paper states that the authors have no competing interests and identifies institutional employment, funding, and individual contributions.
- Consent for publication: The authors declare that they have no competing interests.
- Consent for publication: MO, HC, and OE are employed by AstraZeneca, while TB received European Union Horizon 2020 funding through the BIGCHEM program.
- Consent for publication: MO contributed concept and implementation, all authors co-designed experiments and interpreted results, and the manuscript was written, reviewed, edited, and approved by the authors.
1 Additional File 1.
The appendix connects the Agent’s cost function to a REINFORCE formulation and includes generated-structure figures for the Prior and trained Agents.
- 1 Additional File 1.: The appendix describes the REINFORCE update using a stochastic policy, a reward baseline, and cumulative reward from each timestep to the episode’s end.
- 1 Additional File 1.: Assigning zero reward except at the final step expresses the episode-level objective through a terminal reward.
- 1 Additional File 1.: The sum of log policy probabilities across timesteps equals the Agent likelihood for the generated sequence.
- 1 Additional File 1.: Choosing the final reward as the squared Prior–Agent log-likelihood difference divided by Agent log likelihood recovers the initial cost function.
- 1 Additional File 1.: The appendix contains figures of randomly selected structures generated by the canonical Prior and Agents trained for sulphur avoidance, Celecoxib analogues, and DRD2 activity.