Source-linked AI summary
Optimization of Molecules via Deep Reinforcement Learning
Zhenpeng Zhou, Steven Kearnes, Li Li, Richard N. Zare, Patrick Riley
TL;DR
Designing molecules with desired properties is important for material design and drug screening, but molecule optimization remains challenging. MolDQN combines chemistry domain knowledge with deep reinforcement learning and achieves equivalent or better performance than established algorithms in generating molecules with better specified properties.
Problem
Designing new molecules with specific desired properties is a fundamental goal, especially for material design and drug screening.
Method
MolDQN combines chemistry domain knowledge with deep reinforcement learning and defines molecule modifications through atom addition, bond addition, and bond removal.
Results
MolDQN reaches equivalent or better performance than several established algorithms in generating molecules with better specified properties.
Takeaways & Limitations
MolDQN provides a framework for molecule optimization that generates molecules with better specified properties.
Takeaways & Limitations
The experiments and metrics commonly used in the literature, including this work, are inadequate for evaluating and comparing molecule-optimization methods.
Abstract
from arXiv · showhide
We present a framework, which we call Molecule Deep $Q$-Networks (MolDQN), for molecule optimization by combining domain knowledge of chemistry and state-of-the-art reinforcement learning techniques (double $Q$-learning and randomized value functions). We directly define modifications on molecules, thereby ensuring 100\% chemical validity. Further, we operate without pre-training on any dataset to avoid possible bias from the choice of that set. Inspired by problems faced during medicinal chemistry lead optimization, we extend our model with multi-objective reinforcement learning, which maximizes drug-likeness while maintaining similarity to the original molecule. We further show the path through chemical space to achieve optimization for a molecule to understand how the model works.
1 Introduction
MolDQN addresses the costly, validity-challenged, and dataset-biased problem of molecule design by optimizing molecules directly with reinforcement learning. It combines chemically valid actions, value-function learning from scratch, and multi-objective optimization.
- Motivation: Molecule design is a fundamental but time- and cost-intensive goal, particularly for material design and drug screening.Finding a new drug can take years and cost millions of dollars.
- Prior limitations: Existing generative approaches face difficult latent-space optimization, while reinforcement-learning string generators can achieve desirable properties but struggle with chemical validity.Autoencoder-based models may generate almost perfectly valid molecules yet typically require separate optimization, whereas string-based reinforcement learning methods struggle with validity.
- Key differences: MolDQN learns from scratch rather than pre-training on a dataset and supports multi-objective reinforcement learning with user-defined relative objective importance.The paper states that pre-training can limit exploration through dataset bias, while the proposed method can discover molecules with better properties.
- MolDQN framework: MolDQN formulates molecule modification as a Markov decision process and permits only chemically valid actions, ensuring that every generated molecule is valid.Deep Q-Networks solve the decision process using desired molecular properties as rewards.
- Key differences: Unlike prior policy-gradient methods, MolDQN uses value-function learning, which is described as usually more stable and sample efficient where applicable.The introduction attributes high variance in policy-gradient gradient estimates as a motivation for this distinction.
2 Methods
MolDQN formulates molecule optimization as a finite-horizon Markov decision process whose actions are chemically valid atom additions, bond additions, or bond removals. It uses deep Q-learning with time-aware states and discounted rewards to select modifications toward future molecular rewards.
- Molecule modification MDP: Each modification is an atom addition, bond addition, or bond removal, while chemically invalid valence-violating actions are excluded from the action space.The method also permits atoms and bonds to be removed as well as added.
- Molecule modification MDP: Molecule optimization is modeled as an MDP in which each state is a valid molecule paired with the number of steps taken, up to terminal horizon T.Initial states may contain a specific molecule or nothing, with t = 0; states with t = T are terminal.
- Molecule modification MDP: The state transition is deterministic: applying an action produces the resulting molecule with probability 1.Bond additions and removals are constrained by valence and connectivity rules, and selected heuristics restrict ring formation to chemically reasonable cases.
- Reward design: Rewards are assigned at every step and discounted by γT−t, with γ = 0.9 typically used so that final states receive the greatest weight.The authors report that rewarding each step empirically produces better learning performance.
- Deep Q-learning: Deep Q-learning estimates action values with a neural-network approximator, and the limited horizon makes both the MDP and optimal policy time-dependent.Including t in the model performs better experimentally, while policies can favor later rewards when many steps remain and sooner rewards near the horizon.
3 Results and Discussion
MolDQN effectively optimizes molecular properties, outperforming or matching published baselines while revealing limitations of unconstrained penalized logP maximization. Its constrained and multi-objective variants improve targets while preserving molecular similarity and scaffold structure.
- Property optimization: MolDQN optimizes penalized logP and QED, with bootstrapped DQN improving QED over naive DQN while offering no significant penalized-logP advantage.Penalized logP maximization mainly requires adding carbon atoms, so it needs little exploration.
- Property optimization: Compared with GCPN, MolDQN performs better on logP and similarly on QED, partly because it learns from scratch without dataset-limited scope.The comparison included random walk, greedy, ε-greedy, ORGAN, JT-VAE, and GCPN baselines.
- Evaluation limitations: Unconstrained penalized logP is a poor evaluation metric because its value almost increases linearly with atom count, making comparisons unfair without equal atom limits.The paper recommends constraints such as atom count or similarity, or targeting a specific logP range.
- Constrained optimization: Under similarity constraints, MolDQN achieves a highly statistically significant improvement over GCPN for all δ values with t < −8.The bootstrap variant also significantly outperforms the naive model except in the stated exception.
- Multi-objective optimization: Increasing the similarity weight raises similarity to the starting molecule but shifts QED improvements downward, demonstrating a trade-off between objectives.The experiments successfully optimize QED while keeping optimized molecules similar to their starting molecules.
- Multi-objective optimization: Molecules generated under w >= 0.4 possess the same scaffold as the starting molecule, indicating scaffold preservation at sufficiently high similarity weight.The framework also generates molecules satisfying specific ranges with 100% success and multiple target values.
4 Conclusion
MolDQN combines deep reinforcement learning with chemical domain knowledge to optimize molecules, matching or surpassing established algorithms while visualizing its decision process. The authors also argue that current generative-model evaluation is inadequate and should prioritize experimentally verifiable therapeutic properties and meaningful prospective tasks.
- Conclusion: MolDQN combines deep reinforcement learning with chemistry domain knowledge for molecule optimization.The model was developed to optimize molecules by integrating these two components.
- Conclusion: MolDQN achieves equivalent or better performance than several established algorithms when generating molecules with specified properties.The comparison concerns molecule generation targeting better specified properties.
- Conclusion: The authors present a visualization method for inspecting decision making and learning optimization strategies in molecular design.The visualization is intended to facilitate learning a strategy for optimizing molecular design.
- Future work: Future work includes testing different Q-function approximators, such as MPNN34, and searching hyperparameters.The authors hope MolDQN will assist medicinal and material chemists in molecular design.
- Evaluation limitations: The authors argue that common experiments and metrics inadequately evaluate generative models for real-world optimization and hinder fair comparison on meaningful prospective tasks.They specifically criticize logP as a broken metric, note boundary effects in QED, and recommend experimentally verifiable therapeutic properties supported by predictive models and experimental validation.
Data Availability
The study uses the publicly available ChEMBL32 and ZINC29 datasets, generated no new dataset, and provides its code online.
- Data Availability: ChEMBL32 and ZINC29, the datasets used in this study, are available online.
- Data Availability: No dataset was generated during the current study.
- Data Availability: The code is available at https://github.com/google-research/google-research/tree/master/mol_dqn.
Figure Captions · S1 Property Targeting
MolDQN targets specified molecular properties through reward-based reinforcement learning, achieving 100% success on single-property tasks and accurately optimizing simultaneous SA and QED targets. The experiments also show that generated molecules can reach arbitrary property values, although deterministic-policy success rates are not comparable.
- S1.1 Single Property Targeting: Molecular weight and hydrophobicity were targeted because both properties are important to drug absorption.The experiments measured the percentage of molecules within specified molecular-weight and logP ranges.
- S1.1 Single Property Targeting: The single-property reward gives positive feedback inside a target range and penalizes the minimum distance needed to enter it otherwise.For a range [l, u], the reward is based on the molecule’s property value p(m).
- S1.1 Single Property Targeting: 500 generated molecules were evaluated after training from an empty initial molecule over 500 episodes.The reported properties came from those generated molecules.
- S1.1 Single Property Targeting: 100% success rates were achieved on all single-property targeting tasks, outperforming the compared models.The ranges were selected so that few ZINC molecules already fell within them, and MolDQN used no expert pretraining on ZINC.
- S1.1 Single Property Targeting: Success rates are not comparable because the deterministic policy produces only 1–3 unique molecules, but MolDQN can find molecules with arbitrary property values.The limitation applies to the success-rate comparison in Table S1.
- S1.2 Multi-Objective Property Targeting: Multi-objective targeting sought molecules close to specified Synthetic Accessibility and Quantitative Estimate of Druglikeness values.The four target pairs were c1 = (2.2, 0.84), c2 = (2.5, 0.27), c3 = (3.8, 0.84), and c4 = (4.8, 0.27).
- S1.2 Multi-Objective Property Targeting: Explicit rewards on target values produced accurate targeted optimization with reinforcement learning despite substantial property changes from small molecular modifications.The generated molecules showed narrow distributions for both SA score and QED.