Source-linked AI summary
Graph Convolutional Policy Network for Goal-Directed Molecular Graph Generation
Jiaxuan You, Bowen Liu, Rex Ying, Vijay Pande, Jure Leskovec
TL;DR
Goal-directed molecular generation must optimize complex molecular properties while obeying chemical rules in a vast, discrete search space. GCPN combines graph representations, reinforcement learning, adversarial rewards, and chemistry-aware constraints to generate molecules toward specified objectives. It reports 61% higher property scores than the best baseline and 184% higher constrained-optimization improvement on average, while maintaining validity and resemblance to realistic molecules.
Problem
Molecular graph generation must optimize desired properties while obeying complex, non-differentiable rules such as chemical valency in a vast chemical space.
Method
GCPN uses graph state representations, policy-gradient reinforcement learning, adversarial rewards, and a chemistry-aware graph-generation environment.
Results
GCPN achieves state-of-the-art results, with 61% higher property scores than the best baseline and 184% higher constrained-optimization improvement on average.
Takeaways & Limitations
GCPN generates goal-directed molecular graphs while maintaining 100% validity and resemblance to realistic molecules across property optimization and targeting tasks.
Takeaways & Limitations
Optimizing empirical property models can produce undesirable molecules with inaccurate predicted properties when prior knowledge is insufficiently incorporated.
Abstract
from arXiv · showhide
Generating novel graph structures that optimize given objectives while obeying some given underlying rules is fundamental for chemistry, biology and social science research. This is especially important in the task of molecular graph generation, whose goal is to discover novel molecules with desired properties such as drug-likeness and synthetic accessibility, while obeying physical laws such as chemical valency. However, designing models to find molecules that optimize desired properties while incorporating highly complex and non-differentiable rules remains to be a challenging task. Here we propose Graph Convolutional Policy Network (GCPN), a general graph convolutional network based model for goal-directed graph generation through reinforcement learning. The model is trained to optimize domain-specific rewards and adversarial loss through policy gradient, and acts in an environment that incorporates domain-specific rules. Experimental results show that GCPN can achieve 61% improvement on chemical property optimization over state-of-the-art baselines while resembling known molecules, and achieve 184% improvement on the constrained property optimization task.
1 Introduction
GCPN addresses goal-directed molecular graph generation by combining graph representations, reinforcement learning, adversarial training, and chemistry-aware constraints. Across three molecule-generation tasks, it achieves state-of-the-art results, including 61% higher property scores than the best baseline and 184% higher constrained-optimization performance on average.
- Motivation: Molecular design remains difficult because chemical space is vast and discrete, while small structural changes can strongly affect molecular properties.Drug-like chemical space has been estimated at 10^23 to 10^60 molecules.
- Motivation: Desired molecular-property objectives are complex and non-differentiable, requiring models that can explore chemical space while generating valid molecular graphs.The desired-property distribution may not necessarily match the distribution represented by training data.
- Present Work: GCPN combines graph representation learning, reinforcement learning, and adversarial training in an end-to-end framework for objective-guided molecule generation.Graph states represent generated molecules, adversarial loss incorporates dataset prior knowledge, and policy gradients optimize the resulting reward.
- Present Work: Graph representations avoid the structural fragility of text-based molecular encodings and allow partially generated graphs to be interpreted as substructures.A single-character perturbation in a text representation can substantially change or invalidate the underlying molecule.
- Present Work: Reinforcement learning represents hard constraints and desired properties through environment dynamics and rewards while enabling active exploration beyond dataset examples.The approach incorporates chemical constraints such as valency and property objectives that cannot be directly included in standard graph-generative objectives.
- Results: 61% higher property scores than the best baseline and 184% higher constrained-optimization performance on average were achieved across the evaluated molecule-generation tasks.The tasks include molecule property optimization, property targeting, and conditional property optimization.
2 Related Work
Prior work applied recurrent or adversarial generative approaches to molecular optimization, often using SMILES strings. GCPN instead uses a graph-based molecular representation within reinforcement-learning approaches.
- Related Work: RNN-based approaches generated SMILES strings while optimizing molecular properties with Monte Carlo tree search or policy gradients.These methods used text-based molecular representations.
- Related Work: Later methods incorporated adversarial loss into reinforcement-learning rewards to encourage similarity to a molecule dataset.The cited approaches included work by Guimaraes et al. and Sanchez-Lengeling et al.
- Related Work: GCPN differs from these text-based approaches by using graph-based molecular representations and their associated structural benefits.The paper contrasts its representation choice with prior SMILES-based methods.
3 Proposed Method
GCPN formulates molecular graph generation as an iterative reinforcement-learning process that optimizes property rewards while enforcing chemical constraints and dataset-based regularization. A graph convolutional policy selects graph modifications, and the environment validates actions, updates states, and returns rewards.
- Problem formulation: The generation problem is formulated as learning an RL agent that iteratively adds substructures and edges to molecular graphs in a chemistry-aware environment.
- Problem formulation: The objective is to maximize a graph property function while incorporating hard constraints and regularization based on example graphs.For molecule generation, chemical valency supplies the hard constraints and an adversarially trained discriminator supplies the distance-based regularization.
- Generation environment: The process is modeled as a Markov decision process, so the current intermediate graph is sufficient for the policy to derive the next action.This avoids conditioning on the full generation trajectory.
- Generation environment: At each step, the environment builds the graph through bond or substructure additions, rejects infeasible actions, and uses chemical rules to update valid intermediate states.The state is the fully observable intermediate graph G_t, initially containing one carbon atom.
- Action space: Actions connect scaffold subgraphs or existing nodes, with the implemented scaffold collection containing all single-node graphs for the different atom types.The action space can also be extended to molecular substructure scaffolds.
- Policy and reward: GCPN uses graph convolutional message passing to encode the extended graph, then predicts a four-component action comprising two node selections, an edge type, and termination.The policy is optimized with PPO, while rewards combine domain-specific property scores, adversarial rewards, and intermediate validity signals.
4 Experiments
The experiments evaluate GCPN on property optimization, property targeting, and constrained property optimization using ZINC-based molecular generation tasks. GCPN outperforms baselines across these tasks while addressing validity, diversity, and realistic-molecule constraints.
- Experimental tasks: The experiments compare GCPN with state-of-the-art and rule-based baselines across property optimization, property targeting, and constrained property optimization.The tasks respectively optimize molecular properties, target specified property ranges, or improve properties while preserving a specified substructure.
- Experimental setup: GCPN is evaluated using ZINC250k molecules, with a maximum of 38 atoms, and uses graph-convolutional policy and discriminator networks.The setup also uses expert pretraining and adversarial training on the ZINC250k dataset.
- Property optimization: 61% average penalized logP improvement over JT-VAE and 186% over ORGAN are achieved, while GCPN also outperforms all baselines on QED optimization.The evaluation reports top property scores and the fraction of chemically valid molecules.
- Limitations: Rare undesirable molecules can receive very high penalized logP predictions when adversarial reward and expert pretraining are reduced, exposing empirical-model generalization issues.The paper addresses this issue by incorporating prior knowledge from realistic known molecules through adversarial training.
- Property targeting: GCPN achieves a significantly higher success rate than baseline methods for generating molecules within specified molecular-weight and logP target ranges, with high diversity.The targeting reward is based on the L1 distance between generated properties and the target-range center.
- Constrained property optimization: 184% higher average penalized logP improvement than JT-VAE is achieved in constrained optimization while preserving similarity to the original molecule's substructure.A single learned policy is applied across all 800 constrained starting molecules.
5 Conclusion
The paper presents GCPN as a graph-based policy network for goal-directed molecular graph generation. It reports consistent performance gains across optimization and targeting tasks while maintaining validity and resemblance to realistic molecules.
- Conclusion: GCPN combines graph state representation and adversarial training for goal-directed molecular graph generation.The approach is also positioned as applicable to graph-generation problems beyond molecules.
- Conclusion: GCPN consistently outperforms state-of-the-art approaches in molecular property optimization and targeting while maintaining 100% validity and resemblance to realistic molecules.The conclusion also identifies possible applications in electric circuits, social networks, and other domain-specific graph optimization settings.
7 Appendix
The appendix specifies validity and chemical-rule filters used in the molecular environment and describes how rewards are scaled and bounded.
- Validity: A molecule is defined as valid when it passes RDKit sanitization checks.Validity is therefore determined by the RDKit-based environment.
- Valency: Valency checks prevent atoms in partially completed graphs from exceeding their element-specific maximum allowable valency.The environment performs this check at each intermediate generation step.
- Chemical filters: The steric strain filter penalizes molecules whose minimized average angle-bend energy exceeds 0.82 kcal/mol.MMFF94 forcefield minimization is used before applying the cutoff.
- Chemical filters: The reactive functional group filter penalizes molecules containing known problematic or reactive groups using the rules adopted for ZINC construction.These rules are implemented in RDKit.
- Reward design: Rewards are linearly mapped to desired ranges for property optimization and target-distance objectives, then thresholded so they do not exceed those ranges.Property optimization uses minimum and maximum ZINC scores, whereas targeting uses absolute distance from the target.