PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning To Navigate The Synthetically Accessible Chemical Space Using Reinforcement Learning

Sai Krishna Gottipati, Boris Sattarov, Sufeng Niu, Yashaswi Pathak, Haoran Wei, Shengchao Liu, Karam M. J. Thomas, Simon Blackburn, Connor W. Coley, Jian Tang, Sarath Chandar, Yoshua Bengio

arXiv:2004.12485v2cs.LGcs.AI

TL;DR

Many de novo drug-design approaches do not ensure that generated molecules are synthetically feasible or provide their synthesis routes. PGFS uses reinforcement learning to search through forward chemical reactions, and the authors report state-of-the-art QED and penalized clogP performance alongside an in-silico evaluation on three HIV targets.

  • Problem

    Many de novo drug-design methods do not explicitly account for synthetic feasibility, and separate synthesis-planning tools do not make the generator learn synthesizability.

    Source: §1 · p. 2

  • Method

    PGFS treats forward synthesis as a sequential decision process, using RL to select reactants and reaction transformations that yield products with desired properties.

    Source: §1 · p. 2

  • Results

    PGFS achieves state-of-the-art performance on QED and penalized clogP tasks, with higher predicted activity against three HIV-related targets than existing benchmarks.

    Source: §1 · p. 3, §5 · p. 13

  • Takeaways & Limitations

    The authors conclude that PGFS navigates synthesizable small-molecule space and generates compounds with full synthetic routes.

    Source: §1 · p. 2, §5 · p. 13

Abstract

from arXiv · show

Over the last decade, there has been significant progress in the field of machine learning for de novo drug design, particularly in deep generative models. However, current generative approaches exhibit a significant challenge as they do not ensure that the proposed molecular structures can be feasibly synthesized nor do they provide the synthesis routes of the proposed small molecules, thereby seriously limiting their practical applicability. In this work, we propose a novel forward synthesis framework powered by reinforcement learning (RL) for de novo drug design, Policy Gradient for Forward Synthesis (PGFS), that addresses this challenge by embedding the concept of synthetic accessibility directly into the de novo drug design system. In this setup, the agent learns to navigate through the immense synthetically accessible chemical space by subjecting commercially available small molecule building blocks to valid chemical reactions at every time step of the iterative virtual multi-step synthesis process. The proposed environment for drug discovery provides a highly challenging test-bed for RL algorithms owing to the large state space and high-dimensional continuous action space with hierarchical actions. PGFS achieves state-of-the-art performance in generating structures with high QED and penalized clogP. Moreover, we validate PGFS in an in-silico proof-of-concept associated with three HIV targets. Finally, we describe how the end-to-end training conceptualized in this study represents an important paradigm in radically expanding the synthesizable chemical space and automating the drug discovery process.

1 Introduction

Prior de novo drug-design approaches often do not ensure synthetic feasibility, while PGFS searches through sequential chemical reactions to generate molecules with desired properties. The authors report state-of-the-art QED and penalized clogP performance and higher predicted activity against three HIV-related targets than existing benchmarks.

1 Introduction

  • Most de novo drug-design methods do not explicitly account for synthetic feasibility, and heuristic scores or separate synthesis-planning pipelines do not ensure that generated molecules can be produced.The authors note that accessibility heuristics can be exploited and that separate synthesis planning leaves synthesizability outside the generator's learning process.Source: §1 · p. 2
  • PGFS achieves state-of-the-art performance on QED and penalized clogP tasks and generates molecules with higher predicted activity against three HIV-related targets than existing benchmarks.The HIV-target evaluation is described as an in-silico proof of concept.Source: §1 · pp. 2–3
  • PGFS frames molecule generation as sequential selection of reactants and reaction transformations, with product molecules as states and reactant choices as actions.The RL agent seeks task-specific product properties while producing a synthetic route.Source: §1 · p. 2

2 Related Work

Related work spans evolutionary, generative, graph-modification, and reaction-based approaches to molecular optimization. The section contrasts their treatment of synthetic feasibility and notes that common benchmarks only partly resemble real drug discovery.

2.1 Genetic Algorithms

  • Genetic algorithms optimize molecular graphs or strings, but typically rely on synthetic-feasibility heuristics and need separate retrosynthesis models or expert review.Graph-based approaches use fragment crossover and mutation; newer approaches also use string representations.Source: §2.1 · p. 3

2.2 Deep Generative Models

  • Deep generative models, including RL-enhanced approaches, optimize molecular properties but share genetic algorithms’ synthetic-feasibility challenge.The cited work covers single- and multi-objective settings.Source: §2.2 · p. 3

2.3 RL-Based Graph Modification Models

  • RL-based graph-modification methods build molecules by adding or removing atoms, bonds, or subgraphs, but do not guarantee synthetic feasibility.The evolving molecule is the state, and actions are selected to optimize its properties.Source: §2.3 · p. 3

2.4 Forward Synthesis Models

  • Forward-synthesis methods use reaction templates and available starting materials, but enumeration can limit efficient exploration; PGFS combines biased reactant selection with multi-step synthesis.Earlier approaches include evolutionary optimization and reaction-vector methods, while other workflows either bias single-step selection or use random multi-step reactants.Source: §2.4 · pp. 3–4

2.5 Benchmarking De Novo Drug Design

  • Common de novo benchmarks assess distribution similarity or objective optimization, but their metrics bear little resemblance to real drug-discovery projects.The section describes MOSES as focusing on property distributions and GuacaMol on goal-directed tasks such as rediscovery and scaffold hopping.Source: §2.5 · p. 4

3 Methods

PGFS casts molecule generation as sequential forward synthesis, selecting reaction templates and compatible commercially available reactants to optimize product properties. It combines hierarchical continuous-action RL with reaction prediction and reward-based product selection, then trains the actor and critic using TD3-style updates.

3.1 Reinforcement Learning

  • PGFS searches chemical space through sequences of reactions on commercially available reactants, selecting templates and compatible reactants to maximize product-property rewards while providing synthesis recipes.Its hierarchical action selection chooses a reaction template first and a compatible reactant second, reducing the action space relative to choosing both simultaneously.Source: §3.1 · p. 4
  • To handle tens of thousands of candidate reactants, PGFS maps continuous actor outputs to discrete molecules with k-nearest-neighbor lookup and uses TD3.The actor has two learnable networks for the two action levels; k-NN is part of the environment, and the critic is not used in forward propagation.Source: §3.1 · p. 4

3.2 Overview

  • Reaction templates constrain the reactant search to molecules with the required substructure, and the method forbids pairs that can participate in multiple reactions.When multiple products remain possible, one is selected randomly; tens of thousands of reactants may still remain, motivating a continuous-action MDP.Source: §3.2 · p. 5
  • The actor selects a reaction template and continuous reactant-feature action; the environment tests nearby reactants, predicts products, scores them, and returns the highest-reward product.The episode ends when the maximum reaction-step count is reached or the next state has no valid templates.Source: §3.2 · p. 5, Figure 2 · p. 6

3.2.1 Training Paradigm

  • Training updates the critic with a one-step temporal-difference target and trains the actor to maximize the critic’s estimated return.Target networks supply the next-state estimate, while the actor loss is the negative Q-value of its chosen action.Source: §3.2.1 · pp. 5–6
  • Training adds clipped noise to smooth the target policy, uses the minimum of two critics for the TD target, and delays actor and target-network updates.The f network also receives cross-entropy supervision from the reaction template obtained for the current reactant.Source: §3.2.1 · p. 6

4 Experiments

The experiments evaluate PGFS with HIV-target QSAR models and compare its reward optimization with random search and prior approaches. PGFS achieves the strongest reported benchmark scores, with caveats about interpreting QED and penalized clogP as drug-discovery measures.

4.1 Predictive Modeling

  • QSAR models predicted pIC50 for CCR5, HIV integrase and HIV-RT from ChEMBL data using repeated five-fold cross-validation.The authors say practical use requires more rigorous curation, stricter applicability-domain formulation and scaffold- or time-split testing, depending on the target.Source: §4.1 · pp. 7–8, §4.2 · p. 9

4.2 Data And Representations

  • Across experiments, MolDSet worked best as input to the k-NN module and thus as actor output, while ECFP worked best for the f, π, and Q networks; reported results used only these two features.MolDSet consists of 35 features; the ECFP4-like Morgan fingerprints are 1,024-bit vectors with radius 2.Source: §4.2 · p. 8
  • The reaction environment used 150,560 Enamine building blocks and 97 templates, after filtering templates with fewer than 50 available second reactants.The templates do not specify stereocenters formed during reactions, limiting prediction of stereoselectivity.Source: §4.2 · p. 9

4.3 Experimental Settings

  • Episodes allowed five reaction steps; random search sampled reactants and compatible templates, using 400,000 reaction steps for the reported benchmark.The experiments used k=1 in the k-NN module during training and inference.Source: §4.3 · p. 9

4.4.1 Baseline comparison

  • On the same 2,000 initial reactants, PGFS significantly outperformed random search on QED and penalized clogP; HIV-target reward distributions shifted higher, including after applicability-domain filtering.Figure 3 shows the QED and penalized clogP scores by step and their five-step maxima; Figure 4 shows HIV-target scores with and without filtering.Source: §4.4.1 · p. 9, Figure 3 · p. 10, Figure 4 · p. 11

4.4.2 Quantitative performance benchmark

  • PGFS achieved the highest maximum scores across defined tasks and the highest HIV-target maximum rewards and top-100 mean rewards among the compared methods in this study.The authors caution that QED and penalized clogP are insufficient for real drug-discovery comparisons: building blocks already include a QED score of 0.948, while molecule size and allowed reaction steps affect penalized clogP.Source: §4.4.2 · p. 12, Table 2 · p. 12, Table 3 · p. 12
  • The authors caution that QED and penalized clogP are insufficient real-drug-discovery comparisons because molecular size and allowed reaction steps affect the scores.The initial Enamine building blocks already include a QED score of 0.948.Source: §4.4.2 · p. 12, Table 2 · p. 12
  • Figure 5 presents a high-predicted-activity CCR5 compound beside the most similar training-set structure, and Figure 6 shows its synthesis path.The comparison reports predicted pIC50 for the proposed compound and experimental pIC50 for the training-set structure.Source: §4.4.2 · p. 12, Figure 5 · p. 13, Figure 6 · p. 13

5 Conclusion and Future Work

PGFS achieves state-of-the-art performance on QED and penalized clogP tasks and shows stable learning and enrichment of high-scoring compounds versus benchmarks. The authors propose policy-gradient and planning variants as future directions that could improve exploration.

5 Conclusion and Future Work

  • PGFS achieves state-of-the-art performance on QED and penalized clogP tasks and shows significant enrichment in high-scoring compounds versus existing benchmarks.The authors also report stable learning across all tasks used in the study.Source: §5 · p. 13
  • The authors propose a second policy gradient for template selection and stopping, or continuous-action RL and hybrid planning approaches, as future work that could improve chemical-space exploration.Source: §5 · p. 13

A Choice of Parameters

The parameter choices combine an initial random-action bootstrapping phase and noisy training actions with delayed network updates. Experiments compare molecule representations and report ECFP state features with RLV2 action features as the best combination for inference reward.

A Choice of Parameters

  • For the first 3,000 training steps, the agent randomly selects valid templates and reactants; afterward, Gaussian noise with standard deviation 0.1 is added to policy actions, but not at inference.Source: A Choice of Parameters · p. 17
  • Target-policy noise is scaled by 0.2 and clipped to -0.2 to 0.2, while actor and target-network updates occur once per two critic updates.Source: A Choice of Parameters · p. 17
  • ECFP state features with RLV2 action features performed best in inference reward for shaped-QED and shaped-HIV-CCR5 tasks.Figures 7 and 8 plot inference reward and network losses across feature combinations as training progresses.Source: A Choice of Parameters · p. 17

B Examples of Proposed Compounds And Their Synthetic Paths, Additional Information On Reaction Templates And Building Blocks

The appendix presents PGFS compounds with maximum QED, penalized clogP, HIV-integrase, and HIV-RT scores alongside their proposed synthetic routes. It also shows feature-combination training plots and the distributions of available reaction templates and second reactants in the building-block set.

B Examples of Proposed Compounds And Their Synthetic Paths, Additional Information On Reaction Templates And Building Blocks

C.1 Training and Cross-validation procedure

The HIV-target QSAR pipeline trains LightGBM regressors on curated measured pIC50 data using repeated five-fold cross-validation and feature selection. Ensembles of 25 models per target provide predictions, with applicability domain estimated using distances to training-set neighbors.

C.1 Training and Cross-validation procedure

  • The models are trained on measured pIC50 data for three HIV-related targets using 199 initial RDKit descriptors and five-fold cross-validation repeated five times.The QSAR pipeline follows Skalic et al.; max-bin and min-split-gain are set to 15 and 0.0001.Source: §C.1 · p. 21
  • Each HIV target's predictions use an ensemble of 25 LightGBM models produced by five-fold cross-validation repeated five times.Feature selection removes correlated features and uses recursive feature elimination based on LightGBM feature importances.Source: §C.1 · pp. 21–22
  • Applicability domain is estimated by comparing a compound's mean distance to its K nearest training compounds with the training-set mean distance plus 1.5 standard deviations.K is the square root of dataset size: 41 for CCR5, 27 for HIV-Int, and 37 for HIV-RT.Source: §C.1 · p. 22

C.2 Formulas used to calculate performance metrics in Table 1 of the manuscript

This section defines the coefficient of determination and its adjusted form for reporting performance. It specifies the sums-of-squares terms and how feature count is handled in fold-level and aggregated statistics.

C.2 Formulas used to calculate performance metrics in Table 1 of the manuscript

  • R^2 is identified as the coefficient of determination, with total, explained, and residual sums of squares and an error term named.The supplied passages introduce the calculation but do not provide a complete readable formula for R^2.Source: §C.2 · p. 22
  • The section presents a separate calculation for adjusted R^2 based on R^2, sample count, and feature count.The adjusted form uses n for dataset instances and p for features.Source: §C.2 · p. 22
  • Adjusted R^2 accounts for dataset size and feature count, using the selected model's feature count per fold and an average across 25 models for aggregated statistics.Here, n is the number of instances and p is the number of features; for aggregated statistics, p is approximated from the 25 models produced by five-fold cross-validation repeated five times.Source: §C.2 · p. 22

C.3 QSAR data collection and curation

The HIV-target QSAR datasets were drawn from ChEMBL25 and curated using structure, measurement-consistency, molecular-weight, and feature-based outlier filters. The resulting datasets contained 1,719 compounds across CCR5, HIV integrase, and HIV-RT.

C.3 QSAR data collection and curation

  • Curation standardized structures with MolVS, removed salts and potential duplicates, averaged differing pIC50 measurements, and discarded compounds with measurement standard deviation above 1.0.Measurements sharing a Canonical SMILES were treated as repeated measurements of the same compound before averaging.Source: §C.3 · p. 22
  • The dataset excluded compounds outside molecular weights 100–700 and outliers with Z-scores above 3.0 for five specified molecular features.The features were QED, MolWt, FpDensityMorgan2, BalabanJ, and MolLogP.Source: §C.3 · p. 22
  • The final curated QSAR datasets included 775 compounds for HIV integrase and 1,392 for HIV-RT, and were reported in the project’s GitHub repository.Only structures with available pChEMBL values corresponding to pIC50 were selected.Source: §C.3 · p. 22

D Experimental setup for GCPN, JTVAE and MSO

The experiments followed published setups for GCPN, JT-VAE, and MSO, with implementation details specified for each baseline. The section also describes cross-validated QSAR predictions for three HIV targets.

D Experimental setup for GCPN, JTVAE and MSO

  • The experiments were based on the published setups for GCPN, JT-VAE, and MSO [71], [31], and, respectively.The MSO configuration was described as the GuacaMol benchmark setup [10].Source: D Experimental setup for GCPN, JTVAE… · pp. 22–23
  • Figure 14 plots predicted against measured pIC50 values for CCR5, HIV-RT, and HIV integrase using repeated five-fold cross-validation.Each plotted prediction averages five models that did not include that compound in training.Source: Figure 14 · p. 23
  • GCPN ran on a 32-core machine for 30 hours using the authors' hyperparameters, training code, and dataset; JT-VAE used the authors' pretrained weights and five Bayesian-optimization iterations.For JT-VAE, each iteration proposed 50 latent vectors, and ten independent runs were combined.Source: D Experimental setup for GCPN, JTVAE… · p. 23
  • MSO used a particle swarm with 200 particles for 250 iterations and 40 restarts per HIV reward function.The setup followed the authors' GuacaMol benchmark configuration.Source: D Experimental setup for GCPN, JTVAE… · p. 23

Related papers

Loading 2004.12485v2…