PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning to Plan Chemical Syntheses

Marwin H. S. Segler, Mike Preuss, Mark P. Waller

arXiv:1708.04202v1cs.AIcs.LGphysics.chem-ph

TL;DR

Computer-aided retrosynthesis has been limited by slow, unsatisfactory search methods, motivating more effective synthesis planning. The paper combines MCTS with neural transformation policies and an in-scope filter, achieving higher speed and solution rates while producing routes that experts did not significantly distinguish from literature routes.

  • Problem

    Computer-assisted synthesis planning has struggled to formalize chemical knowledge and produce satisfactory routes without extensive manual encoding.

  • Method

    The method combines MCTS with neural expansion and rollout policies, an in-scope reaction filter, and transformation rules extracted from reaction data.

  • Results

    MCTS solved more targets than neural and heuristic BFS at tested runtimes, while experts preferred its routes over the baseline and did not significantly prefer literature routes.

  • Takeaways & Limitations

    The data-driven 3N-MCTS approach provides full retrosynthetic pathways within seconds without tedious expert encoding or curation.

  • Takeaways & Limitations

    Natural product synthesis remains beyond the method’s capabilities because training data are sparse, and stereochemical prediction remains an unresolved challenge.

Abstract

from arXiv · show

From medicines to materials, small organic molecules are indispensable for human well-being. To plan their syntheses, chemists employ a problem solving technique called retrosynthesis. In retrosynthesis, target molecules are recursively transformed into increasingly simpler precursor compounds until a set of readily available starting materials is obtained. Computer-aided retrosynthesis would be a highly valuable tool, however, past approaches were slow and provided results of unsatisfactory quality. Here, we employ Monte Carlo Tree Search (MCTS) to efficiently discover retrosynthetic routes. MCTS was combined with an expansion policy network that guides the search, and an "in-scope" filter network to pre-select the most promising retrosynthetic steps. These deep neural networks were trained on 12 million reactions, which represents essentially all reactions ever published in organic chemistry. Our system solves almost twice as many molecules and is 30 times faster in comparison to the traditional search method based on extracted rules and hand-coded heuristics. Finally after a 60 year history of computer-aided synthesis planning, chemists can no longer distinguish between routes generated by a computer system and real routes taken from the scientific literature. We anticipate that our method will accelerate drug and materials discovery by assisting chemists to plan better syntheses faster, and by enabling fully automated robot synthesis.

Automatic extraction of transformation rules from reaction datasets

The system extracts reaction transformations from large reaction datasets and combines neural policies, an in-scope classifier, and MCTS to focus retrosynthetic search.

  • Rule extraction: 12.4 million single-step reactions from Reaxys were used to extract transformation rules for the planning system.The database covers organic and organometallic reactions published from 1771 onward.
  • Neural policies: Neural policy networks predict promising transformations from products and learn contextual information such as functional-group tolerance.The expansion policy uses a deep highway network, while the rollout policy uses a smaller network for faster predictions.
  • Search-space reduction: The expansion policy restricts branching to at most 50 transformations or until cumulative predicted probability reaches 0.995.This focuses expansion on highly likely transformations while reducing the search space.
  • Reaction filtering: An in-scope filter classifies proposed reactions as feasible or infeasible, retaining positively classified transformations as legal moves.The pipeline applies policy-selected transformations, predicts complete reactions, and filters unlikely reactions before adding precursor positions.
  • MCTS integration: The expansion and rollout procedures are incorporated into four iterated MCTS phases to build the retrosynthetic search tree.The two rulesets provide a powerful expansion policy and a faster rollout policy for estimating position values.

1. Selection

Selection traverses the search tree by repeatedly choosing promising positions until reaching a leaf node, while balancing value and exploration.

  • 1. Selection: Selection starts at the target molecule and sequentially chooses promising positions until a leaf node is reached.The algorithm balances selecting high-value positions with exploring previously unexplored positions.
  • 1. Selection: A leaf node visited for the first time is evaluated directly by rollout, whereas a second visit triggers expansion through the expansion policy.

2. Expansion

Expansion generates follow-up retrosynthetic positions for a selected leaf and adds them to the search tree before rollout.

  • 2. Expansion: Expansion selects possible transformations for the current position using the expansion procedure.
  • 2. Expansion: Predicted follow-up positions are added as children of the leaf node, after which the most promising child is selected for rollout.

3. Rollout

Rollout estimates a position’s outcome by recursively applying actions from the rollout network until the synthesis is solved or the maximum depth is reached.

  • 3. Rollout: A solved position receives a reward greater than 1 without further rollout.
  • 3. Rollout: For non-terminal positions, actions are sampled recursively from the rollout network.
  • 3. Rollout: Sampling stops when the state has been deconstructed into building blocks or the maximal depth is reached.

4. Update

After each rollout, 3N-MCTS updates the search tree with the achieved reward and repeats its phases until a time or iteration limit is reached. The final synthesis plan follows the highest-valued retrosynthetic steps until the target is solved or the depth limit is exceeded.

  • 3N-MCTS assigns reward 1 to solved rollouts, partial rewards to partially solved states, and –1 when no solution is found.
  • The four search phases repeat until the time budget or maximal iteration count is exceeded.
  • The final plan repeatedly selects the retrosynthetic step with the highest value until reaching a solved position or exceeding maximum depth.

Evaluating the performance characteristics of 3N-MCTS

3N-MCTS was evaluated against neural and heuristic BFS using time-split targets, route examples, component comparisons, and double-blind chemist preferences. It solved substantially more targets quickly, matched a published route, and produced routes preferred over the heuristic baseline while remaining comparable to literature routes.

  • Experimental design: The evaluation used models trained before 2015 and post-2014 test molecules, with searches directed toward predefined building blocks.The building-block set contained 423,731 molecules, including 84,253 supplier building blocks.
  • Route example: A six-step route to a drug-candidate intermediate matched the published route and was found autonomously in 5.4 seconds.
  • Quantitative performance: MCTS solved more than 80% of targets within 5 seconds, versus 40% for neural BFS and 0% for heuristic BFS.At 60 seconds, MCTS solved 92%, compared with 71% for neural BFS and 4% for heuristic BFS.
  • Quantitative performance: 92% of the test set was solved by MCTS within 60 seconds per molecule, while neural BFS solved 71% and heuristic BFS 4%.
  • Component analysis: MCTS with the expansion policy solved the most targets and required the least average time among the compared search variants.The comparison used a runtime limit of 3×300 seconds with three restarts.
  • Double-blind AB tests: Chemists did not significantly prefer literature routes over 3N-MCTS routes, choosing the computer-generated routes in 57.0% of comparisons.The literature routes were chosen in 43.0% of comparisons across nine target molecules.
  • Double-blind AB tests: Chemists preferred 3N-MCTS routes over heuristic BFS routes in 68.2% of choices, compared with 31.8% for the baseline.The baseline used the same transformation rules but lacked the policy network and in-scope filter.

Discussion

The authors report that 3N-MCTS performs chemical synthesis planning effectively, rapidly producing routes that chemists did not generally prefer less than literature routes. They identify remaining challenges in natural-product synthesis and stereochemistry while anticipating broader machine assistance for chemistry.

  • 3N-MCTS combines Monte Carlo Tree Search with deep neural network policies and an in-scope filter for chemical synthesis planning.The system provides full retrosynthetic pathways within seconds and can be set up from data without tedious expert encoding or curation.
  • 3N-MCTS showed the best performance characteristics among the compared search methods in speed and number of solved problems.
  • Organic chemists did not generally prefer literature routes over routes found by 3N-MCTS in double-blind AB testing.The authors also report that 3N-MCTS proposed more reasonable routes than heuristic best-first search without neural-network guidance.
  • Natural-product synthesis remains beyond the method’s capabilities, partly because training data are sparse in this area.The passage also identifies stereochemistry, including quantitative prediction of enantiomeric or diastereomeric ratios, as an unresolved challenge.
  • The authors anticipate that strong general planning algorithms trained on chemistry’s history will become valuable assistants in real-world chemistry.

Supporting Information

The supporting information formulates retrosynthesis as sequential decision-making over molecular states and explains how MCTS searches this space. Neural-network priors, rollout rewards, and branch objectives guide exploration toward solved routes under computational constraints.

  • Retrosynthesis is represented as a Markov Decision Process with molecular positions as states, transformations as actions, and learned or defined transition and reward functions.The chemical environment is expensive to query directly, so the system uses automatic rule extraction and an in-scope filter to construct an approximate environment model.
  • A state is a set of molecules, and applying a legal transformation replaces one molecule with precursor molecules while retaining the others.
  • A state is solved when every molecule belongs to the specified building-block set; otherwise, it is terminal when no legal actions remain.Changing the building-block set changes state terminality and the associated value function.
  • Each MCTS edge stores an action value Q(s, a), a visit count N(s, a), and a prior probability P(s, a) from the expansion policy.
  • The tree policy uses prior probabilities to explore promising analysis lines first, while repeated visits reduce that prior contribution and expose alternatives.
  • Expansion adds the reduced top50 successor states directly, rollout samples actions recursively, and updates propagate action values and visit counts back to the root.Solved states receive rewards greater than 1, partially solved rollouts receive rewards in [0, 1], and unsuccessful terminal states receive -1.
  • Custom branch objectives can reward shorter, convergent, atom-economic, or confident routes, while the reported reward uses a damping factor k=0.99 and maximal branch length Lmax = 25.The prior policy probabilities bias rewards toward more confident branches.
  • After the search budget is exhausted, the plan is selected greedily by highest action value until a solved terminal state or maximum depth is reached.

Automatic transformation rule extraction

The authors avoid hand-coded chemistry knowledge by automatically extracting transformation rules from reaction data. The resulting symbolic rules remain interpretable and linkable to literature precedent, while their scope and reactivity limitations are addressed through supervised policies.

  • Manual chemistry encoding takes years for modest knowledge bases and is considered to have exhausted its potential as chemical knowledge grows exponentially.The passage estimates that chemical knowledge doubles roughly every 15 years.
  • Transformation rules are extracted automatically and stored in RDKit reaction SMARTS format rather than manually encoded.
  • The expansion rules capture reaction-center atoms, implicit hydrogen, and neighboring-atom counts, while rollout rules additionally include directly neighboring atoms.Rules were extracted from reactions with one, two, or three reactants and a single product.
  • The proof-of-concept extraction does not exclude reactions by low yield or extreme conditions because such criteria are subjective and context-dependent.
  • Automatic rule extraction can leave reaction scope, competing reactivity, and mechanistically needed functional-group effects insufficiently defined.The authors state that supervised policies can address these disadvantages by predicting applicable rules.
  • Symbolic rules communicate results in chemists’ language and link transformations directly to literature precedent.

Policy networks

The system uses neural policy networks to guide retrosynthetic search and an in-scope classifier to filter implausible reactions. These models learn from reported reactions and support rapid reaction-feasibility assessment.

  • Policy networks: The neural policy networks were trained by supervised multiclass classification on literature transformations.Training minimized the negative log likelihood of the literature transform used to make each molecule.
  • Policy networks: The expansion policy represents molecules with processed ECFP4 fingerprints and outputs probabilities over possible transformations.A softmax layer produces the action distribution p(a|m).
  • Policy networks: The rollout policy uses processed ECFP4 fingerprints and a neural network with one hidden dense layer to sample transformations during rollout.Its fingerprint representation is modulo-folded to 8192 dimensions.
  • Policy networks: The in-scope classifier rapidly filters nonsensical reactions that can arise in rule-based systems.The authors leave more sophisticated but slower reaction-prediction approaches to future work.
  • Policy networks: The classifier combines reaction and product fingerprints through separate neural branches and predicts the probability of obtaining the expected product.The branch embeddings are compared by cosine proximity and passed to a sigmoid unit.
  • Policy networks: Negative examples include incorrect rule-generated reactions and mismatched reaction–product pairs created by negative sampling.The rule-generated negatives represent cases where a naive rule-based system would fail.

Performance Evaluation Studies

The evaluation compares heuristic and neural search cost functions and reports CPU-based single-thread searches, while also noting that predictions can run on a laptop.

  • Performance Evaluation Studies: Heuristic breadth-first search sorts branches in a priority queue using a cost function based on the SMILES heuristic.The method adds each branch to the queue and orders branches by cost.
  • Performance Evaluation Studies: Neural breadth-first search calculates branch cost from expansion-policy probabilities for the transformations along the branch.The reported cost is f(b) = product over s∈b of (1−P(a_i|s_i)).
  • Performance Evaluation Studies: Evaluation studies used a single search thread on the CPU of a 24-core commodity cluster node without a GPU.This setup was chosen to make the comparison more meaningful.
  • Performance Evaluation Studies: The system can also perform predictions in real time on a 2013 MacBook Pro.This statement concerns prediction speed rather than the full evaluation setup.

AB testing

The AB tests compare expert preferences among literature, MCTS, and heuristic-BFS routes. The supplied material also includes a fine-tuned MCTS-parameter table and figures concerning route examples and classifier performance.

  • AB testing: 45 postgraduate organic-chemistry students participated in double-blind AB tests.Participants came from institutes in Münster and Shanghai, and neither participants nor conductors knew route origins during testing.
  • AB testing: The study evaluated statistical significance with the Wilcoxon signed-rank test.The test was applied to paired route-preference data.
  • AB testing: 43.0% of 128 AB tests favored literature routes and 57.0% favored MCTS routes, with p=0.26.The null hypothesis that both datasets came from the same source could not be rejected.
  • AB testing: 68.2% of 129 submitted tests preferred 3N-MCTS solutions, compared with 31.8% for heuristic BFS.The experts significantly favored MCTS, with p=0.01277.
  • AB testing: Figure 6 describes an example where chemists preferred a literature Grignard route over an MCTS Seyferth–Gilbert route.The MCTS route was chemically reasonable but used less conventional chemistry, while later key steps were shared.
  • AB testing: Figure 8 shows an autonomous 10-step synthesis route found within 30 s that resembles the published route.The figure concerns a complex intermediate in a drug synthesis.
Loading 1708.04202v1…