Source-linked AI summary
Retrosynthesis Planning via Worst-path Policy Optimisation in Tree-structured MDPs
Mianchu Wang, Giovanni Montana
TL;DR
Average or cumulative rewards across route paths can overlook the worst path, even though one unsuccessful path invalidates the synthetic route. InterRetro interacts with a tree MDP, identifies successful subtrees ending in commercially available compounds, and uses weighted self-imitation to improve a search-free policy. InterRetro reports state-of-the-art success rate, route length, and sample efficiency, achieving up to 100% benchmark success, reducing steps by 4.9%, and reaching near-SOTA performance with only 10% of training data.
Problem
Average or cumulative rewards across route paths can overlook the worst path, even though one unsuccessful path invalidates the synthetic route.
Source: §1 · p. 2Method
InterRetro interacts with a tree MDP, identifies successful subtrees ending in commercially available compounds, and uses weighted self-imitation to improve a search-free policy.
Source: §1 · p. 2, §3.1 · p. 3, §4.2 · p. 7Results
InterRetro reports state-of-the-art success rate, route length, and sample efficiency, achieving up to 100% benchmark success, reducing steps by 4.9%, and reaching near-SOTA performance with only 10% of training data.
Source: §1 · p. 2Takeaways & Limitations
InterRetro generates high-quality synthetic routes without search at inference time while maintaining state-of-the-art planning performance.
Source: §6 · p. 9Proposed reactions are assumed feasible, and computational methods cannot guarantee practical feasibility without experimental validation.
Source: §6 · p. 9
Abstract
from arXiv · showhide
Retrosynthesis planning aims to decompose target molecules into available building blocks, forming a synthetic tree where each internal node represents an intermediate compound and each leaf ideally corresponds to a purchasable reactant. However, this tree becomes invalid if any leaf node is not a valid building block, making the planning process vulnerable to the "weakest link" in the synthetic route. Existing methods often optimise for average performance across branches, failing to account for this worst-case sensitivity. In this paper, we reframe retrosynthesis as a worst-path optimisation problem within tree-structured Markov Decision Processes (MDPs). We prove that this formulation admits a unique optimal solution and provides monotonic improvement guarantees. Building on this insight, we introduce Interactive Retrosynthesis Planning (InterRetro), a method that interacts with the tree MDP, learns a value function for worst-path outcomes, and improves its policy through self-imitation, preferentially reinforcing past decisions with high estimated advantage. Empirically, InterRetro achieves state-of-the-art results - solving 100% of targets on the Retro*-190 benchmark, shortening synthetic routes by 4.9%, and achieving promising performance using only 10% of the training data.
1 Introduction
Retrosynthesis planning must produce routes whose leaves are purchasable compounds, but existing approaches face inference-time search costs and may not plan full routes independently. The paper reframes planning around the worst-performing path and introduces InterRetro, which learns search-free route generation through self-imitation.
1 Introduction
- Multi-step planning recursively decomposes targets into intermediates, seeking routes whose leaves are purchasable compounds.Early decisions influence later steps and the overall route outcome.Source: §1 · p. 1, Figure 1 · p. 2
- Real-time heuristic search, including MCTS, uses repeated model calls to explore routes, and can require hundreds of calls per molecule.MCTS balances exploration and exploitation; later work sought to improve this trade-off or stabilise value estimates.Source: §1 · pp. 1–2
- Search-trajectory fine-tuning can improve reaction choices but adapts models to molecules encountered during search, potentially impairing independent route construction.That mismatch limits use where fast, search-free planning is desired.Source: §1 · p. 2
- The worst-path objective focuses on the hardest root-to-leaf path because a single unsuccessful path invalidates the synthetic route.Existing approaches often optimise average or cumulative rewards across paths.Source: §1 · p. 2
- InterRetro constructs routes, identifies successful subtrees ending in commercially available compounds, and imitates their decisions to improve its policy.The framework is designed to generate complete routes without search at inference time.Source: §1 · p. 2
- InterRetro is a search-free approach to multi-step planning that outperforms state-of-the-art methods on benchmark success rate, achieving up to 100%, and route length, reducing steps by 4.9%.Source: §1 · p. 2
2 Related Work
Prior work spans single-step reaction prediction and multi-step search, while recent trajectory-based fine-tuning reduces but does not eliminate inference-time search. The paper situates its approach alongside these methods and self-imitation learning.
Single-step Prediction
- Single-step methods predict reactants for a product, but high accuracy on known reactions does not ensure synthesisability from commercially available compounds.Approaches include template-based, template-free, and semi-template strategies.Source: §2 · pp. 2–3
Multi-step Planning
- Multi-step methods use heuristic search to find routes ending in commercially available compounds, with approaches including proof-number search, A*, and MCTS.These methods require extensive inference-time computation, often hundreds of model calls per molecule.Source: §2 · p. 3
- Trajectory-based fine-tuning uses successful planning decisions to improve the single-step model, but still relies on search during inference.It reduces search iterations rather than eliminating the computational burden.Source: §2 · p. 3
Self-imitation Learning
- Self-imitation refines a policy using its own high-return decisions, with support constraints that keep learning close to the data-generating policy.Here, the model is refined using successful decisions that lead to commercially available building blocks.Source: §2 · p. 3
3 Retrosynthesis via Worst-path Optimisation in Tree MDPs
The section formalizes retrosynthesis as worst-path optimization because any branch that fails to reach a building block invalidates the route. It develops value-based self-imitation with optimality and monotonic-improvement guarantees.
3 Retrosynthesis via Worst-path Optimisation in Tree MDPs
- Weighted self-imitation uses successful trajectories to prioritize improvement on failure-prone paths under the worst-path objective.The authors report monotonic improvement guarantees for the learning algorithm.Source: §3 · p. 3, §3.4 · p. 6
3.1 Tree-structured MDPs
- A tree MDP models molecules as states, reactions as actions, and each reaction's set of reactants as the branching successor states.A tree succeeds only when all leaves are commercially available building blocks.Source: §3.1 · p. 3
3.2 Worst-path Objective
- 98.6% of USPTO-50k reactions have three or fewer reactants; tree quality is therefore evaluated by its longest path, with any dead end reducing its value to zero.The cited figure shows the reactant-count distribution.Source: §3.2 · p. 4, Figure 2 · p. 4
3.3 Advantage Estimation
- A learned value function estimates worst-path returns and reaction advantages; a contraction Bellman operator guarantees a unique optimal value and convergent value iteration.A deterministic stationary policy greedy with respect to the optimal value is also optimal.Source: §3.3 · p. 5
3.4 Self-imitation Learning for Retrosynthesis
- Policies remain within the pretrained model's support, while advantage-weighted imitation favors better-than-average reactions and the exact update guarantees monotonic improvement.Restricting policy support is described as preserving chemically valid reactions.Source: §3.4 · p. 6
4 The InterRetro Algorithm
InterRetro grows synthetic trees by recursively expanding non-building-block reactants, then learns from successful branches. A value network estimates worst-path outcomes and guides advantage-weighted policy updates.
4 The InterRetro Algorithm
- InterRetro combines tree-MDP interaction, worst-path value learning, and policy fine-tuning to plan retrosynthesis routes.The algorithm section describes these components as its interaction, learning, and implementation stages.Source: §4 · p. 6
4.1 Environment Interactions
- Starting from a target, the planner expands reactants and queues non-building blocks until all leaves are building blocks or a step limit is reached.The environment interaction returns the constructed synthetic tree.Source: §4.1 · p. 6
4.2 Value Function and Policy Learning
- Successful-subtree branches enter a FIFO replay buffer; the value network learns Bellman targets, while the policy uses clipped exponential advantage weights to imitate decisions.The policy update increases the probability of reproducing high-advantage past decisions.Source: §4.2 · p. 7
4.3 Implementation Details
- Training uses six parallel explorers, 36 trees per iteration, five network updates, a 20,000-branch buffer, and about 48 hours on one RTX A5000 GPU.The reported convergence time is without pretraining the single-step model.Source: §4.3 · p. 7
5 Experimental Results
InterRetro is evaluated against planning baselines on three benchmarks, with results covering success rate, route length, training-data efficiency, and ablations of value estimation and advantage weighting.
5.1 Benchmark Results
- With 500 model calls, InterRetro achieves success rates of 100% on Retro*-190, 98.2% on ChEMBL-1000, and 99.5% on GDB17-1000, outperforming SOTA algorithms across all three test sets.On GDB17-1000, prior PDVN and Self-improve results are 26.9% and 15.0%, respectively.Source: §5.1 · p. 8
- InterRetro outperforms the SOTA by 4.85% in route length on the 138 target molecules that all methods can resolve.Route length is the number of reactions needed to synthesise a target molecule; Table 2 reports average route length on Retro*-190.Source: §5.1 · p. 8, Table 2 · p. 9
- With 10% of the training set, InterRetro achieves near-SOTA performance (92%) in direct generation and reaches SOTA performance with 200 Retro* search iterations.At 100% training data, direct generation achieves a 95.78% success rate; similar trends are reported on ChEMBL-1000 and GDB17-1000.Source: §5.1 · p. 9, Figure 4 · p. 9
5.2 Ablation Studies
- The estimated-depth value function reflects molecule synthesis difficulty, with better capability on molecules requiring less deep synthetic routes.Figure 4b compares estimated synthetic-tree depth with real depth from direct generation.Source: §5.2 · p. 9
- On Retro*-190, direct-generation performance rises from 16% with uniform imitation to more than 90% at β=2 and more than 95% at β≥10.β controls imitation strength; larger values focus imitation on actions with high advantage.Source: §5.2 · p. 9, Figure 4 · p. 9
6 Conclusions and Discussion
The paper concludes that InterRetro combines worst-path optimization with weighted self-imitation to generate routes without inference-time search. It identifies reaction feasibility as an unresolved boundary and notes possible extensions beyond chemistry.
6 Conclusions and Discussion
- InterRetro frames retrosynthesis as worst-path optimization and uses weighted self-imitation to generate high-quality synthetic routes without search at inference time.The paper reports state-of-the-art performance in success rate, route length, and sample efficiency.Source: §6 · p. 9
- The method assumes proposed reactions are feasible, but computational approaches cannot guarantee practical feasibility without experimental validation.The authors propose integrating reaction-feasibility checks into InterRetro.Source: §6 · p. 9
- The weakest-link principle may apply beyond chemistry to sequential decision problems where the least reliable component constrains overall success.The paper gives robust project planning and multi-agent systems as examples.Source: §6 · p. 10
- The authors propose governing access to retrosynthesis models or source code through a Responsible Use License acknowledging responsible and lawful application.They note that models can suggest routes for beneficial drugs and potentially harmful compounds.Source: §6 · p. 10
A Case Studies
Figure 5 shows complete InterRetro synthetic trees for three randomly selected Retro*-190 targets, with yellow circles marking reaction centres. All three illustrated routes use feasible chemical transitions and terminate in commercially available building blocks.
A Case Studies
- The sulfonylated chiral cyclohexanol derivative is assembled in four steps, including sulfonylation, electrophilic bromination, cross-coupling, and N-arylation.Source: A Case Studies · p. 13
- The alkyne-substituted diazine is constructed in two steps through an N-formyl intermediate and SN2 N-propargylation.Source: A Case Studies · p. 13
- The triazolopyridine-fused scaffold is assembled in three steps, including benzylic etherification, cross-coupling, and N-alkylation.Source: A Case Studies · p. 13
B Proofs and Further Theory
The proofs characterize worst-path value through terminal building-block rewards and minimum continuation values, then express policy value using action values over the sampled actions.
B.1 Proof of Proposition 1
- A building block receives reward 1 and ends the path, while non-building-block continuation is evaluated through the minimum next-state value for an action.Source: §B.1 · pp. 13–14
- With deterministic transitions, the worst-case continuation from a non-building-block state under action a is the minimum value among the resulting next states.Source: §B.1 · pp. 13–14
B.2 Proof of Proposition 2
- For Proposition 2, Qπ(s, a) equals r(s) plus the discounted minimum successor value, and the action is sampled from policy π(· | s).Source: §B.2 · p. 14
B.3 Proof of Proposition 3
The proof establishes that the worst-path Bellman optimality operator is a contraction in the max norm. Its unique fixed point supports convergent value iteration and an optimal stationary deterministic policy.
B.3 Proof of Proposition 3
- The operator treats building-block states as terminal with reward 1 and applies a continuation term only at non-building-block states.Source: §B.3 · p. 15
- The Bellman optimality operator is a γ-contraction in the L∞-norm over bounded value functions.Source: §B.3 · p. 15
- Because 0 < γ < 1, value iteration converges to a unique fixed point, which implies an optimal stationary deterministic policy exists.Source: §B.3 · p. 16
B.4 Proof of Proposition 4
The proof shows that weighted self-imitation increases value unless all action advantages are zero. The update weights actions by exponentiated advantage and retains the behaviour policy's support.
B.4 Proof of Proposition 4
- The value-difference identity uses the expected advantage under the updated policy, with advantage defined as Qπi(s, a) − Vπi(s).Source: §B.4 · p. 16
- The weighted imitation update makes the new policy proportional to the old policy times exponentiated advantage and preserves its support set.Source: §B.4 · p. 16
- The weighted average of advantages is strictly non-negative unless all advantages are zero, yielding a non-negative policy value difference.Source: §B.4 · p. 16
C Graph2Edits as Single-step Model
Graph2Edits performs one-step retrosynthesis by autoregressively applying scored graph edits until it predicts reactants. Its reported accuracy and validity support its use as InterRetro’s single-step model.
C Graph2Edits as Single-step Model
- Graph2Edits embeds each intermediate graph with a directed message-passing neural network and scores possible bond edits, atom edits, and termination.The highest-scoring edit is applied repeatedly until Terminate is selected and the intermediates are converted to predicted reactants.Source: C Graph2Edits as Single-step Model · p. 16
- Its training-mined edit vocabulary covers 99.9% of USPTO-50k reactions and encodes stereochemistry and functional leaving groups.Source: C Graph2Edits as Single-step Model · p. 16
- Trained with teacher forcing, the system achieves 55% top-1 exact-match accuracy on USPTO-50k and maintains high validity and diversity for long or stereochemically rich reactions, making it an ideal single-step model in InterRetro.Source: C Graph2Edits as Single-step Model · p. 16