PaperlayerView exact revision on arXiv

Source-linked AI summary

MolHop: Scaffold-Hopping Molecular Design with Agentic Reinforcement Learning

Yanyi Su, Chenyan Wang, Yulei Gong, Yeyun Chen, Xiaohui Yang, Jun Cheng

arXiv:2610.19920v1cs.LGcs.CL

TL;DR

Molecular design agents often optimize near a reference molecule, but replacement tasks require distinct scaffolds while retaining target properties. MolHop combines hierarchical scaffold navigation, molecular editing, and trajectory memory; MolHop-9B outperforms generation baselines on drug and commercial tasks, while finding candidates with larger scaffold differences.

  • Problem

    Many molecular design agents focus on local refinement, while replacement tasks require exploring structurally distinct scaffolds and maintaining target properties.

    Source: §1 · p. 1, §2.1 · p. 3

  • Method

    MolHop organizes about 1.6 million scaffolds hierarchically and combines scaffold navigation, molecular editing, and trajectory memory in an agentic framework.

    Source: §1 · p. 2, §3 · p. 3

  • Results

    MolHop-9B achieves joint@H1 of 55.5% on the ChEMBL test set and 73.5% on the Enamine test set, outperforming one-shot generation and general LLM baselines.

    Source: §4.3 · p. 5

  • Takeaways & Limitations

    Experiments show that structured chemical search spaces and action interfaces support cross-scaffold exploration by frozen LLM agents, while agentic reinforcement learning trains policies coordinating scaffold selection and molecular editing.

    Source: §5 · p. 9

Abstract

from arXiv · show

Molecular design is often formulated as an optimization problem over existing molecular spaces. However, molecular optimization and the discovery of structurally distinct replacement molecules represent two fundamentally different search problems. LLM-based molecular design agents typically perform the former, conducting local refinement within the neighborhood of existing structures to improve target properties through structural modifications. In contrast, the latter requires exploration of a broader chemical space to discover molecules with entirely different scaffolds while preserving the functional characteristics of reference molecules. We propose MolHop, an agentic framework for cross-scaffold molecular discovery that enables models to navigate and edit within a pre-organized chemical space. MolHop organizes approximately 1.6 million molecular scaffolds from commercial molecules into a coarse-to-fine hierarchical action space. The agent selects new exploration regions through scaffold navigation actions and adjusts candidate properties through molecular editing. The framework maintains historical trajectory memory to support long-horizon search processes. On drug molecule and commercial compound tasks, connecting Grok-4.6 and DeepSeek-Flash-V4.1 to the MolHop framework substantially improves their cross-scaffold joint success rates. We develop MolHop-9B through agentic reinforcement learning, and MolHop-9B outperforms generative baselines including GPT-5.6-sol, Claude-5-Opus, and Gemini-3.1-Pro, while generating candidate molecules with larger scaffold differences from the reference molecules. These results demonstrate that effective molecular discovery beyond the traditional optimization paradigm requires structured chemical and action spaces, enabling agents to explore new structural regions rather than merely refine existing molecules.

1 Introduction

MolHop treats cross-scaffold discovery as a search problem distinct from local molecular optimization, combining scaffold navigation, molecular editing, and trajectory memory. The paper reports improved cross-scaffold discovery with MolHop for frozen models and for MolHop-9B trained through agentic reinforcement learning.

1 Introduction

  • Cross-scaffold discovery differs from optimization because it seeks structurally distinct alternatives that maintain multiple target properties.New scaffold starting points can alter multiple physicochemical properties, requiring repeated structural exploration and property recovery.Source: §1 · p. 1
  • The paper frames scaffold hopping as an explicit search operation, rather than an incidental outcome of local molecular optimization.The main contributions are a hierarchical scaffold action space, a long-horizon exploration harness, and agentic reinforcement learning for cross-scaffold discovery.Source: §1 · pp. 2–3
  • MolHop organizes approximately 1.6 million commercial molecular scaffolds into a hierarchical action space for navigation, alongside molecular editing and trajectory memory.Its navigation actions select exploration regions, editing actions adjust candidate properties, and memory records explored scaffolds, actions, and property feedback.Source: §1 · pp. 2–3
  • MolHop-9B outperforms generative baselines on drug and commercial molecule tasks while finding candidates with larger scaffold differences from references.The framework also improves cross-scaffold joint success rates when Grok-4.6 and DeepSeek-Flash-V4.1 use it without additional training.Source: §1 · p. 2

2 Related Work

Prior molecular design agents commonly use iterative molecular edits guided by property feedback, while distant scaffold transitions are not usually explicit search decisions. MolHop addresses this gap with scaffold navigation before local optimization.

2.1 Goal-directed molecular search and LLM-based molecular agents

  • Existing molecular agents typically refine candidates through sequential editing actions guided by property feedback.Earlier goal-directed approaches used reinforcement learning or evolutionary search, with later work improving representations, sampling, and generation policies.Source: §2.1 · p. 3
  • Most existing approaches do not explicitly model transitions between distant scaffold regions as search decisions.This leaves scaffold replacement and alternative-molecule tasks—where distinct structures must retain target properties—as a gap addressed by MolHop's scaffold-navigation action.Source: §2.1 · p. 3

3 Methods

MolHop combines a hierarchical scaffold space, structured navigation and editing actions, and environment-maintained trajectory memory. Together, these components let agents select new structural regions, adjust candidate properties, and use historical feedback during long-horizon search.

3 Methods

  • MolHop organizes approximately 1.6 million valid Bemis–Murcko scaffolds from the Enamine Screening Collection into an H1–H7 hierarchy for multiscale navigation.Higher-level nodes support exploration across regions, while lower-level nodes support finer scaffold selection; the representation progressively restores connectivity, atom types, and bond types.Source: §3 · p. 3
  • For cross-scaffold tasks, MolHop records the reference molecule's H1 region and checks the final candidate's region for the required transition.The hierarchy construction and scaffold query process are described in the appendix.Source: §3 · p. 3
  • The structured action interface lets agents alternate between scaffold exploration and molecular editing using environment-returned constraints and feedback.Its four action categories are scaffold navigation, molecular inspection, local editing, and search control; editing includes fragment, atom-and-bond, and ring changes.Source: §3 · pp. 3–4
  • Environment-maintained trajectory memory records candidate states, property feedback, scaffold visits, editing history, and search budget for subsequent decisions.Memory is updated from interaction events and returned with tool feedback; it supports decisions within a single episode, with the agent-facing state shaped by the context budget.Source: §3 · p. 4

4 Experiments

MolHop is evaluated on drug-molecule and commercial-compound tasks requiring scaffold transition alongside property matching and lower synthetic accessibility. The experiments compare frozen LLMs with and without the harness, train MolHop agents through SFT and GRPO, and test their performance, action patterns, component contributions, and behavior beyond library coverage.

4.2 Effect of MolHop Harness on Frozen LLM Agents

  • With the harness, Grok-4.6 rises from 41.0% and 36.5% to 64.0% and 69.0%, while DeepSeek-Flash-V4.1 rises from 15.5% and 17.5% to 52.5% and 60.5%.Both scaffold Tanimoto similarity and MCS ratio decrease among successful candidates.Source: §4.2 · p. 5

4.3 Agentic RL Enables Cross-Scaffold Molecular Design

  • MolHop-9B achieves joint@H1 of 55.5% on ChEMBL and 73.5% on Enamine, improving over the best generation model by 2.0 and 12.0 percentage points.The comparison covers generation models and oracle-driven optimization methods.Source: §4.3 · p. 5, Table 2 · p. 6
  • MolHop-9B scaffold Tanimoto medians are 0.123 and 0.115, while MCS medians are 0.660 and 0.568 on ChEMBL and Enamine, respectively.MolHop models lie closer to the higher-success, lower-scaffold-retention region of the joint@H1–MCS plots.Source: §4.3 · p. 6, Figure 2 · p. 6
  • Across model scales, joint@H1 rises from 28.5% to 48.0% to 55.5% on ChEMBL and from 33.5% to 52.5% to 73.5% on Enamine.The 2B, 4B, and 9B agents are trained using SFT followed by GRPO in the MolHop environment.Source: §4.1.3 · p. 4, §4.3 · p. 6

4.4 Agent behavior under structured action space

  • MolHop-4B and MolHop-9B shift between scaffold selection and editing, whereas general LLM agents typically continue editing after initial scaffold selection.MolHop-2B mainly searches by navigation and selection, with relatively few subsequent edits.Source: §4.4 · p. 7, Figure 3 · p. 7

4.5.1 Effect of SFT on Tool-use Ability

  • After SFT, Valid Episode Rate rises from 21.8%, 18.1%, and 4.6% to 91.1%, 97.8%, and 97.8% for the 2B, 4B, and 9B models.Budget Overflow Rate and Invalid Tool Invocation Rate also decrease substantially.Source: §4.5.1 · p. 7, Table 3 · p. 7

4.5.2 Ablation of MolHop Components

  • Removing editing, trajectory memory, or the scaffold library reduces final performance, with larger decreases when memory or the hierarchical library is removed.Without the scaffold library, training reward can rise while validation joint@H1 falls substantially.Source: §4.5.2 · p. 8, Figure 4 · p. 8

4.5.3 OOD Optimization beyond Scaffold Library Coverage

  • On OOD targets outside the restricted library’s property region, DeepSeek + MolHop harness achieves 43.4% joint@H1, substantially improving over scaffold retrieval alone.The reported result tests continued property optimization through molecular editing despite a lack of close scaffold retrieval candidates.Source: §4.5.3 · p. 8, Figure 5 · p. 9

5 Conclusion

MolHop frames cross-scaffold discovery as a search problem combining structural exploration with property optimization. Its conclusion emphasizes that structured chemical spaces and action interfaces support exploration beyond local molecular edits.

5 Conclusion

  • MolHop substantially improves frozen LLM agents’ cross-scaffold discovery ability, while agentic RL trains policies coordinating scaffold selection and molecular editing under multi-objective constraints.The conclusion presents structured environments as making exploration decisions explicit rather than relying only on local molecular transformations.Source: §5 · p. 9

AI use statement

  • Grok-4.6 generated SFT demonstration trajectories from 2,000 starting points, and the authors reviewed AI-assisted outputs and take responsibility for the final work.Cursor was used for coding assistance; Codex was used for language polishing and figure preparation.Source: AI use statement · p. 9

A.1 Hierarchical Scaffold Space

  • The scaffold space is built from commercial molecules and hierarchically organizes approximately 1.6 million valid Bemis–Murcko scaffolds from coarse ring structures to typed scaffolds.The hierarchy uses deterministic graph-abstraction rules; selected H7 scaffolds become selectable leaves, and cross-scaffold tasks compare the reference and final candidate’s H1 regions.Source: §A.1 · p. 13, Figure A1 · p. 14

A.2 Scaffold Navigation and Molecular Editing Actions

  • The harness provides structured scaffold navigation, molecular inspection, local editing, and search-control actions, with environment validation and feedback rather than free-form SMILES editing.Navigation can query, filter, and select scaffolds; editing covers fragments, atoms and bonds, and rings, while undo and submission manage episode states.Source: §A.2 · pp. 14–15

A.3 Trajectory Memory

  • Environment-maintained trajectory memory records scaffold visits, molecular revisions, property feedback, and search state to inform subsequent actions.The harness reconstructs memory after actual interaction events and summarizes older history under predefined rules to accommodate limited context windows.Source: §A.3 · pp. 15–16

B.1 Dataset Construction

The experiments use filtered ChEMBL drug molecules and Enamine commercial compounds, with fixed sampling procedures and exclusions between training and test starting molecules. Evaluation recalculates properties and synthetic-accessibility scores using a unified offline scoring function.

B.1 Dataset Construction

  • Drug references come from approved synthetic small-molecule ChEMBL records, while commercial compounds come from the August 2026 Enamine Screening Collection.ChEMBL records require max_phase=4 and exclude withdrawn entries; the study does not revalidate regional marketing status.Source: §B.1 · p. 17
  • Reference molecules are filtered by heavy-atom count, ring count, logP, and TPSA, and target properties and SA scores are recalculated with the project’s unified implementation.Test sets are scaffold-deduplicated and stratified by ring count, then sampled by MaxMin diversity using Morgan fingerprints; training starting molecules are excluded from test sets.Source: §B.1 · pp. 17–18

B.2 Evaluation Details

  • Joint success counts invalid, missing, or constraint-violating outputs as failures, using the total number of reference tasks as its denominator.The evaluation also reports scaffold Tanimoto and MCS ratio for structural change; decomposed metrics use distinct denominators for failure analysis.Source: §B.2 · p. 18

B.3 SFT and Agentic RL Training

MolHop models are trained through supervised fine-tuning followed by online GRPO reinforcement learning. The training data preserve varied search behaviors, while final evaluation distinguishes valid submission from joint success.

B.3 SFT and Agentic RL Training

  • SFT uses 2,000 Grok-4.6 demonstration trajectories, including 1,214 normal submissions and 786 trajectories reaching the 50-turn limit.Trajectories are split by starting point to prevent starting-point overlap between training and validation sets.Source: §B.3 · pp. 18–19
  • Reinforcement learning is initialized from the corresponding SFT checkpoint and uses online GRPO on separate RL starting points; multiple complete episodes are sampled per starting point, with within-group advantages calculated from terminal rewards.Source: §B.3 · p. 19
  • Training used 4 NVIDIA A800 80GB GPUs, with SFT and RL taking approximately 3 and 18 hours, respectively.Tables B2 and B3 provide the SFT and GRPO configurations.Source: §B.3 · p. 19, Table B2 · p. 19, Table B3 · p. 19
  • A valid terminal evaluation receives a continuous reward, while invalid or missing submissions receive -5.The supplied passages do not specify the continuous-reward formula.Source: §B.3 · pp. 19–20
  • Successful submission means completing the tool protocol and receiving a terminal evaluation, not necessarily satisfying joint@H1.Final joint@H1 requires structural transition, property matching, and SA improvement.Source: §B.3 · p. 20

C.1 Analysis of MolHop Harness on Frozen LLM Agents

The harness improves cross-scaffold joint success for frozen agents by improving both scaffold transition and property matching. The decomposition also shows how scaffold transition can exclude otherwise qualifying candidates.

C.1 Analysis of MolHop Harness on Frozen LLM Agents

  • After harness integration, both models produced valid outputs for the full test-set size on both the drug and commercial test sets.H1 transition and property matching also improved substantially.Source: §C.1 · p. 20, Table C1 · p. 20, Table C2 · p. 20
  • The three-condition intersection counts candidates meeting structural validity, property matching, and SA improvement, but does not require scaffold transition.Joint@H1 adds the scaffold transition requirement.Source: §C.1 · p. 20
  • On drug tasks, DeepSeek-Flash-V4.1’s H1 transition count rose from 87 to 192, its MD<1 count from 107 to 183, and its joint success count from 31 to 105.The H1 transition constraint explains the difference between candidates meeting the three other conditions and joint@H1.Source: Table C1 · p. 20, §C.1 · p. 20

C.2 Detailed Agentic RL Results

The detailed results separate property and validity successes from actual scaffold hopping. Under relaxed transition criteria, LLM baselines’ joint success rises, but their scaffold MCS does not substantially decrease.

C.2 Detailed Agentic RL Results

  • Some strong generation models meet validity, MD<1, and SA-improvement conditions but remain within the reference scaffold family, excluding them from joint@H1.Tables C3 and C4 count outputs satisfying validity, property matching, SA improvement, and H1 transition for the drug and commercial datasets.Source: §C.2 · p. 20, Table C3 · p. 21, Table C4 · p. 21
  • Figure C1 plots GRPO training reward and validation joint@H1 performance for MolHop-2B, MolHop-4B, and MolHop-9B.The supplied caption identifies the plotted measures but does not report their values.Source: Figure C1 · p. 21
  • LLM-based generation baselines show higher joint success under relaxed H1-to-H7 scaffold criteria, while scaffold MCS does not substantially decrease.MolHop and GenMol remain largely unchanged across the criteria; the baseline increase mainly reflects easier scaffold constraints, and most successes still correspond to molecular optimization.Source: §C.2 · p. 20, Figure C2 · p. 21

C.3 Scaffold Library Ablation Analysis

Removing the scaffold library shifts search toward editing the current structure, but the ablation struggles to coordinate scaffold transition, property matching, and synthetic accessibility improvement.

C.3 Scaffold Library Ablation Analysis

  • The ablation diagnosis examined terminal candidates for property matching, SA improvement, and scaffold transition.The evaluation covered 400 samples at step 150.Source: §C.3 · p. 22
  • Among 400 evaluation samples at training step 150, 96.1% met MD<1, but median SA score change was +0.393 and final joint success was 19.4%.Although 67.7% met validity, property-matching, and H1-transition conditions, they failed the strict SA-improvement requirement.Source: §C.3 · p. 22
  • Without the scaffold library, local editing struggles to reach distant structural regions and to ensure simultaneous SA improvement alongside property matching and scaffold transition.The model therefore cannot consistently coordinate all three joint-success objectives.Source: §C.3 · p. 22

C.4 Harness Tool Use and Tool-Call Efficiency

MolHop gives agents scaffold navigation, molecular editing, and search-control interfaces, supporting more efficient, state-dependent tool use than general LLM agents.

C.4 Harness Tool Use and Tool-Call Efficiency

  • MolHop agents allocate interaction budgets across navigation, scaffold selection, editing, and finishing, while general LLM agents tend to continue editing over longer trajectories.MolHop-2B more often retrieves and submits early; larger MolHop agents incorporate more edits to optimize properties.Source: §C.4 · p. 22
  • The learned policy uses the structured action space to reduce redundant exploration while maintaining structural-transition and property-optimization constraints.Source: §C.4 · p. 22
  • MolHop models use fewer tool calls to obtain joint@H1-successful candidates while maintaining high TESR, indicating more effective search decisions rather than more invalid calls.Table C5 compares tool-use efficiency.Source: Table C5 · p. 22, §C.4 · p. 22

D OOD Property Optimization beyond the Restricted Visible Scaffold Library

The OOD experiment restricts scaffold visibility to test whether editing can recover suitable candidates when retrieval alone falls short. In the examples, navigation, editing, and memory support varied trajectories, but close property matching alone does not guarantee joint success.

D OOD Property Optimization beyond the Restricted Visible Scaffold Library

  • The restricted visibility mask retains 287,888 H7 leaf nodes, or 17.3% of all leaves, with all five properties inside their library-wide interquartile ranges.Source: D OOD Property Optimization beyond the… · p. 23, Table D1 · p. 23
  • The 83 OOD targets have no close library neighbors; retrieval-only candidates have joint@H1 of 0 and mean match distance 2.376.Targets are selected from 400 ChEMBL-200 and Enamine-200 references whose five properties fall within the restricted property box.Source: D OOD Property Optimization beyond the… · p. 23
  • DeepSeek outputs move toward target regions outside the restricted library’s central property cluster, including both successful and unsuccessful candidates.The PCA projection indicates that editing-based optimization extends beyond the property range covered by bare scaffold retrieval.Source: D OOD Property Optimization beyond the… · p. 23
  • 43.4% (36/83) of OOD targets achieve joint@H1 with DeepSeek + harness, versus 0 with scaffold-library retrieval alone.The mean match distance decreases from 2.376 to 0.542, and 75/83 candidates satisfy md<1.Source: Table D2 · p. 23, D OOD Property Optimization beyond the… · p. 23

E Examples

  • The six ChEMBL trajectories show that memory recovery and scaffold exploration interact with editing, while property matching alone does not ensure joint success.One MolHop-4B trajectory undoes an edit that improves TPSA but worsens logP, then reloads a remembered candidate and succeeds. Another model comparison contrasts a successful lower-SA submission with a closer property match that fails; a separate MolHop-4B failure has all five property gaps at most 3.04% but SA rises from 2.427 to 3.289. Further baselines include a successful Grok-4.6 submission and DeepSeek budget exhaustion without submission.Source: E Examples · p. 23, Figure E1 · p. 24, Figure E2 · p. 24
Loading 2610.19920v1…