M3OS: A Monte Carlo Graph Search-Orchestrated Multi-Agent LLM System for Evidence-Traced Molecular Optimization

1 Introduction

• Prior approaches include Bayesian optimization, genetic algorithms, Monte Carlo tree search, reinforcement learning, generative molecular models, chemistry-focused LLMs, and agent-based optimization workflows.Related work also describes agents that use planning, memory, tools, and computational feedback.Source: §1 · p. 2
• Iterative optimization must connect each decision to previously evaluated structures, transformation histories, and task requirements, which conversational tracking can leave vulnerable to omitted evidence or candidate–result mix-ups.The motivating task also requires balancing often-competing objectives while refining molecular candidates.Source: §1 · p. 2
• M3OS uses a persistent molecular graph whose canonical-identity nodes retain provenance, evaluation evidence, and search statistics, while edges record parent–child transformations.Rediscovered molecules reuse existing nodes, and MCGS rewards and visit statistics guide parent shortlisting for LLM selection.Source: §1 · pp. 2–3
• Two specialized design agents share the selected parent and screening feedback but use distinct proposal strategies: tool-driven candidate generation and knowledge- and case-guided medicinal-chemistry editing.The MedChem Retrieval Agent supplies source-linked guidance, and a common Critic evaluates both branches against the task contract.Source: §1 · p. 3
• An execution harness separates free-text agent reasoning from persistent-record updates through task contracts, extraction of explicit outputs, molecular validation, and structured-evaluation requirements.It retains linked transformations, rationales, and evaluations for later decisions and inspection.Source: §1 · pp. 3–4
• Across three benchmarks, M3OS reports higher mean candidate-pool success in property optimization and higher final-solution success in activity-constrained lead optimization than Codex and Claude Code.The comparisons use the same LLM backbone and molecular-tool access.Source: §1 · p. 4

2.1 Overview of M3OS

• Each round expands a selected parent; Creative and Rational branches propose candidates, which undergo validation and Critic evaluation before structured records update the persistent graph.The graph retains candidate identity, parent transformations, tool results, and search statistics for later selection and expansion.Source: §2.1 · p. 4

2.2 Benchmark Datasets and Subset Construction

• The benchmarks cover eight single- or double-property tasks, five tasks involving three or four properties, and 93 activity-constrained lead-optimization tasks.Comparators include chemistry-specific language models, general-purpose language models, and coding agents.Source: §2.2 · p. 4, §2.3 · pp. 5–6

2.4 Metrics

• MolOpt strict success applies task-specific improvement thresholds, while MuMOInstruct reports joint success, similarity, and directional relative improvement; invalid MolOpt candidates count as failures.MolOpt rates are macro-averaged across eight tasks; MuMOInstruct metrics are averaged across records with valid candidates and then across five tasks.Source: MolOpt-120 · p. 6, MuMOInstruct-100 · pp. 6–7

2.5 Performance Comparison

• Across benchmarks, R3 mean strict success is 0.70 on MolOpt-120 (R1: 0.47), MuMOInstruct-100 mean joint success is 0.85, and SMDD-Bench-93 final success is 0.59.The SMDD-Bench-93 rate compares with 0.48 for Codex and 0.42 for Claude Code.Source: §2.5.1 · p. 7, Table 1 · p. 8, §2.5.2 · p. 8

2.6.1 Generator composition and reasoning effort.

• On MuMOInstruct-100, complete low-effort M3OS reaches third-round joint success of 0.91 versus 0.74 for Creative-only and 0.70 for Rational-only; high effort raises it to 0.94.The branches retain distinct improvement and similarity profiles, and high effort raises RI from 2.19 to 2.40.Source: §2.6.1 · pp. 8–9, Figure 2 · p. 10

2.6.2 Retrieved context.

• With Rational-only generation on the hERG/BBBP subset, both retrieval sources yield first-round loose/strict success of 0.86/0.60, versus 0.78/0.54 without case retrieval.Without medicinal-chemistry retrieval, the corresponding rates are 0.84/0.57.Source: Figure 2 · p. 10, §2.6.2 · p. 10

2.6.3 Search depth.

• On MuMOInstruct, joint success reaches 96.0% within five rounds and 97.5% within ten; RI rises from 1.28 at R1 to 2.63 at R10.The reported trajectories describe cumulative search-graph expansion.Source: Figure 2 · p. 10, §2.6.3 · p. 10

2.7 Case Study

• In the DHFR 16 case study, three rounds produce 19 candidates satisfying the full task contract, including coupled activity, similarity, ADMET, and hard-constraint requirements.Across the run, 766 proposals yielded 111 structures passing basic filters and 26 evaluated graph candidates; the reported edit series preserve the diaminoquinazoline anchor while varying distal-arm geometry and benzimidazole N-substitution.Source: §2.7 · p. 11

• M3OS coordinates agents around evaluated candidates, linking each proposal to a parent, shared task contract, and retained record for subsequent expansion.Source: §3 · p. 11
• Because the evaluation uses predicted properties, synthesis and assays are still needed to establish candidates’ experimental activity and ADMET profiles.Source: §3 · p. 11
• Figure 3 combines the three-round hDHFR search graph, edit trajectories, residue contacts, pocket views, and terminal-hydroxyl distances grouped by arm length and N-substitution.Source: Figure 3 · p. 12

4.1 Task representation and shared search state

• Tasks encode an initial molecule, objective, constraints, and optional protein context, while a persistent graph links canonical-SMILES candidates to evaluations, transformations, and search statistics.The graph retains evaluated intermediates, reuses rediscovered molecules, and gives both generator branches completed-round screening history while they maintain separate reasoning memories.Source: §4.1 · pp. 12–13

4.2 Graph-guided optimization loop

• MCGS ranks reachable nodes by accumulated rewards and visit counts, then propagates each expansion’s highest graph score along its selected path to guide later search.The LLM selects an existing shortlisted node; search stops at the expansion budget or task-specific criteria, and final candidates are chosen by graph score.Source: §4.2 · p. 13
• Parallel candidate-generation and medicinal-chemistry branches feed extracted, validated, deduplicated proposals to a shared Critic that evaluates them against the task contract.The Creative Molecule Explorer builds and screens a pool; the Rational Medicinal Designer proposes evidence-, knowledge-, and case-guided edits, with parent-aligned structural comparisons accompanying candidates.Source: §4.2 · p. 13

4.3 Controlled updates and evidence interfaces

• Structured extraction, molecular validation, role-specific tool allowlists, and local wrappers constrain graph updates, while procedural skills guide recurring design and evaluation decisions.The Auditor uses only explicitly stated molecules and evidence; callable capabilities combine external computational services with task-bound local functions.Source: §4.3 · p. 13, §A.1 · p. 14, §A.2 · pp. 14–15

Appendix B Medicinal-Chemistry Knowledge Base Construction and Retrieval

• The optimization-oriented collection distilled 1,221 documents and 45 functional-group and scaffold chapters into 86 core documents, each accompanied by a scope description for retrieval.The concept-oriented knowledge base links canonical concepts to source records and cited articles, with draft articles withheld from routine retrieval; approved articles and the optimization collection are queried through a dedicated agent.Source: Appendix B · pp. 17, 19, 20

Appendix C Construction of Molecular Evolutionary Graphs

• Property-specific patent graphs encode single-site improvements extracted with MMPA [43], and scaffold matching retrieves up to three connected components for each requested property.Graphs are made acyclic and transitively reduced; model-generated transformation descriptions and rationales are design hypotheses, not verified medicinal-chemistry claims, and edits are decided by subsequent tool-based evaluation.Source: Appendix C · pp. 20–21

Appendix D Optimization Report Generation and Validation

• On request, M3OS hands a versioned graph snapshot to an isolated reporting workflow that assembles candidate evidence, molecular assets, and an interactive search graph into self-contained HTML.Deterministic validation checks document completeness, external-resource absence, graph-component inclusion, node/detail coverage, and rendering of valid graph relationships before release.Source: §D.1 · p. 21, Figure D2 · p. 22, §D.2 · p. 22

D.4 Validation before delivery.

• Any failed check among complete non-truncated SMILES, candidate-card and audit coverage, synthesis-list consistency, or persistent review state prevents report delivery.Source: §D.4 · p. 24

E.1 Task Initialization

• Task initialization extracts the starting molecule, objectives, constraints, generation requirements, search budget, and optional protein context into a task brief and structured contract.Monotonic ADMET directions are frozen for relevant endpoints; contracts can encode thresholds, intervals, baseline-relative objectives, and hold tolerances when supplied.Source: §E.1 · p. 24

E.2 Session state, memory and structured records

• Shared session state retains the task contract, protein context, search graph, proposals, evaluations, and role-specific memories, while graph records link molecular evidence and transformations.Canonical SMILES identify molecule-associated records, and completed-round screening metrics, target gaps, and failure reasons are reused by both generator branches.Source: §E.2 · p. 24
• RDKit parsing and canonicalization validate SMILES; failed representations or evaluations are excluded from graph insertion, while evaluated candidates may be recorded despite failing task requirements.Generators may attempt up to three intent-preserving repairs; the Critic can only correct a copied structure against a valid input candidate.Source: §E.2 · p. 24

E.3.1 Overview.

• Each iteration selects an expansion base, generates and evaluates candidates, inserts valid results, and updates search statistics; MCGS ranks reachable nodes for LLM-assisted parent selection.The parent-based visit statistic aggregates direct-parent visits for multi-parent molecules, while the exploration term retains a node-specific denominator; selection uses task context and node evidence.Source: §E.3.1 · pp. 24–25, §E.3.2 · p. 25

E.3.3 Parallel molecular expansion

• Parallel expansion combines a tool-driven Creative Molecule Explorer candidate pool with targeted edits from a Rational Medicinal Designer using chemistry knowledge and retrieved cases.Both agents receive shared parent analysis and completed-round screening history, but retain separate role-specific memories.Source: §E.3.3 · p. 25

E.3.4 Candidate validation and Critic evaluation.

• RDKit parsing and canonicalization remove invalid representations and duplicate structures before evaluation, and valid candidates are compared with their parent using maximum-common-substructure alignment when available.The Critic evaluates the complete valid set using applicable task tools and returns candidate-specific rationales, property records, and heuristic scores; deterministic activity rank remains separate.Source: §E.3.4 · p. 26

E.3.5 Graph insertion and reward propagation.

• Valid molecules with structured evaluations enter the graph even when they fail task objectives, and the highest-scoring candidate's reward propagates along its selected root-to-candidate path.Unseen molecules create nodes, rediscovered molecules receive additional incoming transformation edges, and updated statistics inform subsequent upper-confidence-based selection.Source: §E.3.5 · p. 26

E.3.6 Termination and final selection.

• Search terminates when the expansion budget is reached or task-specific stopping criteria are satisfied, after which the final candidate set is selected by graph search scores.Source: §E.3.6 · p. 27

E.4 Benchmark instruction content

• The benchmarks provide a starting molecule and property directions or constraints, with task and candidate-output protocols varying across MolOpt-120, MuMOInstruct-100, and SMDD-Bench-93.MolOpt-120 has eight single- or double-property tasks, MuMOInstruct-100 has five three- or four-property combinations, and SMDD-Bench-93 adds protein targets and coupled lead-optimization requirements; M3OS uses a selected pool from its cumulative graph.Source: §E.4 · p. 27
• Within M3OS, generator inputs include the selected parent, shared analysis, and prior screening feedback, while Critic inputs include validated structures, parent comparisons, and tool evaluation records.Retrieval requests are focused knowledge questions, and Auditor inputs are finalized generator or Critic responses.Source: §E.4 · p. 27

E.5 Baseline prompting

• Baseline prompts convey the starting molecule and optimization objectives through interfaces adapted to each method's wording, conversation structure, and response conventions.Source: §E.5 · p. 27

Chemistry-specific language models.

• Chemistry-specific model prompts follow original-work conventions, with benchmark molecules and requested changes incorporated into model-specific formats.When objectives extend beyond a model's original task set, instructions are adapted within its prompt style.Source: Chemistry-specific language models · p. 27

General-purpose language models.

• Direct LLM baselines receive a shared task template requesting chemically valid, distinct candidates and structured outputs, without external molecular-evaluation tools.The template includes the initial SMILES, optimization goal, candidate count, and additional task requirements.Source: General-purpose language models · p. 27

General-purpose coding agents.

• Coding-agent baselines receive benchmark tasks and tool-use instructions for candidate generation, local filtering, batch predictions, and activity prioritization within tool-call budgets.They can use returned evidence to refine candidates and submit ranked SMILES with a designated best molecule.Source: General-purpose coding agents · p. 28

F.1 Benchmark subset construction

• For MolOpt-120, solubility, BBBP, and hERG evaluations use surrogate predictions because the original iDrug endpoint was unavailable.Starting molecules were sampled without replacement from the 500-molecule MolOpt-Instructions test pool and assigned to nonoverlapping task groups.Source: §F.1 · p. 28
• MuMOInstruct-100 sampling yielded 96 unique starting molecules; one task combination is in-domain and the remaining four are out-of-domain.Records were sampled without replacement within each task from the official test split.Source: §F.1 · p. 28

F.2.1 Search settings.

• At each expansion, the five highest-UCB eligible non-root nodes form the shortlist, distinct from the 20-candidate benchmark-evaluation pool.The search uses c = 0.01 and a Critic–similarity weight α = 0.8.Source: §F.2.1 · p. 28
• New nodes start with zero visits and accumulated reward equal to their initial graph score; the root instead starts with its initial Critic score.No additional backpropagated reward is assigned at initialization.Source: §F.2.1 · p. 28

F.2.2 Model and decoding configuration.

• The experiment logs record Kimi’s API identifier but not an immutable served-weight revision.Requests omitted temperature, nucleus-sampling probability, sampling seed, and completion-token cap; documented defaults were not explicitly enforced in the runs.Source: §F.2.2 · pp. 28–29
• Table F5 summarizes aggregate token consumption across the three benchmarks, with values reported in millions of tokens.Source: Table F5 · p. 29

F.2.3 Token accounting.

• Output counts include tokens used for internal thinking or reasoning as well as final visible responses.Source: §F.2.3 · p. 29
• Token totals characterize each complete harness rather than the underlying language model in isolation.Harnesses differ in interaction patterns, model turns, retries, context construction, and prompt caching.Source: §F.2.3 · p. 29
• Codex’s substantially larger SMDD-Bench-93 input count reflects longer agent trajectories and repeated attempts.Source: §F.2.3 · p. 29

F.3 Run configuration details

• For SMDD-Bench-93, each method is limited to eight Nesso and 15 ADMET-AI batch submissions per instance.Each submission counts as one call regardless of the number of molecules in the batch.Source: §F.3 · p. 29
• Retrieval ablations use Kimi-K3 Low, while extended-search runs retain all knowledge components.Source: §F.3 · p. 29