From Benchmark to Bench: Can Agents Survive Real-World Drug Discovery?

1 Introduction

• Lead optimization balances potency, safety, and developability through repeated design–make–test–analyze cycles, while agent evaluations often emphasize tool execution or generic benchmarks over real project constraints.Scoring services act as the oracles through which agent decisions are expressed, so their applicability bounds what the agent can achieve.Source: §1 · p. 1
• Across nine campaigns, objective attainment tracked model applicability, while the reported contribution identifies predictive-model reliability rather than agent coordination as the bound on attainment.Source: §1 · p. 2
• The work contributes an auditable architecture with executable, versioned objectives shared across LLM and REINVENT routes, and evaluates validity, attainment, workload, and limitations on nine industry campaigns.Source: §1 · p. 2

2 Related works

• Prior agent evaluations and time-resolved industrial benchmarks leave limited evidence on temporally held-out project chemistry, which MAGI exposes only after its pre-cutoff generation context.REINVENT 4 maximizes expert-defined scoring functions but does not author or revise them.Source: §2 · p. 2

3.1 System

• MAGI separates orchestration, generation, and evaluation, connecting generators and prediction services through a common REST contract and exposing 24 tools for campaign workflows.Source: §3.1 · p. 2

3.2 Objective and configuration space

• Versioned therapeutic target profiles specify scoring services, targets or acceptable intervals, desirability transforms, weights, and separate pass/fail guardrails for both LLM evaluation and REINVENT scoring.Endpoint predictions are transformed to [0, 1] desirabilities and combined using a weighted geometric mean.Source: §3.2 · pp. 2–3

4 Experimental setup

• Each campaign ran for five rounds, generating and evaluating molecules before carrying an agent-authored shortlist of approximately five compounds into the next round.The evaluation covered nine tasks: four GSK series, four Sanofi projects, and one AstraZeneca campaign; metrics included molecular quality, cofolding, and operational load.Source: §4 · p. 3

5.1 Molecular quality and chemical-space exploration

• REINVENT showed greater intra-round diversity in both organizations, while novelty and similarity patterns varied by dataset and objective.GSK proposals were more novel overall; within Sanofi, REINVENT departed furthest from shown chemistry, and Sanofi proposals more closely resembled held-out compounds than GSK proposals.Source: §5.1 · p. 3

5.2 Primary-endpoint and objective-attainment evaluation

• Three of four GSK series and the AstraZeneca campaign met their objectives, while only FXA and PPARδ cleared Sanofi thresholds.GSK-S1 missed its kinetic-solubility target outside the training range; the agent promoted thermodynamic solubility as a proxy, followed by improved proposals.Source: §5.2 · p. 3
• Shortfalls coincided with scorer applicability problems: GSK-S1 extrapolated beyond its training range, and the RENIN model had weak held-out correlation.The LLM reached objectives more often than REINVENT on Sanofi campaigns, reproducing historical programme quality without surpassing it.Source: §5.2 · pp. 3–4

5.3 Operational behaviour and medicinal chemistry evaluation

• REINVENT required four-to-nine times more operations than the LLM, and each campaign explored 100,000 more structures, increasing tool calls by four orders of magnitude.Observed operations exclude scorer and model calls inside REINVENT’s reinforcement-learning loop; MAGI also revised objectives, search parameters, and molecular strategy in response to campaign evidence.Source: §5.3 · pp. 4–5, Figure 2 · p. 5

6 Discussion

• LLM generation focused on local chemistry, while REINVENT broadened diversity and novelty; neither route consistently dominated model-space attainment.The discussion attributes attainment to scorer reliability, noting that generator breadth cannot compensate for biased, range-compressed, or weakly ranking models.Source: §6 · p. 5

7 Limitations

• Proposal endpoints are predictions rather than measurements, and each condition used one backend, one trajectory, and no static-objective control.Temporal splits restrict train/test overlap but cannot exclude pretraining exposure, and the design therefore does not isolate the agent’s contribution to attainment.Source: §7 · p. 5

8 Conclusion

• Across nine industrial campaigns, LLM generation provided local optimization with fewer operations, while REINVENT broadened exploration at higher operational cost.Neither route consistently won on objective attainment; campaigns succeeded when predictors were reliable and failed otherwise.Source: §8 · pp. 5–6
• The conclusion identifies scorer applicability and the quality of the agent’s SAR reasoning as two critical ceilings on real projects.The authors state that real project data exposed limits ranging from weak model performance on industrial data to autonomous adaptation through proxy endpoints.Source: §8 · p. 6

S1 Extended related work

• Early chemistry agents coupled language models with tools, wet-lab automation, or multi-agent hypothesis refinement.Examples include ChemCrow, Coscientist, and multi-agent architectures [6].Source: S1 Extended related work · p. 10
• Drug-discovery orchestrators have addressed pipeline assembly, tool selection, production deployment, governed hierarchies, memory, structure-based generation, modular execution, and affinity prediction.The passage cites systems spanning these approaches [7] [8] [9] [11] [12] [13] [14].Source: S1 Extended related work · p. 10
• Lead-optimization agents include explicit-reasoning, trajectory-aware, role-specialized, and full-process systems; MolClaw is named as the closest published stack to MAGI.The cited examples include [16] [17] [18] [19] and MolClaw [21].Source: S1 Extended related work · p. 10

S2.1 System

• MAGI uses a monitored gateway and MCP server to coordinate versioned TTPs, asynchronous optimization jobs, generators, data sources, and independently deployable scoring services.Shared storage moves structures, configurations, and results between services; the gateway records interactions and monitors job health.Source: S2.1 System · p. 10

S2.2 Services usage

• LLM and REINVENT proposals use the same TTP and molecular-evaluation services, although the REINVENT route also optimizes a job and selects from its generated pool.Campaign service choices vary: GSK and AstraZeneca use general ADMET without cofolding, while Sanofi uses project models and Boltz-2/PoseBusters.Source: S2.2 Services usage · p. 10, Figure S1 · p. 11

S2.3 TTP scoring semantics

• The TTP transforms endpoint predictions into desirabilities and combines them with a weighted geometric mean; full attainment separately requires every mandatory objective and fixed guardrail to pass.The geometric mean prevents strong performance on one endpoint from compensating for complete failure on another; missing values remain unevaluable.Source: S2.3 TTP scoring semantics · p. 12

S2.4 Scoring contract

• External scorers use an index-aligned REST contract that returns scores, metadata, and artifacts, while scoring outputs retain component values, warnings, failures, and TTP identity.Network or HTTP failures are logged and yield NaN by default; any native-only fallback is labelled.Source: S2.4 Scoring contract · p. 12

S2.5 System prompt

• The system prompt requires tool-grounded molecular claims and directs the agent to author TTPs, monitor campaigns, select candidates, and report terminal outcomes.The configured backend was Claude Sonnet 4.6, using provider-side sampling defaults without a fixed reasoning-model seed.Source: S2.5 System prompt · p. 13

S2.6 Agent adaptation, error recovery, and TTP provenance

• MAGI records messages, tool interactions, TTP versions, and scoring configurations, distinguishing evidence-linked scientific adaptations from execution repairs and defining observable errors and recoveries.A recovery counts only when a later agent action resolves an error and the intended operation succeeds; it is autonomous absent intervening human intervention or manual configuration.Source: S2.6 Agent adaptation, error recovery,… · p. 13

S2.7 Compute requirements

• Experiments used containerized services on AMD EPYC 9575F CPUs and AMD Instinct MI355X accelerators, with REINVENT and Boltz-2 on separate GPUs.Structure standardization, PoseBusters, and tabular prediction were CPU-parallelized.Source: S2.7 Compute requirements · p. 13

S3.1 Global ADMET data

• The global ADMET dataset contains 156,797 unique molecules and 186,120 sparse labels across seven endpoints, with source records harmonized before combination.Endpoints include logP, log D7.4, thermodynamic and kinetic solubility, Caco-2 permeability, and two efflux ratios.Source: S3.1 Global ADMET data · p. 13

S3.2 BET BD2 ADMET series

• Four published GSK BET BD2 series supply measured analogues, with endpoint definitions varying by series and one starter plus four analogues shown to the agent.The remaining published structures are withheld; repeated measurements use medians and missing endpoint values are excluded without imputation.Source: S3.2 BET BD2 ADMET series · p. 14

S3.3 AstraZeneca campaign

• The AstraZeneca campaign targets log D7.4, but its internal programme, target, therapeutic area, and compound structures are confidential.Only aggregate distributions and the campaign objective are reported; reference and held-out cohorts provide measured endpoint values without structures.Source: S3.3 AstraZeneca campaign · pp. 14–15

S3.4 RetroDMTA projects

• Four Sanofi projects cover FXa, MMP-8, PPARδ, and RENIN, with fixed temporal cutoffs defining pre-cutoff and future cohorts for each target.Primary endpoints are pKi for FXa, pIC50 for MMP-8 and RENIN, and pEC50 for PPARδ; FXa also includes solubility and log D7.4 objectives.Source: S3.4 RetroDMTA projects · p. 15

S3.5 Data processing and sample flow

• Structure processing standardizes parsed molecules, retains the largest organic fragment, applies chemical-content and size filters, and excludes unsupported or invalid structures.Duplicate global-ADMET records follow endpoint-specific agreement and retention rules, and missing endpoints are never imputed.Source: S3.5 Data processing and sample flow · pp. 15–16

S3.6 Information availability and leakage controls

• Temporal splits prevent explicit future endpoint values from being supplied to MAGI, but cannot rule out exposure through public literature or model pretraining.Interaction similarity measures recovery of a selected crystallographic binding pattern; MMP8 showed the highest similarity, followed by PPARδ and RENIN, with FXA lower.Source: S3.6 Information availability and… · p. 16

S4.1 Global ADMET models

• The global ADMET ensemble combines descriptor-and-fingerprint XGBoost models with two multitask neural model families, and reports pooled out-of-fold performance across five shared Butina folds.Operational applicability requires both Avalon nearest-neighbour Tanimoto ≥0.4358 and Chemprop latent-space cosine ≥0.7334; these thresholds indicate training-space similarity, not calibrated prediction correctness.Source: S4.1 Global ADMET models · pp. 16, 18

S4.2 Local project-specific models

• Sanofi project-specific ligand models use temporally restricted training data and provide campaign-specific predictions plus a baseline orthogonal to Boltz-2.Rank correlations exceed 0.4 for FXA, MMP8, and PPARδ, while RENIN has ρ ≈0.31; no error-calibrated applicability threshold was established.Source: S4.2 Local project-specific models · p. 18

S4.3 Boltz-2

• Boltz-2 and ligand-based predictions agree only for MMP8; they disagree on FXA extremes and do not agree for PPARδ or RENIN.The authors describe ligand- and structure-based predictions as complementary rather than redundant.Source: S4.3 Boltz-2 · p. 18

S4.4 PoseBusters

• PoseBusters evaluates ligand geometry and chemistry from the Boltz complex without receptor-dependent docking validation, and reports the fraction of 12 checks passed.Pass fractions range from 0.833 to 1.0; a fraction below 1 is not a complete pass.Source: S4.4 PoseBusters · p. 18

S4.5 Interaction Profiler

• The interaction profiler returns typed protein–ligand contacts and a human-readable interaction fingerprint, or a medicinal-chemistry fragment taxonomy in SMILES-only mode.In structure-plus-SMILES mode it verifies ligand identity and records fragment, interaction geometry, and protein-residue details.Source: S4.5 Interaction Profiler · p. 20

S5.1 Attainment scope and limitations

• Complete TTP attainment requires passing every required objective and guardrail, so primary-endpoint attainment alone does not establish multiparameter success.The analyses report primary, secondary, fixed-guardrail, and complete-TTP attainment separately.Source: S5.1 Attainment scope and limitations · p. 20

S5.2 Molecular quality

• Molecular quality includes novelty, diversity, synthesizability, scaffold provenance, and calibration; a cofolding ablation compares rankings with and without the Boltz-2 term.Novelty uses maximum ECFP4 Tanimoto similarity to the shown set; diversity is one minus mean pairwise Tanimoto, and the ablation uses Spearman correlation and top-K overlap.Source: S5.2 Molecular quality · p. 20

S5.3 GSK and AstraZeneca primary-endpoint outcomes

• Three of four GSK projects produced at least one proposal meeting the model-space primary threshold; neither route met GSK-S1’s kinetic-solubility objective.LLM also remained below GSK-S2’s permeability criterion; weak, range-compressed held-out agreement made GSK-S1 predictions near target extrapolative, so the agent promoted thermodynamic solubility as a proxy.Source: S5.3 GSK and AstraZeneca… · p. 20

S5.4 Statistical analysis

• Route means are compared with paired Wilcoxon tests and Holm adjustment; molecule-level Mann–Whitney tests are exploratory because analogues remain chemically dependent.Chemist-evaluation proportions use Wilson 95% intervals.Source: S5.4 Statistical analysis · pp. 20–21

S6.1 Project objectives

• Each task supplied measured ADMET for a starter and four reference analogues, an endpoint objective, and guardrails; project-specific targets included solubility, permeability, log D7.4, and rotatable bonds.The agent received the starter and analogue measurements but never saw held-out compounds; Table S6 covers objectives, guardrails, scoring directions, target bounds, and required evaluations.Source: S6.1 Project objectives · p. 21, Table S6 · p. 22

S6.2 Agent experiment

• The LLM route requested approximately five final structures per round, whereas REINVENT selected five; route figures and statistics used common complete rounds 1–5, and Sanofi designated project-model prediction and Boltz-2 affinity co-primary.Source: S6.2 Agent experiment · p. 23

S6.3 Medicinal chemist evaluation

• In the blinded evaluation, discrimination did not separate from chance for any of four chemists or pooled analysis, and no source was preferentially selected in prioritization.The instrument included source-discrimination, candidate-prioritization, and SAR-reasoning questions.Source: S6.3 Medicinal chemist evaluation · pp. 24–25

S7.1 Human-participant ethics

• The blinded evaluation involved four participants, and no personally identifying participant information is released.Source: S7.1 Human-participant ethics · p. 25

S7.2 Responsible release

• MAGI outputs are predictions rather than experimentally validated compounds and require expert review; molecular-design systems may also generate unsafe or inappropriate structures.Source: S7.2 Responsible release · p. 25

S8 Supplementary figures

• Extended figures support the analyses presented in the main text.Source: S8 Supplementary figures · p. 25
• Main-text cross-references identify the relevant supplementary figure panels.Source: S8 Supplementary figures · p. 25
• Underlying numerical values and tests are provided in the accompanying data package.Source: S8 Supplementary figures · p. 25

• The checklist asks whether the abstract and introduction accurately reflect the paper’s contributions and scope.Source: NeurIPS Paper Checklist · p. 26
• The abstract’s claim that predictor calibration, rather than generation route, governs attainment is supported by Section 5.2.Source: NeurIPS Paper Checklist · p. 26
• The three stated contributions map to the shared versioned objective, nine industry-derived tasks, and evidence-driven adaptation results.Source: NeurIPS Paper Checklist · p. 26
• Checklist guidance says claims should match experimental results and reflect assumptions, limitations, and expected generalization.Source: NeurIPS Paper Checklist · p. 26

Limitations

• MAGI outputs are predictions, while historical measurements represent a different evidence level; each condition used one reasoning backend and trajectory, and temporal splits cannot rule out pretraining exposure.These limitations qualify how the campaign results should be interpreted.Source: §2 · p. 26
• AstraZeneca campaign results cannot be reproduced from released artifacts alone because those data are not releasable.The MAGI agent, MCP server, and global ADMET ensemble are slated for open-source release; RetroDMTA data and GSK series have cited sources.Source: §2 · p. 28
• Molecular-design systems may generate unsafe, reactive, or otherwise inappropriate structures, and MAGI outputs require expert review because they are predictions, not experimentally validated compounds.The paper frames the system as an orchestration layer over existing generators and predictors, adding no generative capability beyond REINVENT 4 and the configured reasoning backend.Source: §2 · pp. 30–31

Property-distribution evaluation

• In the GSK and Sanofi property plots, historical and held-out values are measurements, while proposal values may be model predictions; TPSA and molecular weight are RDKit descriptors.The AstraZeneca log D7.4 plot is an exception: reference and held-out measurements are available only for log D7.4.Source: Figure S12 · p. 42, Figure S13 · p. 43, Figure S14 · p. 44
• The Sanofi five-compound Reference samples are visualization context, not the parents used in the campaigns.The plots also show the complete pre-cutoff project-model training set and future held-out compounds.Source: Figure S15 · p. 45, Figure S16 · p. 46

Operational accounting

• Input-token and observable-operation comparisons exclude internal REINVENT scorer calls.The plotted counts cover model-context tokens and agent-visible operations during rounds 1–5.Source: Figure S17 · p. 47

Blinded chemist evaluation

• The blinded chemist evaluation involved four chemists using a 28-question instrument, including 12 source-discrimination pairs, 11 prioritization cases, and five SAR rationales.The figure describes source-discrimination accuracy, candidate-source selections, and rationale verdicts, but the supplied caption gives no outcome values.Source: Figure S18 · p. 47