Sample Efficiency Matters: A Benchmark for Practical Molecular Optimization

Motivation

• Molecular design seeks functional molecules with desired properties while satisfying constraints such as stability and synthesizability, using combinatorial optimization [1, 2], predictive models [3], and generative models.Recent progress has improved validity [8] [9], diversity, and synthesizability [11] [12].Source: §1 · p. 1
• Existing comparisons are limited by unreported oracle budgets, trivial or self-designed tasks, and substantial run-to-run randomness, with prior benchmarks covering no more than five methods.Because valuable experimental or high-accuracy simulation oracles are costly, identifying compounds with few calls is essential.Source: §1 · p. 2

Contribution

• PMO benchmarks 25 methods across 23 widely used oracle functions, using multiple independent trials, a 10,000-query limit, and AUC of top-10 performance versus oracle calls.The study combines optimization ability with sample efficiency under a standardized experimental protocol.Source: §1 · p. 2
• None of the existing algorithms efficiently solves de novo molecular optimization within hundreds of experiments, while state-of-the-art methods often fail to outperform predecessors.The findings motivate further analysis of algorithmic contributions and oracle landscapes.Source: §1 · p. 2

2 Algorithms

• Molecular optimization combines a molecular assembly strategy defining chemical space with an optimization algorithm navigating it, and Table 1 categorizes representative methods by both components.The benchmark is not intended as an exhaustive catalog, but as a representative basis for meaningful conclusions.Source: §2 · p. 2, Table 1 · p. 3

2.1 Preliminaries

• The benchmark targets general-purpose, single-objective optimization of small organic molecules with scalar properties evaluated by an inaccessible black-box oracle over an impractically large chemical space.Practical optimization is motivated by the substantial cost of experiments or high-accuracy simulations.Source: §2.1 · pp. 2–3
• QED and LogP are excluded because QED can be maximized by random sampling near 0.948, while LogP is unbounded and easily increased by adding carbons.LogP performance therefore depends strongly on chemical-space definition and allowed steps rather than optimization ability, and maximizing it is not meaningful for drug design.Source: §2.1 · p. 2

2.2 Molecular assembly strategies

• Molecule assembly strategies comprise string-based SMILES or SELFIES, atom- or fragment-based graphs, and synthesis-based construction from commercial starting materials and reliable transformations.Synthesis-based strategies constrain search toward synthesizable candidates and may be template-free [11] or template-based [12].Source: §2.2 · p. 3

2.3 Optimization algorithms

• The benchmarked optimization algorithms span screening, genetic algorithms, MCTS, Bayesian optimization, latent-variable VAEs, score-based models, hill climbing, reinforcement learning, and gradient ascent.These methods differ in how they navigate molecular strings, graphs, synthesis pathways, or latent spaces; BOSS [30] and ChemBO [19] produced no meaningful results, potentially because of poor kernel scaling.Source: §2.3 · p. 4

3.1 Benchmark setup

• PMO contains 23 pharmaceutically relevant oracle functions, including QED, target-bioactivity models, and Guacamol multi-property objectives, with scores normalized from 0 to 1.Docking oracles are excluded because they are more costly and remain coarse estimates of bioactivity.Source: §3.1 · p. 5
• AUC Top-10 is the primary metric because it rewards reaching high top-10 average property values with fewer oracle calls, using K = 10 and a 10,000-call limit.AUC values are min-max scaled to [0, 1].Source: §3.1 · p. 5

3.2 Results & Analysis

• Among all compared methods, REINVENT and Graph GA are the best-performing algorithms, despite being released years ago, while newer models do not seem to improve performance by the reported metrics.Table 2 reports mean AUC Top-10 and standard deviation from 5 independent runs, while Table 3 ranks methods using different metrics.Source: Table 2 · p. 6, Table 3 · p. 7, §3.2 · p. 8
• Most methods are not efficient enough to optimize even simple toy objectives within hundreds of oracle calls, while token- or atom-by-atom assembly from one start point is especially data-inefficient.These strategies can waste oracle queries on undesired, unstable, or unsynthesizable candidates.Source: §3.2 · p. 7
• Most SELFIES variants do not outperform corresponding SMILES methods in optimization ability or sample efficiency, although SELFIES-based genetic algorithms show an advantage when sequence-manipulation rules are manually designed.The SELFIES advantage for genetic algorithms is not a head-to-head representation comparison because performance also depends heavily on mutation and crossover rules.Source: §3.2 · p. 8, Figure 2 · p. 8
• Model-based optimization can improve sample efficiency over random sampling, but predictive-model quality and inner- and outer-loop design determine whether it helps.GP BO outperformed Graph GA on 12 of 23 tasks, yet Graph GA won in summation, while GFlowNet outperformed GFlowNet-AL on almost every task.Source: §3.2 · pp. 8–9

• PMO evaluates 25 methods across 23 objectives and finds that older methods can outperform newer ones, while algorithm choice remains task- and oracle-landscape-dependent.LSTM HC may use data inefficiently; REINVENT and Graph GA can outperform recent methods, and model-based approaches require careful inner-loop, outer-loop, and predictive-model design.Source: §4 · p. 10
• The study is limited by incomplete method and hyperparameter coverage, possible similarity-oracle bias, and limited investigation of synthesizability and diversity.Oracle calls are counted from scratch, including data used to train surrogate models, although previously collected datasets may justify pretrained surrogates.Source: §4 · p. 10
• Subsequent studies should use equal oracle budgets, multiple oracle types, independent runs with outcome distributions, and hyperparameter tuning after changing tasks or testing environments.These practices are recommended to improve fair comparison and reproducibility for non-deterministic molecular optimization methods.Source: §4 · p. 10

• The authors state that the abstract and introduction accurately reflect the paper’s contributions and scope, and that limitations are described in Section 4.Source: Checklist · p. 15
• The paper provides code, parameters, releasable data, reproduction instructions, experimental details, error bars, and compute-resource information.The repository is available at https://github.com/wenhao-gao/mol_opt; Appendix B covers setup and implementation, Table 2 reports error bars, and Sections B and C.2 provide supporting details.Source: Checklist · p. 15
• The authors cite and license existing assets and release new assets, with the repository and appendices documenting datasets, implementation, experimental setup, and hardware.Source: Checklist · p. 15
• All data and code are publicly available, and the paper involves no human subjects, personally identifiable information, or offensive content.Crowdsourcing, participant-risk, consent, and compensation requirements are marked not applicable where relevant.Source: Checklist · pp. 15–16

A.1 AUC Top-10 Table

• The benchmark ranks methods by summed mean AUC Top-10 across all tasks, with results reported as mean ± standard deviation from five independent runs.The table includes methods using fragment, SELFIES, atom, and synthesis assembly strategies.Source: Table 4 · p. 17, §A.1 · pp. 18–19
• MIMOSA achieves 0.787±0.003 on osimertinib_mpo, exceeding MolPal at 0.775±0.004 and MARS at 0.749±0.007.These are AUC Top-10 values for the osimertinib_mpo task.Source: §A.1 · p. 19
• MolPal records the highest listed AUC Top-10 on albuterol_similarity at 0.580±0.023, above MARS at 0.447±0.007 and MIMOSA at 0.390±0.008.The comparison is task-specific and uses the values reported in the table.Source: §A.1 · p. 19
• MolPal reaches 0.934±0.002 on qed, narrowly above MARS at 0.931±0.002 and MIMOSA at 0.902±0.005.These values correspond to the qed task’s AUC Top-10 results.Source: §A.1 · p. 19
• The table reports task-level AUC Top-10 results for 23 optimization tasks, including similarity, MPO, rediscovery, isomer, SMARTS-based, and QED objectives.Reported tasks include albuterol_similarity, osimertinib_mpo, qed, valsartan_smarts, and zaleplon_mpo.Source: §A.1 · p. 19

A.2 Optimization Curves

• Optimization curves visualize top-10 average performance while optimizing similarity-based oracles.This comparison is presented in Figure 4.Source: Figure 4 · p. 20
• Additional optimization curves cover similarity-based MPO, isomer-based, QED, SMARTS-based, and machine learning oracles.These oracle categories are shown in Figures 5–8.Source: Figure 5 · p. 21, Figure 6 · p. 21, Figure 7 · p. 22

A.3 Synthesizability

• Synthesis-based methods consistently achieved lower SA_Score values across all tasks, with lower scores indicating better synthesizability.The comparison uses SA_Score values for the Top-100 molecules from each method, averaged across all runs.Source: Figure 9 · p. 23, §A.3 · p. 23

A.4 Diversity

• Stronger optimization performance generally corresponded to less diverse Top-100 molecular results.Diversity was computed for the Top-100 molecules from each run and averaged across runs.Source: §A.4 · p. 24
• Higher-diversity methods may have an advantage when oracle noise is non-ignorable.Source: §A.4 · p. 24
• Figure 10 compares methods by Top-100 diversity, with higher values indicating greater diversity.The heat map reports diversity averaged from all runs.Source: Figure 10 · p. 24, §A.4 · p. 24

B Implementation Details

• Shared implementation properties for all methods are summarized in Table 7.Source: B Implementation Details · p. 25

B.1 Hyperparameter setup

• Each method provides separate files for its default setup and several hyperparameter choices available for tuning.“hparams_default.yaml” specifies the default setup, while “hparams_tune.yaml” specifies tunable alternatives.Source: §B.1 · p. 25

B.2 Shared Setup: dataset; B.3 Shared Setup: Early Stop; B.4 Shared Setup: Bayesian optimization for all VAEs; B.5 Shared Setup: Pretraining; B.6 Shared Setup: PyTorch based

• ZINC supplies training or pretraining data, initial molecules, and vocabularies; pretraining requires no oracle calls, while iterative methods stop after five unimproved iterations.All methods use PyTorch and Adam; VAE-based Bayesian optimization uses BoTorch exact Gaussian processes with shared pretrained models.Source: §B.2 · p. 25, §B.5 · p. 25, §B.3 · p. 25

B.7 Action space manipulation; B.8 Screening; B.9 MolPAL

• Methods use autoregressive, one-hot, or cross action spaces, corresponding respectively to sequential construction, whole-molecule modification, or population-based genetic search.Screening randomly evaluates ZINC molecules without learning, whereas MolPAL predicts properties online to prioritize higher-scoring molecules.Source: §B.7 · p. 25, Table 7 · p. 26, §B.8 · p. 27

B.10 SMILES-VAE; B.11 SELFIES-VAE; B.12 DST; B.13 Pasithea

• VAE and gradient-based methods optimize learned molecular representations: SMILES-VAE and SELFIES-VAE use BO, while DST and Pasithea update molecules through differentiable predictors.DST uses graph convolutions, population offspring, DPP diversity, and ZINC substructures; Pasithea uses a multilayer perceptron over SELFIES multi-hot vectors with online updates.Source: §B.10 · p. 27, §B.11 · p. 27, §B.12 · p. 27

B.14 MolDQN; B.15 MIMOSA; B.16 MARS; B.17 GFlowNet; B.18 GFlowNet-AL

• Sequential and sampling methods include MolDQN’s graph-editing MDP, MIMOSA and MARS’s MCMC procedures, and GFlowNet variants using graph neural networks and building-block vocabularies.MolDQN permits 40 steps per episode with one oracle call per step; MIMOSA masks substructures and online-updates on top-scoring molecules, while MARS uses adaptive proposals and simulated annealing.Source: §B.14 · p. 28, §B.15 · p. 28, §B.16 · p. 28

B.19 JTVAE; B.20 GP BO; B.21 DoG-AE; B.22 DoG-Gen

• JTVAE, GP BO, DoG-AE, and DoG-Gen optimize or generate molecular graphs through latent-space Bayesian optimization, fingerprint-guided graph search, or synthesizable reaction-based construction.GP BO uses a 2-radius 2048-bit fingerprint and Graph-GA internally; DoG-Gen samples 3,000 molecules and retains 1,000 by fitness using a USPTO-pretrained reaction oracle.Source: §B.19 · p. 29, §B.20 · p. 29, §B.21 · p. 29

B.23 SynNet

• SynNet constructs synthetic trees with four MLP modules over molecular fingerprints, selecting actions, reactants, and reaction templates through masked classification and nearest-neighbor retrieval.Its genetic process uses populations of 128, offspring sets of 512, and mutation probability 0.5.Source: §B.23 · pp. 29–30

B.24 REINVENT; B.25 SELFIES-REINVENT; B.26 SMILES-LSTM Hill Climbing (SMILES-LSTM HC); B.27 SELFIES-LSTM Hill Climbing (SELFIES-LSTM HC)

• REINVENT and SELFIES-REINVENT use recurrent models combining prior likelihood with reward, while SMILES- and SELFIES-LSTM hill climbing pretrain on ZINC and retain high-scoring populations.REINVENT’s tuned reward weight σ is 500 rather than the original 60; its SELFIES variant changes only the vocabulary, as does SELFIES-LSTM HC relative to SMILES-LSTM HC.Source: §B.24 · p. 30, §B.25 · p. 30, §B.26 · p. 30

B.28 GA+D (SELFIES-GA); B.29 STONED; B.30 SMILES-GA; B.31 Graph-GA; B.32 Graph-MCTS

• GA+D, STONED, SMILES-GA, Graph-GA, and Graph-MCTS cover genetic and parameter-free graph searches using SELFIES, SMILES, or molecular graphs with specified populations, mutations, or tree exploration.GA+D sets discriminator weight β to 0; STONED performs mutation-only SELFIES search, SMILES-GA omits crossover for validity, Graph-GA combines crossover and mutation, and Graph-MCTS starts from ethane with exploration coefficient 4.3.Source: §B.28 · p. 30, §B.29 · p. 30, §B.30 · p. 31

B.33 Methods Not Included

• BOSS and ChemBO were excluded because they generated fewer than 200 molecules before stopping and scaled poorly with data dimension and dataset size.Their computational cost arises from intensive approximate-function and kernel evaluations, with poor scaling attributed partly to string subsequence kernels [70].Source: §B.33 · p. 31
• ORGAN, MolGAN, and rationaleRL were excluded because ORGAN had less than 1% validity, MolGAN lacked comparable optimization performance, and rationaleRL required more than 10K oracle calls.RationaleRL also depends on property-aware rationale extraction and Monte Carlo Tree Search.Source: §B.33 · p. 31

C.1 Software

• The software setup unifies most methods in a conda environment using Python 3.7, PyTorch, PyTDC, RDKit, wandb, and YaML, with optional SELFIES and BoTorch support.ChemBO, DoG-AE, and DoG-Gen instead require individual environments or additional packages following their original instructions.Source: §C.1 · p. 32

C.2 Hardware

• The experiments use an Intel Xeon E5-2690 machine with 256G RAM and eight NVIDIA Pascal Titan X GPUs, with GPUs accelerating most neural-network methods.Source: §C.2 · p. 32

C.3 License

• The package uses the MIT license, relies on the freely usable ZINC database, and makes implementations for all 25 methods publicly available on GitHub.Source: §C.3 · p. 32

C.4 Run with one-line code

• After conda setup, all methods can be run with one line of code using provided pretrained models, data, and configuration files.Example commands support graph-GA runs, production tasks with five runs and QED/JNK3/DRD2 oracles, tuning, offline wandb logging, and a 10,000-call oracle limit.Source: §C.4 · p. 32

D.1 SELFIES strings collapse

• Valid SELFIES strings can collapse to truncated strings, so replacing SMILES does not necessarily improve molecular optimization or expand chemical-space exploration.The examples use tokens from ZINC 250k’s vocabulary and the official SELFIES decoder; Figures 11–13 illustrate the collapse.Source: §D.1 · p. 33, Figure 11 · p. 33, Figure 12 · p. 33

D.2 Hyper-parameter Tuning

• Hyper-parameter choices substantially affect algorithm performance, motivating broad tuning and visualization across representative isomer-based and similarity-based oracles.The endpoint sums AUC Top-10 for zaleplon_mpo and perindopril_mpo, averaged over three runs per task.Source: §D.2 · p. 34
• REINVENT’s sigma has a large performance impact, with the optimum much larger than the original paper’s default setting [5].Figures 14 and 15 report the two-part REINVENT tuning results.Source: Figure 14 · p. 34, Figure 15 · p. 34

D.3 Additional Tables

• Additional hyper-parameter tuning results are provided for Graph GA, GP BO, DoG-AE, DoG-Gen, and GFlowNet.The corresponding results appear in Figures 16–20.Source: Figure 16 · p. 34, Figure 17 · p. 35, Figure 18 · p. 35

D.4 Distribution of ZINC 250k’s properties

• Further tuning results cover GFlowNet-AL, Graph MCTS, SMILES LSTM HC, MARS, MIMOSA, Pasithea, GA+D, SMILES GA, and STONED.These results are shown in Figures 21–30.Source: Figure 21 · p. 37, Figure 22 · p. 37, Figure 23 · p. 37
• The ZINC 250k property analysis includes the distribution of Zaleplon MPO values, showing the top 1000 molecules while all remaining molecules are below 0.02.This distribution is presented in Figure 31.Source: Figure 31 · p. 40

D.3 Additional Tables

• Supplementary tables report mean and standard deviation for AUC Top-1, AUC Top-100, Top-100, Top-10, and Top-1 across five independent runs.Methods are ranked by summed mean performance across tasks for each metric.Source: Table 8 · p. 41, Table 11 · p. 44, Table 14 · p. 47