PaperlayerView exact revision on arXiv

Source-linked AI summary

Sample Efficiency Matters: A Benchmark for Practical Molecular Optimization

Wenhao Gao, Tianfan Fu, Jimeng Sun, Connor W. Coley

arXiv:2206.12411v2cs.CEq-bio.BM

TL;DR

Molecular optimization lacks standardized comparisons that account for costly oracle evaluations and sample efficiency. PMO benchmarks 25 methods on 23 objectives under controlled oracle budgets, finding that older methods often remain strongest and that performance varies with oracle landscapes and model design.

  • Problem

    Existing studies often use trivial or self-designed tasks and rarely report oracle-query budgets, limiting direct comparison of molecular optimization methods.

  • Method

    PMO is a reproducible benchmark evaluating 25 molecular optimization methods across 23 oracle functions with controlled query budgets and independent trials.

  • Results

    Older methods such as REINVENT and Graph GA outperform more recent methods in the benchmark, while no implemented method efficiently solves some objectives within hundreds of oracle calls.

  • Takeaways & Limitations

    Sample efficiency, standardized reporting, and careful algorithm and oracle-landscape analysis are central for assessing practical molecular optimization.

  • Takeaways & Limitations

    The study does not exhaustively evaluate or tune all methods, may be biased toward similarity-based oracles, and does not thoroughly investigate synthesizability or diversity.

Abstract

from arXiv · show

Molecular optimization is a fundamental goal in the chemical sciences and is of central interest to drug and material design. In recent years, significant progress has been made in solving challenging problems across various aspects of computational molecular optimizations, emphasizing high validity, diversity, and, most recently, synthesizability. Despite this progress, many papers report results on trivial or self-designed tasks, bringing additional challenges to directly assessing the performance of new methods. Moreover, the sample efficiency of the optimization--the number of molecules evaluated by the oracle--is rarely discussed, despite being an essential consideration for realistic discovery applications. To fill this gap, we have created an open-source benchmark for practical molecular optimization, PMO, to facilitate the transparent and reproducible evaluation of algorithmic advances in molecular optimization. This paper thoroughly investigates the performance of 25 molecular design algorithms on 23 tasks with a particular focus on sample efficiency. Our results show that most "state-of-the-art" methods fail to outperform their predecessors under a limited oracle budget allowing 10K queries and that no existing algorithm can efficiently solve certain molecular optimization problems in this setting. We analyze the influence of the optimization algorithm choices, molecular assembly strategies, and oracle landscapes on the optimization performance to inform future algorithm development and benchmarking. PMO provides a standardized experimental setup to comprehensively evaluate and compare new molecule optimization methods with existing ones. All code can be found at https://github.com/wenhao-gao/mol_opt.

1 Introduction

Molecular design seeks functional molecules under property, stability, and synthesizability constraints, but comparing optimization methods remains difficult. PMO addresses this gap by benchmarking methods under controlled oracle budgets and reproducible protocols.

  • Molecular design is a constrained multi-objective search for molecules with desired properties, stability, and synthesizability.
  • Many existing studies and benchmarks make algorithm comparison difficult by omitting oracle-query budgets.Oracle calls can span orders of magnitude, although experiments and high-accuracy simulations are costly.
  • PMO benchmarks 25 methods across 23 oracle functions using independent trials and a maximum budget of 10,000 oracle queries.Performance is measured by the area under the curve of top-10 average performance versus oracle calls.
  • Under realistic budgets, existing algorithms often fail to outperform predecessors, and none efficiently solves some de novo optimization problems.The study also analyzes algorithmic contributions and oracle-landscape effects on optimization performance.

2 Algorithms

The benchmarked methods combine molecular assembly strategies with optimization algorithms to search chemical space. The study spans string-, graph-, and synthesis-based representations alongside screening, evolutionary, tree-search, Bayesian, generative, and reinforcement-learning approaches.

  • Preliminaries: A molecular optimization method combines an assembly strategy that defines chemical space with an algorithm that navigates it.
  • Preliminaries: The study focuses on general-purpose single-objective optimization of small organic molecules with scalar properties relevant to therapeutic design.
  • Molecular assembly strategies: String-based strategies modify SMILES or SELFIES representations directly, while graph-based strategies alter atoms, bonds, or molecular fragments.SELFIES enforces chemical validity through its representation.
  • Molecular assembly strategies: Synthesis-based strategies assemble pathways from commercially available starting materials and reliable transformations to constrain searches to synthesizable candidates.
  • Optimization algorithms: Screening samples pre-enumerated libraries, whereas genetic algorithms explore through mutation and crossover of molecular representations.The benchmark includes random screening, model-based screening, SMILES GA, STONED, Graph GA, SynNet, and GA+D.
  • Optimization algorithms: MCTS searches promising branches, Bayesian optimization combines surrogate predictions with uncertainty, and VAEs optimize molecules through latent representations.
  • Optimization algorithms: Score-based models sample target-property distributions, hill climbing retrains generative models with high-scoring molecules, and reinforcement learning maximizes cumulative reward.

3 Experiments

PMO evaluates molecular optimization under a controlled oracle budget, combining optimization performance with sample efficiency across diverse methods, objectives, and chemical representations. Results show strong differences by algorithm age, representation, model-based design, and oracle landscape.

  • Benchmark setup: PMO pursues optimization ability, sample efficiency, generalizability, and robustness across molecular design tasks.The benchmark evaluates methods across multiple objectives, independent trials, and controlled oracle usage.
  • Benchmark setup: 25 methods are benchmarked on 23 oracle functions using AUC Top-10 over at most 10,000 oracle calls.AUC Top-10 rewards methods that reach high top-10 property values with fewer oracle evaluations.
  • Results & Analysis: AUC Top-10 reveals sample-efficiency differences that top-10 performance alone can obscure, including between SMILES LSTM HC, Graph GA, and REINVENT.SMILES LSTM HC requires more oracle queries than Graph GA for comparable performance, while REINVENT requires fewer.
  • Results & Analysis: REINVENT and Graph GA are the best-performing compared algorithms, despite being released several years earlier.The benchmark notes that newer methods generally do not improve performance under its metrics, although some target validity or synthesizability instead.
  • Results & Analysis: Most SELFIES variants do not outperform corresponding SMILES variants in optimization ability and sample efficiency, although SELFIES-based GA has an observed advantage over SMILES-based GA.The GA comparison is not fully head-to-head because mutation and crossover rules strongly affect performance.
  • Results & Analysis: Model-based methods can improve sample efficiency, but predictive-model quality and inner- and outer-loop design determine whether they help.MolPAL and screening benefit from predictive models over random sampling, whereas GP BO and GFlowNet-AL do not consistently outperform their counterparts.
  • Results & Analysis: String-based genetic algorithms perform relatively well on isomer-type oracles, illustrating that method suitability varies with oracle landscape.Figure 3 clusters oracles by relative AUC Top-10 and distinguishes similarity-based, isomer-based, and other landscapes.
  • Results & Analysis: Hyperparameter reoptimization and multiple independent runs are required because default settings can be suboptimal and performance depends on random seeds.REINVENT’s best-performing σ was much larger than values suggested in its original paper.

4 Conclusions

PMO standardizes molecular optimization evaluation around sample efficiency, comparing 25 methods across 23 objectives. The study finds that algorithm choice, model design, oracle landscape, reproducibility practices, and task-specific evaluation all materially shape conclusions.

  • PMO evaluates 25 methods across 23 objectives with sample efficiency as a central benchmarking focus.The benchmark is intended to support standardized comparisons under shared evaluation conditions.
  • Strong baseline methods such as LSTM HC may use data inefficiently, while older methods including REINVENT and Graph GA can outperform newer methods.
  • Model-based methods may improve sample efficiency but require careful design of the inner loop, outer loop, and predictive model.
  • Different optimization algorithms may excel on different tasks because oracle landscapes affect relative performance.
  • The study cannot exhaustively evaluate every method or hyperparameter, may be biased toward similarity-based oracles, and omits thorough analysis of synthesizability and diversity.These scope limitations constrain how broadly the benchmark conclusions should be generalized.
  • Subsequent studies should use equal oracle budgets, multiple oracle types, independent runs with outcome distributions, and task-specific hyperparameter tuning.

Checklist

The checklist records that the study reports its contributions and limitations, uses publicly available assets, and provides reproducibility information. It also documents ethics and participant-related items as inapplicable or absent because the work does not involve human subjects.

  • The paper reports that its contributions and scope are accurately described and that its limitations are discussed in Section 4.
  • The experiments provide code, parameters, releasable data, training details, and hardware information through the repository and appendices.
  • Existing assets are cited and licensed, while new assets are released through the supplemental material or a URL.
  • The data and code are publicly available, and the paper states that it contains no personally identifiable information or offensive content.
  • Human-subjects, crowdsourcing, participant-risk, compensation, and theoretical-proof checklist items are marked inapplicable where relevant.

A.1 AUC Top-10 Table

The appendix reports AUC Top-10 results and optimization curves across methods, assembly strategies, and oracle families. Table 4 ranks methods by summed mean AUC Top-10 across tasks, while Figures 4–8 visualize optimization curves for distinct oracle categories.

  • Table 4 reports mean and standard deviation of AUC Top-10 from 5 independent runs and ranks methods by summed mean AUC Top-10.
  • The listed methods span fragment, SELFIES, synthesis, atom, and graph-based molecular assembly strategies.
  • The table includes task-level AUC Top-10 results for similarity, MPO, isomer, QED, SMARTS-based, and machine-learning oracle tasks.
  • Figures 4–8 show top-10-average optimization curves for similarity-based, similarity-based MPO, isomer-based, QED, SMARTS-based, and machine-learning oracles.

A.3 Synthesizability

The benchmark evaluates synthesizability using SA_Score on the Top-100 molecules from each method. Synthesis-based methods consistently produce lower SA_Score values across tasks.

  • SA_Score is calculated on the Top-100 molecules from each method and averaged across runs.
  • Synthesis-based methods consistently achieve lower SA_Score across all tasks.The figure treats lower SA_Score as better.
  • Figure 9 visualizes the averaged SA_Score values as a heat map.

A.4 Diversity

The benchmark measures diversity among each method’s Top-100 molecules using averaged internal distance based on Tanimoto similarity. Stronger optimization is generally associated with less diverse results, while diversity can help under noisy oracles.

  • Diversity is computed as the averaged internal distance within each Top-100 molecular batch, measured by Tanimoto similarity.
  • Stronger optimization generally produces less diverse molecular results.
  • Higher-diversity methods have an advantage when oracle noise is non-ignorable.
  • Figure 10 shows diversity values as a heat map averaged across runs.Higher diversity is treated as better in the figure.

B Implementation Details

The implementation-details section documents shared method properties and provides default and tunable hyperparameter configurations for reproducible experiments.

  • Table 7 summarizes shared properties across all benchmarked methods.
  • Each method includes a hparams_default.yaml file specifying its default setup.
  • Each method includes a hparams_tune.yaml file listing possible hyperparameter choices for tuning.

B.2 Shared Setup: dataset

The benchmark standardizes datasets, optimization implementations, training practices, and method descriptions to support controlled comparisons across diverse molecular optimization approaches.

  • Shared setup: dataset: ZINC is used consistently for model training or pretraining, initial molecule sets, and vocabulary extraction.This avoids bias from using different datasets across methods.
  • Shared setup: training: Early stopping terminates selected iterative methods after five iterations without performance improvement.The strategy is used to reduce computational cost.
  • Shared setup: optimization: Bayesian optimization is unified across VAE methods using BoTorch with an exact Gaussian-process model.Sparse Gaussian processes could improve scalability, but the evaluated methods generally reached several thousand oracle calls.
  • Shared setup: training: Pretraining uses ZINC and does not require oracle calls.
  • Method taxonomy: The benchmark includes auto-regressive, one-hot, and cross state-action spaces, associated respectively with RL, VAE or gradient methods, and genetic algorithms.
  • Method implementations: Screening randomly samples ZINC, whereas MolPal trains and updates a property predictor to prioritize molecules with higher predicted scores.
  • Method implementations: MolDQN treats molecular optimization as a Markov Decision Process and selects graph edits using estimated Q-values.Each episode allows up to 40 steps, with one oracle call per step.

B.33 Methods Not Included

The paper excludes several representative molecular optimization methods because they are not comparable under the benchmark setting or require separate environments. It documents a unified setup, public implementations, and reproducible run commands.

  • Methods excluded: BOSS and ChemBO were excluded because their Bayesian optimization procedures scale poorly and generated fewer than 200 molecules before stopping.The paper attributes this partly to the poor scaling of string subsequence kernels.
  • Methods excluded: ORGAN, MolGAN, and rationaleRL are listed among representative methods not included in the benchmark.
  • Experimental setup: The benchmark provides a unified conda environment for most methods, while ChemBO, DoG-AE, and DoG-Gen require individual environments.The individual environments follow modified or original installation instructions.
  • Reproducibility: The experiments use ZINC, publicly release implementations for all 25 methods, and provide pretrained models or required configuration files.The package uses the MIT license, and methods can be run after environment setup.
  • Reproducibility: Example commands support production runs, hyper-parameter tuning, offline logging, and a maximum of 10,000 oracle calls.

D Additional Results

Additional results examine representation effects, hyper-parameter sensitivity, property distributions, and aggregate benchmark metrics. They show that valid SELFIES strings can still collapse exploration and that tuning substantially affects performance.

  • SELFIES strings collapse: Valid SELFIES strings do not necessarily improve molecular optimization because different strings can decode to the same truncated string.Figures 11–13 illustrate this collapse using strings built from the ZINC 250k vocabulary.
  • Hyper-parameter tuning: Most algorithms are sensitive to hyper-parameter choices, which are visualized using summed AUC Top-10 scores for zaleplon_mpo and perindopril_mpo.The endpoint averages results from three runs for each task.
  • Hyper-parameter tuning: REINVENT’s sigma has a large impact on optimization performance, with the optimal value much larger than the original paper’s default.
  • Hyper-parameter tuning: Figures 15–20 report hyper-parameter tuning results for REINVENT, Graph GA, GP BO, DoG-AE, DoG-Gen, GFlowNet, GFlowNet-AL, Graph MCTS, SMILES LSTM HC, MARS, MIMOSA, Pasithea, GA+D, SMILES GA, and STONED.
  • Property distributions: The Zaleplon MPO distribution in ZINC 250k is highly concentrated among the top 1,000 molecules, while all remaining molecules have values below 0.02.
  • Aggregate results: Tables 8, 11, 14, 17, and 20 summarize mean and standard deviation across five independent runs for aggregate AUC Top-1, AUC Top-100, Top-100, Top-10, and Top-1 metrics.
Loading 2206.12411v2…