PaperlayerView exact revision on arXiv

Source-linked AI summary

GuacaMol: Benchmarking Models for De Novo Molecular Design

Nathan Brown, Marco Fiscato, Marwin H. S. Segler, Alain C. Vaucher

arXiv:1811.09621v2q-bio.QMcs.LGphysics.chem-phq-bio.BM

TL;DR

De novo molecular-design models lacked consistent validation and comparative evaluation against established algorithms. The paper introduces GuacaMol, a standardized benchmark framework, and reports that deep-learning models can reach the optimization performance of gold-standard discrete algorithms, while some tasks are too easy for meaningful discrimination.

  • Problem

    Neural molecular-design models showed promising results, but consistent validation and comparative studies against established de novo design models were lacking.

  • Method

    GuacaMol defines two evaluation dimensions—distribution learning and goal-directed generation—and implements quantitative standardized benchmarks for de novo design.

  • Results

    Deep-learning models can reach the optimization performance of gold-standard discrete algorithms, with both model types achieving similar scores.

  • Takeaways & Limitations

    GuacaMol provides a framework for quantitative benchmarking and comparison of de novo molecular-design models across distinct evaluation dimensions.

  • Takeaways & Limitations

    Some benchmark tasks can be too easily solved by most models, indicating a need for harder de novo molecular-design tasks.

Abstract

from arXiv · show

De novo design seeks to generate molecules with required property profiles by virtual design-make-test cycles. With the emergence of deep learning and neural generative models in many application areas, models for molecular design based on neural networks appeared recently and show promising results. However, the new models have not been profiled on consistent tasks, and comparative studies to well-established algorithms have only seldom been performed. To standardize the assessment of both classical and neural models for de novo molecular design, we propose an evaluation framework, GuacaMol, based on a suite of standardized benchmarks. The benchmark tasks encompass measuring the fidelity of the models to reproduce the property distribution of the training sets, the ability to generate novel molecules, the exploration and exploitation of chemical space, and a variety of single and multi-objective optimization tasks. The benchmarking open-source Python code, and a leaderboard can be found on https://benevolent.ai/guacamol

1 Introduction

De novo molecular design searches chemical space by generating relatively few molecules instead of enumerating large libraries. GuacaMol addresses inconsistent validation and limited comparisons by providing standardized benchmarks, implementations, and baseline results.

  • Motivation: Virtual screening can routinely process datasets of about 10^13 compounds, but drug-like chemical space may contain 10^24 to 10^60 structures.This leaves only a tiny fraction of chemical space covered by routine screening.
  • Motivation: De novo design explores chemical space through search or optimization while explicitly generating only a relatively small number of molecules.Focusing on relevant regions can in principle support queries over larger chemical spaces without full enumeration.
  • Research gap: Existing validation methods for neural generative models were inconsistent, and comparative studies against established de novo design models had not yet been performed.These gaps made model strengths, weaknesses, practical suitability, and opportunities for improvement difficult to assess.
  • Motivation: Standardized benchmarks can enable straightforward model comparison and provide information about model strengths and weaknesses.The authors connect this role to the progress associated with standardized benchmarks in computer vision and natural language processing.
  • Contribution: GuacaMol introduces a benchmarking framework, a suite of benchmarks, an assessable Python package, and baseline implementations and results.The framework is intended to standardize assessment of de novo molecular design models.

2 Models for De Novo Molecular Design

De novo design methods combine molecule generation, scoring, and optimization, with molecular representation shaping the explored chemical space. The paper describes graph- and sequence-based approaches, including neural models, and reports prospective evidence for SMILES-LSTM generation.

  • Design components: De novo design approaches require molecule generation, a scoring function, and optimization or search for better-scoring molecules.Molecules may be built atom-by-atom, from fragments, or in a binding-pocket context.
  • Representations: Molecular representation determines the potential range of chemical-space exploration and may use atoms and bonds, fragments, SMILES strings, or molecular graphs.General atom-by-atom or bond-by-bond representations can potentially explore all chemical space.
  • Design trade-offs: Unconstrained general representations can produce molecules that are difficult to synthesize or potentially unstable, while building-block approaches may drastically limit explorable space.This creates a central compromise between realistic molecules and broad molecular-space exploration.
  • Machine learning: Machine-learning approaches are proposed as an alternative to hand-specified rules because they can learn what reasonable molecules look like from data.The paper distinguishes property prediction from the less-attended inverse QSAR problem of generating structures from target properties.
  • Neural models: Neural molecular generators include VAEs, RNNs, GANs, AAEs, and graph convolutional networks using SMILES strings or graphs.SMILES-LSTM transfer learning pretrains on a general molecular corpus and can be fine-tuned on actives or coupled to an external scoring function.
  • Prospective evidence: SMILES-LSTM-generated compounds 1–3 were synthesized and showed micro- to nanomolar activity against RXR and PPAR receptors.This prospective study provided experimental validation of the transfer-learning approach.
  • Chemical-space exploration: A SMILES RNN trained on 0.1% of GDB13 covered a large portion of the complete GDB13 chemical space and generated molecules omitted from the original set.GDB13 is an enumerated set covering potentially stable molecules up to 13 heavy atoms.

3 Assessing De Novo Design Techniques

GuacaMol evaluates two independent uses of de novo models: learning a molecular distribution and generating molecules for predefined goals. Its benchmarks cover validity, uniqueness, novelty, distribution matching, similarity, rediscovery, isomers, and conflicting multi-target objectives.

  • Evaluation dimensions: The framework evaluates distribution learning and goal-directed generation as separate dimensions.Distribution learning asks models to generate novel molecules resembling a training set, whereas goal-directed generation seeks molecules satisfying a predefined goal.
  • Evaluation dimensions: The two categories are evaluated independently because there is no one-to-one correspondence between distribution-learning and goal-directed tasks.This preserves flexibility for models that support only one of the two uses.
  • Distribution learning: Distribution-learning benchmarks use a standardized ChEMBL subset to assess how well models reproduce a training-set molecular distribution.The suite contains five distribution-learning benchmarks.
  • Distribution learning: Validity penalizes invalid molecular structures, uniqueness penalizes repeated generations, and novelty penalizes molecules already present in the training set.These tests respectively address chemical validity, generation diversity, and training-set overfitting.
  • Distribution learning: FCD compares generated and reference molecular distributions using ChemNet hidden representations, with lower FCD indicating more similar distributions.The comparison uses means and covariances of penultimate-layer activations.
  • Distribution learning: KL-divergence benchmarks compare physicochemical-descriptor distributions between training and generated molecules, with smaller values indicating better capture of the training distribution.The benchmark also captures diversity to some extent.
  • Goal-directed generation: Goal-directed benchmarks combine molecular features such as structural properties, physicochemical properties, similarity, and substructure presence or absence into scoring functions.Models generate molecules with high scores and can iteratively improve their best candidates.
  • Goal-directed generation: Similarity, rediscovery, isomer, and median-molecule benchmarks test relatedness to targets, exact target recovery, molecular-formula conformity, and simultaneous similarity to conflicting targets.Median-molecule tasks also examine exploration of chemical space between target molecules.

4 Methods

The paper implements GuacaMol as an open-source, user-oriented Python framework and evaluates five distribution-learning and twenty goal-directed benchmarks. It provides a standardized comparison across neural, genetic-algorithm, MCTS, graph, SMILES, random, and dataset-picking baselines.

  • Framework: GuacaMol is available as an open-source framework with user-friendly interfaces for coupling molecular-generation models to its benchmarks.The implementation uses RDKit for chemoinformatics operations.
  • Benchmark suite: The benchmark suite contains five distribution-learning benchmarks and twenty goal-directed benchmarks.Specifications and implementation details are provided in separate appendices for the two categories.
  • Data: Several generation approaches require a training dataset, for which the paper uses ChEMBL 24.The dataset-generation procedure is described in Appendix 8.1.
  • Baselines: The paper evaluates baselines spanning deep-learning generative models, established genetic algorithms, and Monte Carlo Tree Search.The selection also covers models using SMILES strings and molecular graphs.
  • Baselines: Random sampling and best-of-dataset picking are included as dummy baselines that provide lower bounds for benchmark scores.Implementations of the assessed baseline models are made available on GitHub.

5 Results and Discussion

The benchmarks compare baseline generative models on distribution learning and goal-directed optimization, revealing trade-offs between fidelity, novelty, validity, optimization performance, and molecule quality.

  • Baselines: The random sampler provides a useful comparison baseline, while Best of Dataset represents selecting existing high-scoring molecules from the dataset.These baselines establish expected performance for distribution-learning and goal-directed tasks.
  • Distribution-learning benchmarks: The SMILES LSTM generates diverse, novel molecules that closely resemble ChEMBL, although it sometimes produces invalid molecules.Its distribution-learning performance is stronger than the random sampler's resemblance to the dataset, while Graph MCTS has higher validity, uniqueness, and novelty but poorer distribution matching.
  • Distribution-learning benchmarks: ORGAN performs poorly across distribution-learning tasks, with more than half of its generated molecules invalid and poor KL divergence and FCD scores.The authors suggest this pattern might indicate mode collapse, a phenomenon often observed when training GANs.
  • Distribution-learning benchmarks: VAE does not achieve the best score in any category but consistently produces relatively good scores across all distribution-learning tasks.The authors report that simpler VAE and LSTM models are more powerful than more complex models in these benchmarks.
  • Goal-directed benchmarks: Graph GA obtains the best results for most goal-directed benchmarks, while SMILES LSTM nearly matches it and outperforms it on three benchmarks.SMILES GA is better than Best of Dataset but lower than Graph GA on similarity tasks; Graph MCTS performs worse than the ChEMBL baseline.
  • Benchmark interpretation: Simple physicochemical-property benchmarks were not well suited to differentiating models when appropriate baselines were used.The authors also report that Graph MCTS performs poorly in both distribution-learning and goal-directed benchmarks, while population-based Graph GA performs extraordinarily well in optimization through fine-grained local edits.
  • Goal-directed benchmarks: 77% of molecules from Best of ChEMBL and the LSTM model pass the quality filters, compared with 40%, 36%, and 22% for Graph GA, SMILES GA, and Graph MCTS.The authors attribute the neural model's higher quality to transfer of learned information about molecular distributions into goal-directed optimization.
  • Goal-directed benchmarks: Neural models appear to partially resolve weaknesses of virtual screening and genetic algorithms by combining good optimization performance with molecule quality.The LSTM model performs slightly worse than Graph GA on goal-directed benchmarks but has considerably higher compound quality, comparable to the virtual-screening baseline.

6 Conclusions and Outlook

GuacaMol introduces standardized benchmarks for assessing de novo molecular design models across distribution learning, molecular optimization, and compound quality. Baseline evaluations show graph-based genetic algorithms lead optimization, while pretrained neural models can exceed them in compound quality; the authors also identify benchmark difficulty and objective quality as continuing challenges.

  • Conclusions: GuacaMol provides a standardized framework with independent evaluations of distribution learning and property-directed molecule generation.The framework encodes target properties as molecule scores and separates the two benchmark categories.
  • Conclusions: The benchmark suite contains 5 distribution-learning benchmarks, 20 optimization benchmarks, and a compound-quality metric.
  • Results: The graph-based genetic algorithm is the best baseline for optimization, while an SMILES-based recurrent neural network is second best with similar scores.The similar scores indicate that deep learning models can reach the optimization performance of established discrete algorithms.
  • Results: When pretrained on large datasets, the neural model outperforms the genetic algorithm in compound quality.
  • Limitations and outlook: Some benchmark tasks are too easily solved by most models, indicating a need for harder de novo design tasks.
  • Limitations and outlook: Future evaluation must address objective measurement of generated-molecule quality, time and sample-efficiency constraints, and movement from SMILES to graph representations.The authors note that SMILES have nevertheless been effective and that models should be assessed with well-understood strengths and limitations.

8.1 Dataset Generation

GuacaMol training data are derived from ChEMBL 24 and post-processed to create a dataset of synthesized, drug-relevant molecules while enabling similarity benchmarks against held-out marketed drugs.

  • The generative-model training datasets are derived from the ChEMBL 24 database, whose molecules have been synthesized and tested against biological targets.
  • Compared with ChEMBL, ZINC is biased toward smaller, more readily synthesizable molecules, while QM9 contains mostly compounds that have not been made.
  • ChEMBL is post-processed by neutralizing charges, removing long SMILES strings, restricting elements, and filtering molecules similar to a holdout drug set.The similarity filter uses an ECFP4 threshold of 0.323 against 10 marketed drugs.
  • The holdout set contains 10 marketed drugs, allowing similarity benchmarks for targets excluded from training.
  • The post-processed dataset is downloadable, and a version-controlled Docker container supports reproducible dataset creation.

8.2 Implementation details: Distribution-learning benchmarks

Distribution-learning benchmarks evaluate validity, uniqueness, novelty, and similarity between generated and reference molecular distributions using standardized samples and descriptor-based scores.

  • Validity is the ratio of valid molecules among 10’000 generated molecules, where validity requires successful parsing of the SMILES representation by RDKit.
  • Uniqueness is the fraction of different canonical SMILES strings among 10’000 valid generated molecules.
  • Novelty is computed from 10’000 different generated canonical SMILES strings as the ratio absent from the training dataset.
  • KL divergence compares distributions of nine physicochemical and fingerprint-based descriptors between sampled and reference sets.The nine descriptors include molecular properties, ring counts, and maximum nearest-neighbour similarities from ECFP4 fingerprints.

8.3 Implementation details: Goal-directed benchmarks

Goal-directed benchmarks combine property, similarity, SMARTS, and isomer scoring functions into aggregate scores designed to reward both high-scoring molecules and broader success among top candidates.

  • Modifier functions transform underlying property values into scores between 0 and 1, including Gaussian, one-sided Gaussian, and thresholded forms.
  • Goal-directed benchmarks aggregate multiple molecule-score contributions using arithmetic or geometric means, with geometric means requiring every contribution to be at least partly met.
  • Benchmark scores are weighted averages of molecule scores that emphasize top molecules while retaining performance across many satisfactory molecules.
  • Many benchmarks combine top-1, top-10, and top-100 scores from molecule scores sorted in decreasing order.
  • The suite includes rediscovery, similarity, isomer, median-molecule, and multi-property optimization tasks involving named drug targets and structural constraints.
  • Similarity, SMARTS, and isomer scoring functions respectively assess fingerprint similarity, pattern presence or absence, and agreement with target molecular formulas.

8.4 Baseline Models

GuacaMol compares random, dataset-scoring, genetic-algorithm, Monte Carlo tree search, recurrent-neural-network, and autoencoder baselines across distribution-learning and goal-directed settings.

  • The random sampler draws requested molecules directly from the dataset, providing a lower bound for goal-directed tasks and an upper bound for two distribution-learning tasks.
  • The dataset-scoring baseline returns the highest-scoring molecules in the dataset, serving as a goal-directed lower bound and supporting no distribution-learning benchmarks.
  • The MCTS baseline generates molecules from a CC starting structure through simulations and roll-outs, returning the best-scoring molecule found for goal-directed tasks.
  • The SMILES LSTM predicts the next character of partial SMILES strings and uses hill-climbing fine-tuning for goal-directed optimization.
  • VAE and AAE baselines encode molecules as latent vectors for distribution learning, while ORGAN combines generator and discriminator networks.

8.5 Results for trivial goal-directed benchmarks

The section shows that trivial goal-directed objectives are too easy to distinguish generative models or demonstrate advantages over virtual screening. Baseline models, including “Best of dataset,” achieve near-perfect or perfect scores on these benchmarks.

  • Selecting optimization objectives that are difficult enough to distinguish generative models from virtual screening is the main benchmarking challenge.Objectives already satisfied by training-set molecules cannot demonstrate whether generative models outperform virtual screening.
  • Table 4 lists seven easily-optimizable benchmarks, while Table 5 reports the corresponding baseline-model scores.
  • All models achieve near-perfect scores on the first three benchmarks targeting specified physicochemical-property values, including “Best of dataset”.
  • All models achieve perfect scores on the CNS MPO benchmark.CNS MPO is included among the trivial optimization objectives evaluated in this section.
  • Generative models achieve near-perfect scores on the C7H8N2O2 and Pioglitazone MPO benchmarks despite their more complex scoring functions.
  • Because ChEMBL already achieves excellent scores, these trivial benchmarks cannot demonstrate generative-model advantages or distinguish model performance.
Loading 1811.09621v2…