GuacaMol: Benchmarking Models for De Novo Molecular Design

1 Introduction

• Current computational constraints allow routine screening of about 10^13 compounds, only a tiny fraction of drug-like chemical space, while complex multi-objective queries are difficult without substantial enumeration.The estimated space is 10^24 to 10^60 possible structures, potentially larger with modalities such as PROTACs.Source: §1 · p. 2
• De novo design generates a relatively small number of molecules and explores chemical space through search or optimization, potentially focusing on areas relevant to a multi-objective query.The authors argue this can in principle explore the full chemical space given any ranking function.Source: §1 · p. 2
• Existing generative-model validation was inconsistent, and comparisons with established design models had not yet been performed, making strengths and weaknesses difficult to assess.Recent optimization studies often focused on easily optimized properties such as drug-likeness or partition coefficients.Source: §1 · p. 2
• GuacaMol introduces a benchmark suite and Python package for assessing de novo molecular design models, with baseline implementations and results.The package is designed to make model assessment easy for researchers.Source: §1 · p. 3

2 Models for De Novo Molecular Design

• Design approaches combine molecule generation, a real-valued molecular scoring function, and optimization or search for molecules with better scores.Scoring functions may combine multiple objectives.Source: §2 · p. 3
• General atom-by-atom construction can explore broad chemical space but yield unstable or hard-to-synthesize molecules, whereas building-block approaches constrain that space.The section presents this realism-versus-exploration tension as a central dilemma of de novo design.Source: §2 · pp. 3–4
• Machine learning offers a way to learn reasonable molecular patterns from data, while predicting structures from target properties—the inverse QSAR problem—has received less attention.QSAR/QSPR traditionally predicts properties from molecular structures.Source: §2 · p. 4
• VAEs encode molecular SMILES into a continuous latent representation in real space and back for optimization; neural networks pretrained on a general corpus can use external scoring to drive generation toward desired properties.Source: §2 · pp. 4–5
• Compounds 1–3 generated with a SMILES-LSTM transfer-learning method showed micro- to nanomolar activity against RXR and PPAR receptors [14, 66, 67].The compounds were synthesized and tested.Source: §2 · p. 5, Figure 1 · p. 5
• SMILES RNNs trained on 0.1% of GDB13 covered a large portion of its chemical space after sufficient sampling and generated molecules omitted from the original set [68].GDB13 enumerates potentially stable molecules with up to 13 heavy atoms.Source: §2 · p. 5

3.1 Distribution-learning benchmarks

• Distribution-learning benchmarks assess how well models reproduce a standardized ChEMBL subset, using five tasks to evaluate generated molecules.The paper notes that learning molecular syntax can accelerate goal-directed tasks for some architectures.Source: §3.1 · p. 6

3.1.1 Validity; 3.1.2 Uniqueness

• Validity checks for realistic molecular structures, while uniqueness penalizes repeated generations.Invalid SMILES syntax and invalid valence are examples of validity failures.Source: §3.1.1 · p. 7, §3.1.2 · p. 7

3.1.3 Novelty

• Novelty penalizes molecules found in the training set, so models that overfit receive low scores.The benchmark reflects the expectation that a model with good chemical-space coverage will rarely reproduce training-set molecules.Source: §3.1.3 · p. 7

3.1.4 Fr´echet ChemNet Distance

• Low Fréchet ChemNet Distance (FCD) indicates similar generated and training-set distributions, measured using ChemNet activation statistics.FCD compares means and covariances of penultimate-layer activations for the reference and generated molecules.Source: §3.1.4 · pp. 7–8

3.1.5 KL divergence

• Small KL divergence scores indicate physicochemical descriptor distributions like the training set, and the benchmark captures diversity to some extent.It compares descriptor distributions for training and generated molecules and rewards matching the training set's property-distribution diversity.Source: §3.1.5 · p. 8

3.2 Goal-directed benchmarks

• Goal-directed benchmarks ask models to iteratively generate high-scoring molecules, separating optimization performance from the choice of scoring functions.Objectives can combine structural, physicochemical, similarity, and substructure features; the scoring functions are not the article's focus.Source: §3.2 · pp. 8–9

3.2.1 Similarity; 3.2.2 Rediscovery

• Similarity tasks generate many molecules close to a held-out target, whereas rediscovery tasks seek the target molecule itself.Similarity can also be used to exclude molecules that are too similar to others.Source: §3.2.1 · pp. 9–10, §3.2.2 · p. 10

3.2.3 Isomers; 3.2.4 Median molecules

• Isomer benchmarks generate molecules matching a target formula, while median-molecule tasks maximize similarity to several molecules simultaneously and probe exploration between them.Isomer benchmarks are fully determined but have no direct drug-discovery application; median-molecule benchmarks are intentionally conflicting.Source: §3.2.3 · p. 10, §3.2.4 · p. 10

3.3 Measuring Compound Quality

• Unrestricted de novo design can produce compounds that are unstable, reactive, difficult to synthesize, or unattractive to medicinal chemists, making compound-quality assessment necessary.The benchmark includes this assessment because unsuitable outputs can erode medicinal chemists’ trust.Source: §3.3 · p. 11
• Exhaustively encoding undesirable substructures is challenging because medicinal-chemistry judgments are subjective and context-dependent, while QSAR models capture this knowledge only incompletely.For example, toxicity risk may be assessed differently in oncology and diabetology.Source: §3.3 · p. 11
• Previously compiled HTS-collection rules are controversial and serve as high-precision, low-recall surrogates: filtering suggests a molecule is probably bad, but passing does not prove it is reasonable.The filter list is incomplete.Source: §3.3 · p. 11
• Using the filters implementation [76], the benchmark applies SureChembl, Glaxo, PAINS (retrieved from ChEMBL [69]), and in-house rules; the collection is in Supporting Information [77].Source: §3.3 · p. 11

4 Methods

• GuacaMol offers user-friendly interfaces for connecting researchers’ models to the benchmarking framework with minimal effort.The framework is available as an open-source Python package.Source: §4 · pp. 11–12
• The benchmark suite comprises five distribution-learning and twenty goal-directed tasks, specified in Appendices 8.2 and 8.3, respectively.Source: §4 · p. 12
• GuacaMol uses RDKit for SMILES handling and canonicalization [78], logP and TPSA calculations [79], and similarity calculations [80].Source: §4 · p. 12
• ChEMBL 24 [69] supplied the training data required by several molecule-generation approaches.Dataset generation details are given in Appendix 8.1.Source: §4 · p. 12
• The evaluation compares recent deep-learning generators, genetic algorithms, and Monte Carlo Tree Search, representing both SMILES- and graph-based methods.Two dummy baselines—random sampling and best-of-dataset picking—provide lower bounds on benchmark scores.Source: §4 · p. 12

5.1 Distribution-learning benchmarks

• SMILES LSTM sometimes generates invalid molecules but is diverse and closely resembles ChEMBL; Graph MCTS generates many valid, unique, novel molecules but reproduces training-set property distributions less precisely.Source: §5.1 · p. 13
• AAE has excellent uniqueness and novelty but generates molecules less close to the dataset than SMILES LSTM, while ORGAN scores poorly, with more than half its molecules invalid and outputs unlike the training set.The authors suggest ORGAN’s results may indicate mode collapse, often observed when training GANs.Source: §5.1 · p. 13
• VAE scores are not best in any distribution-learning category but are consistently relatively good, supporting the authors’ finding that simpler VAE and LSTM models outperform more complex models.The random sampler provides a useful baseline for expected KL-divergence and Fréchet-benchmark scores, while its novelty is zero because it only generates dataset molecules.Source: §5.1 · pp. 12–13

5.2 Goal-directed benchmarks

• Graph GA achieves the best results on most goal-directed benchmarks, while SMILES LSTM nearly matches it and outperforms it on three benchmarks.SMILES GA beats the Best of Dataset baseline but scores lower than Graph GA, especially on similarity tasks.Source: Table 2 · p. 14, §5.2 · p. 14
• Simpler benchmarks centered on physicochemical properties such as logP and QED do not differentiate models well when appropriate baselines are used.Source: §5.2 · p. 15
• In the goal-directed benchmarks, 77% of molecules from Best of ChEMBL and the LSTM passed the quality filters, compared with 40% for Graph GA, 36% for SMILES GA, and 22% for Graph MCTS.Source: Figure 3 · p. 15, §5.2 · p. 15
• Graph MCTS performs poorly across distribution-learning and goal-directed benchmarks, suggesting that refining one root molecule in a search tree may suit de novo design less well than populations of good compounds.Source: §5.2 · p. 16
• Graph GA’s population-based, fine-grained local edits support strong optimization, but GAs produce inconsistent-quality molecules because they lack pre-encoded molecular-distribution knowledge.The authors conclude neural models partially address the weaknesses of virtual screening and GAs by combining good optimization performance with molecule quality.Source: §5.2 · p. 16

6 Conclusions and Outlook

• GuacaMol quantitatively benchmarks models for de novo design, providing standardized assessment of new models to improve comparability and evaluating their ability to learn from molecule datasets and generate novel molecules.Source: §6 · pp. 16–17
• The graph-based genetic algorithm ranked best and the SMILES recurrent neural network second with similar optimization scores; pretraining the neural model on large datasets improved compound quality beyond the genetic algorithm.The authors interpret the similar optimization performance as evidence that deep-learning models can reach the performance of gold-standard discrete algorithms for de novo design.Source: §6 · p. 17
• Baseline evaluation found some tasks too easy, motivating harder benchmarks; measuring generated-molecule quality objectively, and accounting for time constraints and sample efficiency, also need further attention.The authors note that these considerations may matter depending on requirements.Source: §6 · p. 17
• Further work should advance neural models from SMILES to graph representations, despite SMILES' effectiveness [17]; the open-source framework aims to enable challenging benchmarks and rigorous assessment of model strengths and limitations.The authors also envision computational chemists confidently harnessing models with well-understood strengths and limitations.Source: §6 · pp. 17–18

8.1 Dataset Generation

• The training datasets used for generative models are derived from the ChEMBL 24 database, making it the data source for those models.Source: §8.1 · p. 18
• ChEMBL post-processing neutralized charges, removed SMILES longer than 100 characters, excluded molecules containing elements other than those listed (H, B, C, N, O, F, Si, P, S, Cl), and removed molecules with ECFP4 similarity above 0.323 to a holdout of 10 marketed drugs.Source: §8.1 · pp. 18–19
• The 10 marketed drugs form a holdout set that enables similarity benchmarks for targets not represented in training.Source: §8.1 · p. 19
• The post-processed ChEMBL dataset is downloadable, and a Docker container with version-controlled dependencies supports reproducible dataset creation [85, 77].Source: §8.1 · p. 19

• Validity is the ratio of molecules among 10’000 generated molecules whose SMILES representations can be successfully parsed by RDKit.Source: Validity · p. 19

• The model is sampled until 10,000 valid molecules have been obtained.Source: Uniqueness · p. 19
• Molecules are treated as duplicates when their canonical SMILES strings are identical.Source: Uniqueness · p. 19
• The uniqueness score is the number of different canonical SMILES strings divided by 10,000.Source: Uniqueness · p. 19

Novelty

• The novelty score collects 10,000 distinct canonical SMILES, then calculates the ratio of molecules not present in an unspecified reference.The supplied passage ends before identifying the reference set.Source: Novelty · p. 19

KL divergence

• KL evaluation compares sampled and reference descriptor distributions, computes per-descriptor divergences, and aggregates them across k = 9 descriptors.Continuous descriptors use kernel density estimation and discrete descriptors use histograms; the comparison also includes maximum nearest-neighbor similarity distributions from ECFP4 fingerprints.Source: KL divergence · p. 20

8.3.1 Formalism

• For many benchmarks, molecule scores average multiple contributions via arithmetic or geometric mean; with a geometric mean, all individual scoring-function requirements must be met, and the final benchmark score is a weighted average of molecule scores.Source: §8.3.1 · pp. 20–21

8.3.2 List of benchmarks

• The benchmark suite spans rediscovery and similarity, isomer, median-molecule, drug-property optimization, and scaffold-hop objectives.The isomer benchmarks include C11H24, while scaffold-hop tasks aim to change a compound's scaffold while retaining specified substituents.Source: §8.3.2 · p. 23

8.3.3 Scoring functions

• Scoring functions use Tanimoto fingerprint similarity, binary SMARTS-pattern matching, and a geometric-mean isomer score based on target element counts and total atom count.Source: §8.3.3 · p. 24

8.3.4 Score modifiers

• Four described modifiers shape property scores: Gaussian targets a value, MinGaussian and MaxGaussian favor one side of a target, and Thresholded applies a cutoff.Source: §8.3.4 · p. 25

8.4 Baseline Models

• The paper describes baseline models and notes that their implementations are available on GitHub.Source: §8.4 · p. 25

8.4.1 Random sampler

• The random-sampler baseline draws the requested number of molecules uniformly from the dataset.Source: §8.4.1 · p. 25
• Random sampling sets a lower bound for goal-directed benchmarks and an upper bound for two distribution-learning benchmarks because it performs no optimization and returns molecules from the original distribution.Source: §8.4.1 · p. 25

8.4.2 Best in dataset

• The “Best in dataset” baseline scores all dataset molecules and returns the highest-scoring ones, providing a lower bound for goal-directed benchmarks; it does not apply to distribution learning.Source: §8.4.2 · p. 26

8.4.5 Graph MCTS

• For goal-directed benchmarks, this baseline uses no initial population; each molecule starts from a CC base and is generated with 40 simulations, considering 25 children at each step.For distribution learning, generation also starts from a CC base; without a scoring function, the first molecule to reach a terminal state is returned.Source: §8.4.5 · pp. 27–28

8.4.6 SMILES LSTM

• The SMILES LSTM predicts the next character and uses hill-climbing fine-tuning for goal-directed tasks, while distribution-learning runs generate molecules without fine-tuning.Each of 20 hill-climbing iterations generates 8192 molecules and fine-tunes on the top-scoring 1024.Source: §8.4.6 · p. 28

8.4.7 Variational Autoencoder; 8.4.8 Adversarial Autoencoder; 8.4.9 Objective-reinforced generative adversarial network

• The VAE encodes and decodes molecules through a continuous latent space, the AAE regularizes that space to a prior, and ORGAN combines generator and discriminator networks; these implementations support distribution-learning benchmarks.Source: §8.4.7 · pp. 28–29, §8.4.8 · p. 29, §8.4.9 · p. 29

8.5 Results for trivial goal-directed benchmarks

• Generative models achieve near-perfect scores on trivial goal-directed benchmarks, but ChEMBL alone scores excellently, so these tasks cannot show an advantage over virtual screening or distinguish model performance.The tasks include physicochemical-property targets, CNS MPO, QED, C7H8N2O2, and Pioglitazone MPO.Source: Table 5 · p. 30, §8.5 · pp. 30–31