Source-linked AI summary
Molecular Sets (MOSES): A Benchmarking Platform for Molecular Generation Models
Daniil Polykovskiy, Alexander Zhebrak, Benjamin Sanchez-Lengeling, Sergey Golovanov, Oktai Tatanov, Stanislav Belyaev, Rauf Kurbanov, Aleksey Artamonov, Vladimir Aladinskiy, Mark Veselov, Artur Kadurin, Simon Johansson, Hongming Chen, Sergey Nikolenko, Alan Aspuru-Guzik, Alex Zhavoronkov
TL;DR
Molecular generative models lack standardized ways to compare their quality and diversity. MOSES addresses this gap with shared datasets, models, and sample-based metrics, and its comparisons identify CharRNN as the strongest performer on key metrics while highlighting limits of aggregate evaluation.
Problem
Molecular generative models are difficult to compare and rank because the field lacks unified evaluation protocols for assessing generated structures.
Method
MOSES standardizes molecular-generation benchmarking through datasets, preprocessing utilities, baseline models, and metrics applicable to samples from both explicit and implicit distributions.
Results
CharRNN performs best on the key FCD, Fragment, and Scaffold metrics, while discovering 11% novel scaffolds in the reported comparison.
Takeaways & Limitations
MOSES provides a unified testbed for fair comparison, with FCD/Test suggested for general evaluation and specialized metrics retained for diagnosis and task-specific goals.
Takeaways & Limitations
FCD/Test compresses multiple aspects into one number but does not reveal which specific issues produce high values, so interpretable metrics remain necessary.
Abstract
from arXiv · showhide
Generative models are becoming a tool of choice for exploring the molecular space. These models learn on a large training dataset and produce novel molecular structures with similar properties. Generated structures can be utilized for virtual screening or training semi-supervised predictive models in the downstream tasks. While there are plenty of generative models, it is unclear how to compare and rank them. In this work, we introduce a benchmarking platform called Molecular Sets (MOSES) to standardize training and comparison of molecular generative models. MOSES provides a training and testing datasets, and a set of metrics to evaluate the quality and diversity of generated structures. We have implemented and compared several molecular generation models and suggest to use our results as reference points for further advancements in generative chemistry research. The platform and source code are available at https://github.com/molecularsets/moses.
1 Introduction
MOSES addresses the difficulty of exploring and evaluating molecular generative models by providing a standardized benchmark for distribution learning. It combines shared data, models, preprocessing, and sample-based metrics to support comparison.
- The chemical space contains an estimated 10^23 to 10^80 pharmacologically sensible molecules, making complete computational exploration intractable.
- Deep generative models support inverse design of drug compounds for target properties such as solubility, ease of synthesis, and toxicity.
- Existing benchmark sets provide common measurements for machine-learning models, but molecular generative modeling lacks standardized data and evaluation protocols.
- MOSES provides a standardized dataset, preprocessing utilities, evaluation metrics, and molecular generation models as a unified testbed.
- Distribution learning models approximate an unknown molecular distribution p(x) with a learned distribution q(x) from training samples.
- MOSES evaluates both explicit and implicit distribution models using metrics that require only samples from q(x), enabling comparison across model types.
3 Molecular representations
Molecules are represented for machine learning primarily as strings or graphs. String representations support sequence-based generative models, while graph representations encode atoms and bonds for chemoinformatics and deep learning.
- Molecular representations for machine learning are grouped into string and graph representations.These are the two representation views discussed for molecular structures.
- String representations: SMILES is the most widely used string representation for generative machine learning models because sequence-modeling tools are abundant.SMILES traverses a molecular graph in depth-first order and stores the resulting structure as a string.
- String representations: DeepSMILES alters branch and ring-closure syntax to reduce invalid sequences, while SELFIES uses an augmented Chomsky type-2 grammar.Other approaches incorporate SMILES syntax into network architectures to increase the fraction of valid molecules; InChI is more verbose and explicit.
- Molecular graphs: In molecular graphs, nodes represent atoms and edges represent bonds, with hydrogens specified explicitly or inferred from atomic valencies.This representation is used for storing and processing molecular data in chemoinformatics.
- Molecular graphs: Deep learning models can learn directly from molecular graphs using Graph Convolutional, Weave, and Message Passing Networks.Graphs can also be encoded as adjacency and node-feature matrices, an approach used by MolGAN on QM9.
4 Metrics
MOSES evaluates molecular generators with metrics covering validity, uniqueness, novelty, diversity, distributional similarity, and structural similarity. Together, these metrics target failure modes including overfitting, mode collapse, and imbalanced structures.
- Validity checks explicit chemical constraints, while Unique@K checks whether generated molecules collapse to a few repeated structures.Validity uses RDKit parsing; uniqueness is computed for the first K = 1,000 and K = 10,000 valid molecules.
- Novelty measures the fraction of generated molecules absent from training data, with low novelty indicating overfitting.
- Fragment and scaffold similarities compare generated and reference substructure frequencies, but similarity 1 can occur even for different molecules.Fragment similarity uses BRICS fragments, while scaffold similarity uses Bemis–Murcko scaffolds.
- SNN measures average Tanimoto similarity between each generated molecule and its nearest reference-set neighbor.Using Morgan fingerprints, it can be interpreted as precision: molecules far from the reference manifold have low similarity.
- Internal diversity measures diversity within generated molecules and detects mode collapse, where limited samples ignore parts of chemical space.Higher IntDivp values correspond to higher diversity; experiments report IntDiv1 and IntDiv2.
- FCD compares generated and reference sets using ChemNet activations that capture chemical and biological properties; lower values are better.FCD is suggested for hyperparameter tuning and final model selection.
- Property distributions are compared with 1D Wasserstein-1 distances for molecular weight, LogP, synthetic accessibility, and QED.These properties support visual and quantitative assessment of distributional similarity.
5 Dataset
MOSES constructs its benchmark from filtered ZINC Clean Leads molecules and supplies standardized train, test, and scaffold-test splits. The resulting dataset contains nearly two million molecules and diverse scaffolds and fragments.
- The source collection contains 4,591,276 molecules constrained by molecular weight, rotatable bonds, and XlogP before additional chemical filtering.The final filtering removes charged atoms, unsupported elements, and cycles larger than 8 atoms.
- The final dataset contains 1,936,963 molecules, 448,854 unique Bemis–Murcko scaffolds, and 58,315 unique BRICS fragments.Its internal diversity is IntDiv1 = 0.857.
- MOSES recommends non-intersecting train, test, and scaffold-test splits containing 1,584,664, 176,075, and 176,226 molecules, respectively.The scaffold-test set is formed from molecules whose Bemis–Murcko scaffolds come from a random scaffold subset.
- The scaffold-test design evaluates whether models produce novel scaffolds absent from training, while the ordinary test set is a random subset of remaining molecules.
6 Baselines
MOSES compares molecular-generation approaches spanning recurrent, autoencoding, adversarial, junction-tree, latent-space, and classical probabilistic models. The baselines retain published architectures while tuning hyperparameters through random search using FCD.
- MOSES evaluates CharRNN, VAE, AAE, JTN-VAE, LatentGAN, n-gram, HMM, and combinatorial-generator baselines.
- The comparison preserves each model’s published architecture while tuning training and architecture hyperparameters by random search.The selected architecture maximized FCD across multiple tested architectures.
- CharRNN models next-token distributions over SMILES and are trained by maximizing the training-data log-likelihood.
- VAEs map molecular representations into and out of a lower-dimensional latent space using reconstruction and KL-divergence regularization.
- JTN-VAE generates a junction tree of valid chemical substructures before assembling those components into a coherent molecular graph.
7 Platform
MOSES packages its dataset, metrics, and baseline models with utilities for reproducible evaluation and contributions from new models. It recommends repeated runs to assess sensitivity to random initialization.
- The repository and PyPI package provide the dataset, metrics, baseline models, and utilities for computing evaluation results.
- New models should train on the MOSES training set, generate 30,000 samples, and compute metrics with the provided utilities.
- Experiments should run at least three times with different random seeds to estimate sensitivity to parameter initialization.
8 Results
MOSES baseline results expose trade-offs across validity, diversity, novelty, distributional similarity, and property matching. Neural models generally capture dataset statistics, while individual models differ in validity, novelty, FCD, SNN, and scaffold generalization.
- HMM and n-gram models fail to produce valid molecules, whereas the combinatorial generator and JTN-VAE achieve 100% validity through built-in constraints.
- All models avoid forbidden structures, while the combinatorial generator exceeds training-set internal diversity and autoencoder-based models show low novelty.The low novelty of autoencoder-based models is interpreted as overfitting to the training set.
- All neural models show low FCD, with CharRNN achieving the best FCD and VAE achieving the best SNN while also showing low novelty.
- CharRNN discovers 11% novel scaffolds, which the authors suggest indicates good generalization.
- Deep generative models closely match test-set distributions for MW, logP, QED, and SA.HMM is biased toward lighter molecules, while the combinatorial generator produces higher molecular-weight variance than the training set.
9 Discussion
CharRNN performs best on the key metrics, while the metric suite supports task-specific evaluation and FCD/Test offers a general comparison. FCD/Test remains limited because it does not identify the specific issues behind its values.
- CharRNN achieves the best FCD, Fragment, and Scaffold scores among the presented models.These results indicate strong distribution capture without overfitting on the training set.
- The metric suite evaluates molecular generators from different perspectives, so downstream tasks can prioritize the most relevant metric.Scaf/TestSF is suggested for objectives involving novel scaffold discovery.
- FCD/Test is recommended for general evaluation because it combines multiple metric aspects into one value.
- FCD/Test does not reveal which specific issues produce high values, making more interpretable metrics necessary for thorough assessment.
10 Conclusion
MOSES provides a benchmark platform for molecular generation with standardized data, baseline models, metrics, and unified interfaces. The platform is intended to support fair comparison and continued extension of generative-model evaluation.
- MOSES provides molecular structures, baseline models, and evaluation metrics for benchmarking molecular generation.
- The benchmark suite uses unified and extendable programming interfaces for generative models and evaluation metrics.
- MOSES is intended to enable fair and comprehensive comparison of new generative models.
- Future work will extend the repository with additional baseline models and evaluation metrics.
A Medicinal chemistry filters and PAINS filters
The dataset is filtered with medicinal-chemistry and PAINS filters to pre-select compounds more appropriate for drug design and reduce structures associated with toxicity, instability, reactivity, or assay interference.
- Medicinal chemistry filters discard compounds containing structural alerts linked to unstable or reactive groups, toxic metabolites, or intermediates.
- The dataset applies medicinal chemistry filters for rational pre-selection of compounds considered more appropriate for modern drug design and development.
- The filters cover electrophilic groups, reactive heterocycles, azides, unstable aminals and acetals, and compounds with many halogens.The listed alerts include Michael acceptors, alkyl halides, epoxides, isocyanates, aldehydes, imines, aziridines, furans, thiophenes, and related groups.
- PAINS filters target substructures associated with false positives, assay artifacts, unspecific bioactivity, reactivity, chelation, colloidal aggregation, or dyes.
- The diverse MOSES subset is constructed by iteratively adding molecules least similar to the nearest compound in the selected set.
C Hyperparameters and training details
MOSES evaluates diverse molecular generators, including recurrent, autoencoder, adversarial, fragment-based, n-gram, and hidden-Markov approaches, using the displayed model configurations and training procedures.
- CharRNN: CharRNN uses three stacked LSTM layers with hidden dimension 768, dropout 0.2, and Softmax output, trained with Adam for 80 epochs.Training uses batch size 64 and learning rate 10^-3, halved every 10 epochs.
- Autoencoder-based models: VAE uses a bidirectional GRU encoder and a three-layer GRU decoder with dropout probability 0.2.Its KL-term weight increases linearly from 0 to 1 during training.
- Autoencoder-based models: AAE combines recurrent encoder-decoder networks with a discriminator network trained using Adam.The encoder and decoder use LSTM layers, while the discriminator has fully connected layers with ELU activation.
- Autoencoder-based models: JT-VAE results are taken from the official JT-VAE repository.
- LatentGAN: LatentGAN combines a heteroencoder with WGAN-GP to generate molecular latent vectors.The heteroencoder is trained on randomly chosen non-canonical SMILES strings, with Gaussian noise added to latent codes.
- Other generators: The combinatorial generator samples BRICS fragments according to training-set frequencies while omitting combinations that produce invalid molecules.
- Other generators: The n-gram model uses 11-gram statistics and shortens context when no matching statistics are available.In extreme cases, it falls back to a single-token context equivalent to a bigram model.
- Other generators: The HMM uses Baum-Welch training with 200 states on the first 100,000 MOSES training molecules.K-means|| initializes parameters to reduce the cost of batch training.