Source-linked AI summary
Fréchet ChemNet Distance: A metric for generative models for molecules in drug discovery
Kristina Preuer, Philipp Renz, Thomas Unterthiner, Sepp Hochreiter, Günter Klambauer
TL;DR
Evaluating molecular generative models is difficult because no consistently used metric captures validity, chemical and biological meaningfulness, and diversity together. The paper introduces Fréchet ChemNet Distance (FCD), which clearly detected simulated chemical, biological, and diversity biases and ranked generated distributions by closeness to real molecules.
Problem
Molecular generative models lack a consistently used evaluation criterion that jointly captures validity, chemical and biological meaningfulness, and diversity.
Method
FCD compares real and generated molecule distributions using Fréchet distance between mean and covariance of ChemNet penultimate-layer activations.
Results
Across experiments, FCD clearly detected four simulated generative-model flaws, including mode collapse and chemical or biological biases, as well as activity-target bias.
Takeaways & Limitations
FCD provides a comprehensive single metric for evaluating molecular generative models and can also assess graph-generated molecules after conversion to SMILES.
Abstract
from arXiv · showhide
The new wave of successful generative models in machine learning has increased the interest in deep learning driven de novo drug design. However, assessing the performance of such generative models is notoriously difficult. Metrics that are typically used to assess the performance of such generative models are the percentage of chemically valid molecules or the similarity to real molecules in terms of particular descriptors, such as the partition coefficient (logP) or druglikeness. However, method comparison is difficult because of the inconsistent use of evaluation metrics, the necessity for multiple metrics, and the fact that some of these measures can easily be tricked by simple rule-based systems. We propose a novel distance measure between two sets of molecules, called Fréchet ChemNet distance (FCD), that can be used as an evaluation metric for generative models. The FCD is similar to a recently established performance metric for comparing image generation methods, the Fréchet Inception Distance (FID). Whereas the FID uses one of the hidden layers of InceptionNet, the FCD utilizes the penultimate layer of a deep neural network called ChemNet, which was trained to predict drug activities. Thus, the FCD metric takes into account chemically and biologically relevant information about molecules, and also measures the diversity of the set via the distribution of generated molecules. The FCD's advantage over previous metrics is that it can detect if generated molecules are a) diverse and have similar b) chemical and c) biological properties as real molecules. We further provide an easy-to-use implementation that only requires the SMILES representation of the generated molecules as input to calculate the FCD. Implementations are available at: https://www.github.com/bioinf-jku/FCD
Introduction
The paper addresses the lack of a consistently used evaluation criterion for molecular generative models, noting that validity and individual property metrics can be misleading. It introduces the Fréchet ChemNet Distance (FCD), which uses ChemNet representations to combine validity-relevant, chemical, biological, and diversity information into one score.
- Motivation: No evaluation criterion is consistently used across publications, while validity can be maximized by generating simple molecules and individual property metrics provide limited assessment.The paper argues that a proficient metric should capture validity, chemical and biological meaningfulness, and diversity.
- Method: FCD measures the distance between real and generated molecular distributions using the mean and covariance of ChemNet penultimate-layer activations.The approach follows the strategy of the Fréchet Inception Distance for image-generation models.
- Method: ChemNet was trained on about 6 000 assays from ChEMBL, ZINC, and PubChem to predict bioactivities from one-hot encoded SMILES representations.The FCD uses the hidden representation of ChemNet’s second LSTM layer after processing the full input sequence.
- Evaluation: The experiments test whether FCD detects low druglikeness, high logP, low SA scores, mode collapse, and biological bias toward molecules from the same target class.The comparison includes mean logP, mean druglikeness, mean SA score, and internal diversity with Tanimoto distance.
- Conclusions: FCD detected four potential flaws in generative models and also captured biological bias, while supporting graph-generated molecules after conversion to SMILES.The paper further compares FCD with a fingerprint-based Fréchet distance.