Source-linked AI summary
SMILES Transformer: Pre-trained Molecular Fingerprint for Low Data Drug Discovery
Shion Honda, Shoi Shi, Hiroki R. Ueda
TL;DR
Molecular property prediction in drug discovery often relies on sparse rule-based fingerprints and labeled datasets that may be small. This paper pretrains a Transformer sequence-to-sequence model on unlabeled SMILES to create molecular fingerprints and introduces a data-efficiency metric. Across MoleculeNet benchmarks, ST is especially effective with limited labeled data and is comparable to GraphConv when larger labeled datasets are available.
Problem
Existing molecular fingerprints and prediction approaches are limited for shallow models or small labeled datasets, while data-efficiency evaluation lacks an established metric.
Method
SMILES Transformer pretrains a Transformer-based sequence-to-sequence model on unlabeled SMILES and pools contextualized symbol representations into 1024-dimensional molecular fingerprints.
Results
ST fingerprints work well across MoleculeNet tasks, are especially effective with insufficient labeled data, and are comparable to GraphConv when large labeled datasets are available.
Takeaways & Limitations
ST fingerprints can be used with arbitrary predictive models, and ST has better DEM than existing methods in 5 out of 10 downstream tasks.
Takeaways & Limitations
Future work identifies ST's sequence-length limit and proposes longer-sequence Transformers, multitask training, and enumerated-SMILES information as improvement directions.
Abstract
from arXiv · showhide
In drug-discovery-related tasks such as virtual screening, machine learning is emerging as a promising way to predict molecular properties. Conventionally, molecular fingerprints (numerical representations of molecules) are calculated through rule-based algorithms that map molecules to a sparse discrete space. However, these algorithms perform poorly for shallow prediction models or small datasets. To address this issue, we present SMILES Transformer. Inspired by Transformer and pre-trained language models from natural language processing, SMILES Transformer learns molecular fingerprints through unsupervised pre-training of the sequence-to-sequence language model using a huge corpus of SMILES, a text representation system for molecules. We performed benchmarks on 10 datasets against existing fingerprints and graph-based methods and demonstrated the superiority of the proposed algorithms in small-data settings where pre-training facilitated good generalization. Moreover, we define a novel metric to concurrently measure model accuracy and data efficiency.
1 Introduction
Drug discovery is costly and time-consuming, while molecular prediction methods often depend on large labeled datasets. The paper introduces SMILES Transformer, a pretrained text-based fingerprint, and a data-efficiency metric for evaluating performance in small-data settings.
- Drug discovery can take more than a decade and requires costly efficacy and safety validation, motivating machine-learning methods.
- Graph-based molecular predictors often outperform SMILES-based approaches but are designed for large fully labeled datasets.
- SMILES Transformer pretrains a Transformer on unlabeled SMILES to produce continuous, data-driven molecular fingerprints for downstream prediction.
- The study proposes a scalar data-efficiency metric that measures model performance across different training-set sizes.
- ST fingerprints enable state-of-the-art data efficiency in 5 out of 10 MoleculeNet datasets.
- The paper evaluates ST training, fingerprint extraction, data efficiency, and comparisons with other methods across 10 datasets.
2 Methods
The method pre-trains a Transformer on unlabeled SMILES to produce 1,024-dimensional molecular fingerprints, then introduces DEM to average predictive performance across training-data sizes.
- Architecture: SMILES Transformer uses an encoder-decoder network with four Transformer blocks per encoder and decoder, each employing four-head attention and 256-dimensional embeddings.
- Pre-training: 861,000 unlabeled SMILES from ChEMBL24 are symbolized, one-hot encoded, and randomly enumerated during unsupervised pre-training.
- Fingerprint extraction: A 1,024-dimensional molecule fingerprint concatenates mean- and max-pooled final-layer outputs with first outputs from the final and penultimate layers.
- Data Efficiency Metric: Data Efficiency Metric averages a task-appropriate metric for models trained with different training-data sizes.
- Data Efficiency Metric: The experiments vary training-data percentages exponentially from 1.25% to 80% to evaluate performance across dataset sizes.
3 Experiments
The experiments evaluate SMILES Transformer from multiple perspectives, including benchmark performance, latent-space structure, model simplicity, SMILES-length stratification, and additional robustness analyses.
- Experimental design: Five experiments examine downstream performance, latent-space visualizations, linear-model validation, SMILES-length stratification, and additional properties of SMILES Transformer.
3.1 Performance on Downstream Tasks
SMILES Transformer is evaluated against fingerprints and graph convolution on 10 MoleculeNet datasets using simple downstream predictors and data-efficiency metrics.
- Datasets: 10 MoleculeNet datasets span physical chemistry, biophysics, and physiology, with differing domains, task types, and sizes.
- Baselines: The comparison includes ECFP4, RNNS2S, and GraphConv alongside the pre-trained SMILES Transformer.
- Evaluation setup: All fingerprints have 1,024 dimensions and are evaluated with simple multilayer perceptron classifiers or regressors across random train-test splits and 20 trials.
- Overall performance: ST achieves the best Data Efficiency Metric score in 5 out of 10 datasets, followed by ECFP and GraphConv.
- Performance by dataset: ST performs best at almost all training sizes in ESOL, FreeSolv, BBBP, and ClinTox, whereas ECFP or GraphConv performs better in Lipophilicity, MUV, BACE, and SIDER.
3.2 Visualization of the Latent Space
The latent-space analysis reduces ST fingerprints to two dimensions, colors molecules by target value, and decodes nearest-neighbor molecules along selected trajectories.
- Visualization procedure: For each dataset, the analysis calculates 1,024-dimensional ST fingerprints, reduces them to two dimensions with t-SNE, and colors points by target value.
- Visualization procedure: A trajectory through the two-dimensional space is divided into 12 points, whose nearest-neighbor molecules are drawn for interpretation.
- FreeSolv: In FreeSolv, target values show a clear upper-left to lower-right gradation while decoded molecules become simpler along the trajectory.
- BBBP and ClinTox: In BBBP and ClinTox, categorical target values are separated successfully, but decoded molecules show no clear trends.
3.3 Application of Simple Predictive Models
The study tested whether ST fingerprints retain their advantage when paired with simple linear predictors. Across eight datasets, they achieved the best scores on five, suggesting the fingerprint—not model complexity—drives much of the performance.
- ST fingerprints with ridge or logistic regression achieved the best scores on 5 of 8 datasets.MUV and SIDER were excluded because highly imbalanced columns caused errors in the scikit-learn solver and ROC-AUC functions.
- The results indicate that ST fingerprints provide strong representations across different predictive-model choices.
3.4 Stratified Scores by the Size of Molecules
The authors examined ST’s advantage on BBBP by grouping molecules according to SMILES length. ST performance increased with sequence length, unlike GraphConv, suggesting longer SMILES provide richer information for discrimination.
- ST’s ROC-AUC increased across BBBP groups with increasing SMILES length.RNNS2S showed a similar trend, whereas GraphConv performance remained more or less constant.
- Longer SMILES were associated with richer information for ST-based discrimination.
3.5 Comparison with Record Scores
The paper compared carefully tuned ST models with reported record scores and examined model behavior across training sizes and latent spaces. ST was competitive with established fingerprints and graph-based methods, but led only on ClinTox in the large-data comparison.
- Latent-space trajectories showed a clear FreeSolv gradation toward simpler molecules, while BBBP and ClinTox target categories were separated without clear decoded-molecule trends.ST fingerprints were reduced to two dimensions with t-SNE, and neighboring points were decoded into molecules.
- DEM comparisons used ridge regression and logistic regression with L2 penalty across eight MoleculeNet datasets.The metric reports whether higher or lower values are better depending on the task.
- ST achieved first place only on ClinTox while performing comparably to ECFP and graph-based models on the other datasets.This comparison used reported MoleculeNet scores for ECFP, GraphConv, and Weave.
4 Conclusions
The paper introduces ST as a Transformer-based, data-driven molecular fingerprint learned from unlabeled SMILES and reports its strongest benefits in limited-label settings. It also proposes DEM and identifies sequence length, multitask training, and SMILES enumeration as future directions.
- ST is produced by a Transformer-based sequence-to-sequence model pre-trained on a large corpus of unlabeled SMILES.
- ST fingerprints performed well with different predictive models and were especially effective when labeled data were limited.With large labeled datasets, ST remained comparable to state-of-the-art baselines such as GraphConv.
- ST achieved better DEM than existing methods in 5 of 10 downstream tasks.
- Future work proposes longer-sequence Transformers, multitask learning with molecular descriptors, and enumerated SMILES to improve representations.The authors connect these directions to alleviating ST’s length limit, learning chemistry-relevant representations, and restricting latent space.