PaperlayerView exact revision on arXiv

Source-linked AI summary

SELFormer: Molecular Representation Learning via SELFIES Language Models

Atakan Yüksel, Erva Ulusoy, Atabey Ünlü, Tunca Doğan

arXiv:2304.04662v2q-bio.QMcs.LG

TL;DR

Molecular representation learning offers a computational approach to chemical analysis, but many chemical language models rely on SMILES, whose validity and robustness have limitations. SELFormer learns representations from SELFIES using a transformer and is pretrained on two million drug-like molecules. The reported evaluation found strong results across molecular-property tasks, including a 10% increase in overall ROC values on SIDER versus MolCLR.

  • Problem

    Many chemical language models use SMILES, which can be non-unique and have limitations in validity, spatial information, and syntactic and semantic robustness.

    Source: §1 · p. 2

  • Method

    SELFormer is a transformer chemical language model that learns molecular representations from SELFIES and was pretrained on 2 million drug-like ChEMBL molecules.

    Source: §1 · p. 3, §3.1 · p. 5

  • Results

    SELFormer outperformed existing approaches on multiple property-prediction tasks, including drug side effects, brain-blood barrier permeability, and aqueous solubility; on SIDER, its overall ROC values increased 10% versus MolCLR.

    Source: §4.3.1 · p. 10, §5 · p. 16

  • Takeaways & Limitations

    The authors provide SELFormer as a programmatic tool with trained models and pre-calculated ChEMBL representation vectors for computational chemistry research.

    Source: §1 · p. 3

    SELFormer's 2 million-molecule, 86-million-parameter scale makes its comparison with MolFormer, trained on 1.1 billion molecules, not fairly comparable.

    Source: §4.3.1 · p. 11, §5 · p. 16

Abstract

from arXiv · show

Automated computational analysis of the vast chemical space is critical for numerous fields of research such as drug discovery and material science. Representation learning techniques have recently been employed with the primary objective of generating compact and informative numerical expressions of complex data. One approach to efficiently learn molecular representations is processing string-based notations of chemicals via natural language processing (NLP) algorithms. Majority of the methods proposed so far utilize SMILES notations for this purpose; however, SMILES is associated with numerous problems related to validity and robustness, which may prevent the model from effectively uncovering the knowledge hidden in the data. In this study, we propose SELFormer, a transformer architecture-based chemical language model that utilizes a 100% valid, compact and expressive notation, SELFIES, as input, in order to learn flexible and high-quality molecular representations. SELFormer is pre-trained on two million drug-like compounds and fine-tuned for diverse molecular property prediction tasks. Our performance evaluation has revealed that, SELFormer outperforms all competing methods, including graph learning-based approaches and SMILES-based chemical language models, on predicting aqueous solubility of molecules and adverse drug reactions. We also visualized molecular representations learned by SELFormer via dimensionality reduction, which indicated that even the pre-trained model can discriminate molecules with differing structural properties. We shared SELFormer as a programmatic tool, together with its datasets and pre-trained models. Overall, our research demonstrates the benefit of using the SELFIES notations in the context of chemical language modeling and opens up new possibilities for the design and discovery of novel drug candidates with desired features.

1 Introduction

Drug discovery is costly and time-consuming, motivating computational molecular representation learning. SELFormer addresses limitations of SMILES-based chemical language models by learning molecular representations from SELFIES.

1 Introduction

  • Drug discovery is expensive, resource-intensive, and time-consuming, making it impractical to study numerous molecules for potential drugs.Source: §1 · p. 2
  • Representation learning automatically discovers meaningful data representations intended to capture underlying structure and patterns, with potentially reusable vectors across tasks.Source: §1 · p. 2
  • SMILES-based chemical language models face issues with non-unique strings, invalid chemical properties, limited spatial information, and weak syntactic and semantic robustness.Source: §1 · p. 2
  • SELFormer uses SELFIES with a transformer-based chemical language model to learn concise, flexible molecular representations for downstream drug-discovery tasks.The model was pretrained on 2 million drug-like ChEMBL molecules, and the authors provide trained models and pre-calculated vectors.Source: §1 · p. 3

2 Related Work

Related work spans chemical language models based on SMILES or InChI and graph-based molecular learning. The cited studies report both competitive benchmark performance and trade-offs involving training scale, model accessibility, and computational cost.

2 Related Work

  • MolBert uses BERT with SMILES inputs, was pretrained on 1.6 million ChEMBL-derived molecules, and achieved high performance on virtual-screening and QSAR benchmarks.Its tested self-supervised tasks included masked language modeling, SMILES equivalence prediction, and physicochemical-property prediction.Source: §2 · p. 3
  • ChemBERTa used masked-token prediction on 77 million PubChem molecules, did not beat baselines on most downstream tasks, but achieved state-of-the-art toxicity prediction on Tox21.The cited work reported that masked language modeling positively affected downstream performance; ChemBERTa-2 later improved scores on the same benchmarks.Source: §2 · p. 3
  • Translating the InChI predicts chemical IUPAC names character by character from standard InChI using a sequence-to-sequence model, with 91% accuracy and lower precision for inorganic compounds.Source: §2 · p. 3
  • ChemFormer uses BART-style denoising pretraining on 100 million ZINC SMILES strings and reported competitive results against SMILES- and graph-based models.Its downstream work included regression, synthesis, and retrosynthesis tasks.Source: §2 · p. 4
  • MolFormer was trained on 1.1 billion molecules and outperformed compared models on regression tasks and half of classification tasks, while its large size may limit accessibility.The authors also concluded that large chemical language models perform better than geometric deep-learning models.Source: §2 · p. 4
  • Graph methods encode atoms and bonds as nodes and edges; D-MPNN aggregates connected-node information and reported performance that matched or exceeded predecessors across benchmark areas.Related approaches include distance-based DimeNet, 2D/3D GeomGCL, graph pretraining, contrastive learning, and knowledge-guided pretraining.Source: §2 · p. 4
  • Graph-based models face expressiveness and generalization challenges, while higher-order or 3D methods can increase computational requirements and expense.The cited discussion also notes that clean large-scale data for 3D modeling may be unavailable and some models may not generalize to large molecules.Source: §2 · pp. 4–5

3 Methodology

SELFormer learns molecular embeddings from SELFIES with a self-supervised transformer encoder, then fine-tunes the pretrained model on molecular property tasks.

3 Methodology

  • The workflow first pre-trains a transformer encoder with masked language modeling on SELFIES, then independently fine-tunes it for molecular property classification and regression.The overview depicts these as pre-training and downstream-task stages.Source: §3 · p. 5, Figure 1 · p. 6

3.1 Pre-training

  • Pre-training converts ChEMBL SMILES into SELFIES and uses byte-level BPE tokenization with a RoBERTa architecture trained on the masked language modeling objective.The conversion yielded 2,084,472 unique SELFIES molecules from 2,084,725 compounds; hyperparameter selection used a sequential search on 100,000 molecules and nearly 100 models.Source: §3.1 · p. 5

3.2 Fine-tuning

  • For fine-tuning, a two-layer classification or regression head maps the pretrained encoder embedding to class probabilities or a continuous prediction across nine MoleculeNet tasks.The tasks include BBBP, SIDER, Tox21, HIV, BACE, FreeSolv, ESOL, Lipophilicity, and PDBbind.Source: §3.2 · pp. 5–6

3.2.1 Classification tasks

  • Classification covers binary BACE, BBBP, and HIV tasks and multilabel SIDER and Tox21 tasks, trained with cross-entropy and assessed using ROC and precision-recall curves.SIDER contains 27 adverse-reaction categories for 1,427 approved drugs; Tox21 covers 12 targets.Source: §3.2.1 · pp. 6–7

3.2.2 Regression tasks

  • Regression evaluates aqueous solubility, hydration-free energy, lipophilicity, and binding affinity with mean squared error training loss and RMSE scoring.Both random and scaffold splits use 80%/10%/10% train, validation, and test partitions; scaffold splitting keeps molecules with the same scaffold in one split.Source: §3.2.2 · pp. 7–8

4 Results & Discussion

SELFormer’s scaffold-split results were strongest on SIDER and ESOL, while performance varied across tasks and split settings. Ablations, molecule-level examples, and UMAP visualizations further examined fine-tuning and the representations learned.

4.1 Pre-training and Optimization

  • Pre-training on the full ChEMBL dataset yielded validation losses of 0.328 for SELFormer-Lite and 0.306 for SELFormer.The models were selected after sequential hyperparameter exploration; Table 2 lists their configurations.Source: §4.1 · p. 8, Table 2 · p. 8

4.2 Ablation Study

  • Fine-tuning improved performance on most datasets, and SELFormer generally outperformed SELFormer-Lite; the initial SIDER results favored the pre-trained model, while 25 epochs slightly beat 50 on some tasks.The authors attributed the SIDER exception as possibly resulting from non-optimal hyperparameters.Source: §4.2 · pp. 8–9, Table 3 · p. 9, Table 4 · p. 9

4.3.1 Classification Tasks

  • On scaffold splits, SELFormer improved SIDER overall ROC by 10% over MolCLR and outperformed ChemBERTa-2 by an average of 12% on BACE, BBBP, and HIV.It also outperformed all existing approaches on SIDER, while KPGT remained ahead on BBBP.Source: Table 5 · p. 10, §4.3.1 · p. 10

4.3.2 Regression Tasks

  • On scaffold splits, SELFormer beat the best ESOL competitor, GEM, by more than 15% in RMSE; on random splits, it led on ESOL and was comparable on other tasks.The scaffold-split results also show mixed comparisons on FreeSolv and Lipophilicity.Source: §4.3.2 · p. 11, Table 7 · p. 12, Table 8 · p. 12

4.4 Use-case Analysis

  • Selected-molecule checks aligned SELFormer’s BBBP and many SIDER predictions with known observations, but ESOL examples showed discrepancies amid uncertainty in reported solubility values.The SIDER example also included a false-positive class prediction.Source: §4.4 · pp. 12–14, Figure 2 · p. 13

4.5 Visualization of the embedding space

  • UMAP showed separation of BACE activity groups even in pre-trained embeddings, with slight fine-tuning improvements; BBBP and HIV embeddings also showed class-related structure.The authors caution that 2-D visualizations may not fully express learned features.Source: §4.5 · p. 14, Figure 3 · p. 15
  • In the ChEMBL target-family analysis, some UMAP clusters were homogeneous, but distinct family-based clusters were not observed, mainly because of missing data and task difficulty.Nearby selected molecules had higher scaffold similarity than molecules from distant cluster pairs.Source: §4.5 · pp. 14–16

5 Impact and Future Directions

SELFormer performs well and surpasses existing approaches on several molecular-property tasks, while larger HIV and Tox21 datasets constrained hyperparameter exploration. The authors point to model and data scaling, multimodal inputs, and further optimization as future directions.

5 Impact and Future Directions

  • SELFormer performs well and surpasses existing approaches on tasks including drug-side-effect prediction, blood–brain barrier permeability, and aqueous solubility.The authors note that larger HIV and Tox21 datasets presented challenges because they required exploration of larger hyperparameter spaces.Source: §5 · p. 16
  • Pre-training alone distinguished molecules across properties, while fine-tuning slightly improved 2-D embeddings and careful tuning could significantly improve prediction performance.Compound-target visualizations captured some bioactivity-related information but did not show distinct target-protein-family inhibitor clusters.Source: §5 · p. 16
  • With a pre-training dataset of 2 million molecules and 86 million trainable parameters, SELFormer was not a “large model”; nevertheless, it performed well on the respective supervised prediction tasks.Source: §5 · p. 16
  • The authors propose pre-training on larger molecule datasets and increasing the number of learnable parameters as possible routes to better results, pending further experiments.They specifically mention the entire ZINC database as a potential larger pre-training source.Source: §5 · p. 16
  • The authors identify multimodal models combining molecular structures with text and functional annotations as a future direction.They suggest biomedical knowledge graphs containing target interactions, pathway associations, and disease indications as a potentially useful data structure.Source: §5 · pp. 16–17
  • ChEMBL molecule embeddings placed structurally similar scaffolds close together, but distinct inhibitor clusters by target-protein family were not observed.The authors interpret the scaffold similarities among nearby molecules as evidence of informative representations and report only a certain amount of functional information in the compound-target visualization.Source: Figure 4 · p. 17
  • With further exploration and optimization, SELFormer or other SELFIES-based models could become alternatives to SMILES-based chemical language models for complex molecular structures.The authors describe this as potential, not an established outcome.Source: §5 · pp. 17–18

Related papers

Loading 2304.04662v2…