PaperlayerView exact revision on arXiv

Source-linked AI summary

Keeping it Simple: Language Models can learn Complex Molecular Distributions

Daniel Flam-Shepherd, Kevin Zhu, Alán Aspuru-Guzik

arXiv:2112.03041v1cs.LGcs.AIq-bio.QM

TL;DR

The paper asks whether simple language models can learn complex molecular distributions as effectively as graph generative models. It constructs three challenging tasks and compares SMILES- and SELFIES-based RNNs with JTVAE and CGVAE. Across these tasks, language models learn the distributions more effectively, including high-penalized-LogP, multi-modal, and very large-molecule distributions.

  • Problem

    The paper investigates whether simple language models can learn complex molecular distributions, which is important for generative models used in inverse design.

  • Method

    The authors define three challenging molecular-distribution tasks and compare SMILES- and SELFIES-trained RNNs with JTVAE and CGVAE graph models.

  • Results

    Language models outperform the graph baselines across the tasks, learning high penalized LogP, multi-modal, and large-molecule distributions.

  • Takeaways & Limitations

    Simple language models are powerful and flexible generative models for learning diverse complex molecular distributions.

Abstract

from arXiv · show

Deep generative models of molecules have grown immensely in popularity, trained on relevant datasets, these models are used to search through chemical space. The downstream utility of generative models for the inverse design of novel functional compounds depends on their ability to learn a training distribution of molecules. The most simple example is a language model that takes the form of a recurrent neural network and generates molecules using a string representation. More sophisticated are graph generative models, which sequentially construct molecular graphs and typically achieve state of the art results. However, recent work has shown that language models are more capable than once thought, particularly in the low data regime. In this work, we investigate the capacity of simple language models to learn distributions of molecules. For this purpose, we introduce several challenging generative modeling tasks by compiling especially complex distributions of molecules. On each task, we evaluate the ability of language models as compared with two widely used graph generative models. The results demonstrate that language models are powerful generative models, capable of adeptly learning complex molecular distributions -- and yield better performance than the graph models. Language models can accurately generate: distributions of the highest scoring penalized LogP molecules in ZINC15, multi-modal molecular distributions as well as the largest molecules in PubChem.

I. RESULTS

The paper tests recurrent language models and graph generative models on three challenging molecular-distribution tasks. It compares generated and training distributions using property plots, validity-related metrics, and Wasserstein distances.

  • Tasks: Three tasks target high penalized LogP molecules, multi-modal molecular distributions, and the largest molecules in PubChem.Each task uses a dataset compiled from relevant subsets of larger molecular databases.
  • Evaluation: Performance is assessed by comparing training and generated property distributions with histograms, KDEs, and Wasserstein distances.The Wasserstein oracle compares separate training samples, while generated samples are deduplicated and matched in size across models.
  • Evaluation: Properties include QED, SA, LogP, molecular weight, Bertz complexity, and natural product likeness, alongside validity, uniqueness, and novelty.These metrics cover molecular properties and whether generated molecules are diverse, real, and distinct from training data.
  • Models: Models include SMILES- and SELFIES-trained LSTM RNNs alongside the JTVAE and CGVAE graph generative models.The models use distinct molecular representations: strings for RNNs and trees or graphs for graph models.

A. Penalized LogP Task

The penalized LogP task asks models to learn a distribution of high-scoring ZINC15 molecules rather than find only an individual high-scoring compound. Language models better reproduce both the main distribution and its high-LogP tail than the graph baselines.

  • Task definition: The task screens ZINC15 for molecules with penalized LogP exceeding 4.0, requiring models to generate from this difficult region of chemical space.The benchmark penalizes LogP by synthesizability and unrealistic rings.
  • Main distribution: Language models perform significantly better than graph models, with the SELFIES RNN producing a slightly closer match to the training distribution.The graph models generate many molecules below the lowest training scores, while the RNNs better match LogP and QED distributions.
  • Quantitative evaluation: Both RNNs achieve lower Wasserstein distances than CGVAE and JTVAE, while the SMILES RNN comes closest to the TRAIN oracle.The oracle represents the optimal baseline for matching the training distribution.
  • High-LogP tail: For penalized LogP values ≥6.0, both RNNs learn the subtle tail, whereas graph models largely ignore it and remain closer to the main mode.High-scoring training molecules typically contain very long carbon chains and fewer rings, which the RNNs capture more closely.

B. Multi-distribution Task

The multi-distribution task tests whether models can learn a dataset combining four molecular-weight regions and producing four modes. Both RNNs captured all modes, while graph models missed or poorly learned several modes; SM-RNN achieved the lowest Wasserstein metrics.

  • Dataset: The training set combines GDB13, ZINC, CEP, and POLYMERS subsets into approximately 200K molecules spanning four molecular-weight modes.GDB13 contributes molecules with MW ≤185, ZINC molecules with MW ≤425, CEP molecules with 460 ≤MW ≤600, and POLYMERS molecules with MW ≥600.
  • Model comparison: Both RNN models captured every mode in the multi-distribution training set with comparable quality.The RNN-generated distributions also produced molecules better resembling the training data than the graph models.
  • Model comparison: JTVAE missed the first mode and poorly learned the ZINC and CEP modes, while CGVAE underestimated ZINC and missed CEP.CGVAE and JTVAE also barely distinguished the main modes in the plotted distributions.
  • Quantitative evaluation: The SMILES RNN achieved the lowest Wasserstein metrics, followed by the SELFIES RNN, JTVAE, and CGVAE.Table II reports Wasserstein metrics for LogP, SA, QED, MW, BCT, and NP, with TRAIN as an oracle baseline.

C. Large Scale Task

The large-scale task evaluates learning distributions of very large, diverse PubChem molecules. Graph models failed to train, whereas both RNNs generated molecules as large and varied as the training data and learned its substructure distribution.

  • Task construction: The dataset contains 300K PubChem molecules with more than 100 heavy atoms, including small biomolecules, photovoltaics, and other molecular types.Most molecules fall in the 1500–2000 molecular-weight range.
  • Substructure learning: Figure 6 compares generated and training molecules through per-molecule fragment histograms and examples at 1500 ≤Molecular Weight ≤2000 and Molecular Weight ≥4000.The figure uses examples to illustrate generation across intermediate and very large molecular-weight ranges.
  • Graph-model performance: Both graph models failed to train and were entirely incapable of learning the large-molecule training data.JTVAE’s tree decomposition produced a fixed vocabulary of approximately 11,000 substructures.
  • RNN performance: Both RNNs generated molecules as large and varied as the training data.The SELFIES RNN matched the data distribution more closely, while SELFIES strings remain valid even for these very long representations.
  • Substructure learning: The RNNs adequately learned distributions of fragments, atom fragments, and single- and multi-ring fragments per molecule.They continued generating molecules from the less frequent large-molecule region, including molecules with molecular weight above 3000.

D. Metrics

The paper evaluates validity, uniqueness, and novelty using 10K generated molecules per model and task. Graph models performed better on these standard metrics in the first two tasks, while SELFIES RNN approached them.

  • Metric definitions: Validity is the ratio of valid to generated molecules, uniqueness is the ratio of unique nonduplicate molecules to valid molecules, and novelty is the ratio of unique non-training molecules to all unique molecules.These statistics are computed from the same 10K generated molecules for each model and task.
  • Results: The SELFIES RNN approached the graph models’ standard-metric performance, while the SMILES RNN performed worst because of its grammar.The SMILES RNN was not substantially worse than the other models.

II. DISCUSSION

The discussion concludes that language models can learn diverse complex molecular distributions and often outperform the graph baselines. SELFIES improves several standard metrics and large-scale performance, but the experiments also provide evidence of language-model overfitting.

  • Main conclusion: The experiments introduce three complex molecular-distribution tasks and show that language models learn varied distributions more effectively than the popular graph baselines.The tasks cover high penalized LogP molecules, multi-modal distributions, and the largest PubChem molecules.
  • Representation effects: SELFIES improves language-model performance, particularly on the large-scale task, and consistently yields better validity, uniqueness, and novelty metrics.The discussion contrasts these improvements with SMILES RNN performance across all tasks.
  • Open issues: The results give evidence that language models can suffer from overfitting, motivating further work on combating it.The discussion links SELFIES’ better standard metrics and worse Wasserstein metrics than SMILES to different overfitting behavior.
  • Future work: The authors propose that the introduced tasks and datasets can motivate generative models addressing the difficulties experienced by graph baselines.Future work will examine language models on increasingly large snapshots of chemical space.

III. METHODS

The methods section specifies task-specific recurrent-language-model architectures and training settings for the penalized LogP and multi-distribution tasks.

  • Penalized LogP Task: For the penalized LogP task, SM-RNN uses a two-layer, 400-unit LSTM with 0.2 dropout and learning rate 0.0001.
  • Penalized LogP Task: For the penalized LogP task, SF-RNN uses a two-layer, 600-unit LSTM, with additional settings truncated in the supplied passage.
  • Multi-Distribution Task: For the multi-distribution task, SM-RNN uses a three-layer, 512-unit LSTM with 0.5 dropout and learning rate 0.0001.
  • Multi-Distribution Task: For the multi-distribution task, SF-RNN uses a two-layer, 500-unit LSTM, with additional settings truncated in the supplied passage.

V. CODE AVAILABILITY

The paper reports that its graph-model training code comes from public repositories, while its recurrent models use publicly available char-rnn code.

  • JTVAE training uses code from the public wengong-jin/icml18-jtnn repository.
  • CGVAE training uses code from the public microsoft/constrainedgraph-variational-autoencoder repository.
  • The RNN models were trained using char-rnn code from the molecularsets/moses repository.
  • Trained models are available upon request.

SUPPLEMENTAL MATERIALS

The supplemental materials provide figures and tables showing training or generated molecules across the penalized LogP, multi-distribution, and large-scale tasks.

  • Molecular examples: The supplemental materials identify examples involving molecules with more than 10 rings and molecules from the SMILES RNN.
  • Penalized LogP Task: The penalized LogP supplemental figures show training molecules with different properties and generated molecules with penalized LogP below 4 from graph models.
  • Model comparisons: Figure labels distinguish TRAIN, CGVAE, JTVAE, SF-RNN, and SM-RNN outputs in the supplemental comparisons.
  • Model-generated molecules: Tables I and II present molecules generated by each model for the penalized LogP and multi-distribution tasks.
  • Large Scale Task: Large-scale supplemental figures show training molecules, generated molecules from SMILES and SELFIES RNNs, and generated molecules with fewer than 100 heavy atoms.
  • Large Scale Task: Additional large-scale figures show training molecules from each LogP mode and examples from lower- and higher-LogP modes.
Loading 2112.03041v1…