PaperlayerView exact revision on arXiv

Source-linked AI summary

Translation between Molecules and Natural Language

Carl Edwards, Tuan Lai, Kevin Ros, Garrett Honke, Kyunghyun Cho, Heng Ji

arXiv:2204.11817v3cs.CLcs.AI

TL;DR

The paper addresses scarce paired molecule-language data and the challenge of translating between molecular structures and natural-language descriptions. It introduces MolT5, which self-supervises on unlabeled text and molecule strings before finetuning on two cross-modal tasks and evaluating them with multiple metrics. MolT5-based models achieve high-quality molecule and caption outputs in many cases, while remaining subject to bias, representation, and medical-use limitations.

  • Problem

    Molecule-language tasks lack abundant paired annotations, while varied descriptions make standard reference-based evaluation inadequate.

  • Method

    MolT5 self-supervises a model on unlabeled natural-language text and molecule strings, then finetunes it for molecule captioning and text-guided molecule generation.

  • Results

    MolT5-based models generate high-quality molecules and captions in many cases across the two translation tasks.

  • Takeaways & Limitations

    The framework supports bidirectional translation between natural language and molecules using single-modal pretraining and cross-modal evaluation.

  • Takeaways & Limitations

    Generated molecules should not be used for medical purposes, and the model remains limited by unintended biases and SMILES representation issues.

Abstract

from arXiv · show

We present $\textbf{MolT5}$ $-$ a self-supervised learning framework for pretraining models on a vast amount of unlabeled natural language text and molecule strings. $\textbf{MolT5}$ allows for new, useful, and challenging analogs of traditional vision-language tasks, such as molecule captioning and text-based de novo molecule generation (altogether: translation between molecules and language), which we explore for the first time. Since $\textbf{MolT5}$ pretrains models on single-modal data, it helps overcome the chemistry domain shortcoming of data scarcity. Furthermore, we consider several metrics, including a new cross-modal embedding-based metric, to evaluate the tasks of molecule captioning and text-based molecule generation. Our results show that $\textbf{MolT5}$-based models are able to generate outputs, both molecules and captions, which in many cases are high quality.

1 Introduction

MolT5 targets translation between molecules and natural language, introducing molecule captioning and text-guided de novo generation while addressing scarce paired chemistry-language data.

  • Motivation and contributions: Drug development can cost over a billion dollars and take over ten years, motivating computational tools for molecule design.The introduction connects this burden with interest in cheminformatics and higher-level natural-language control.
  • Motivation and contributions: Molecule-language translation includes generating descriptions for molecules and generating molecules from natural-language descriptions.These tasks extend vision-language paradigms into chemistry.
  • Challenges: Molecule descriptions vary widely, making expert annotation difficult, paired data scarce, and reference-based metrics such as BLEU inadequate.A single molecule may be described through names, properties, applications, functional groups, or synthetic routes.
  • MolT5: MolT5 pretrains on unlabeled natural-language text and molecule strings with denoising, then finetunes on limited gold-standard annotations.The framework is proposed to address chemistry-domain data scarcity.
  • MolT5: The work evaluates the new tasks with multiple metrics, including a cross-modal retrieval similarity metric based on Text2Mol.The contributions explicitly include both task design and evaluation methodology.

2 Tasks

The paper defines two bidirectional molecule-language tasks: describing a molecule in text and generating a molecule that matches a text description.

  • Molecule captioning: Molecule captioning generates a description of a given molecule, such as a molecule represented by a SMILES string.SMILES is treated as a molecular language, making the task a sequence-to-sequence translation problem.
  • Text-based de novo molecule generation: Text-based de novo molecule generation creates a molecule matching a natural-language description.It swaps the input and output roles of molecule captioning.

3 Evaluation Metrics

The evaluation combines language, molecular similarity, chemical-information, and validity measures to assess whether outputs match the intended molecule or description.

  • Cross-modal evaluation: A Text2Mol-based cross-modal metric uses cosine similarity between molecule and text embeddings to compare generated and ground-truth outputs.A base multilayer perceptron is first trained for molecule-text retrieval, then repurposed for evaluation.
  • Molecule captioning: Captioning is evaluated with BLEU, ROUGE, and METEOR despite their limitations with one reference caption and linguistically varied descriptions.The metrics remain useful for comparison with prior captioning work.
  • Molecule generation: Molecule generation uses MACCS FTS, RDK FTS, Morgan FTS, exact SMILES matches, Levenshtein distance, and SMILES BLEU scores.FTS denotes fingerprint Tanimoto similarity, averaged over the evaluation dataset.
  • Chemical-information metrics: Fréchet ChemNet Distance compares molecules using latent chemical and biological information rather than string similarity.ChemNet representations are derived from a network trained to predict drug-molecule activity.
  • Validity: Validity is the percentage of generated molecules that can be processed by RDKit, addressing syntactic invalidity in SMILES outputs.This measure is reported because SMILES-based generation can produce invalid strings.

4 MolT5 – Multimodal Text-Molecule Representation Model

MolT5 adapts a T5 encoder-decoder to jointly pretrain on natural-language and SMILES sequences, then finetunes it for either molecule captioning or generation.

  • Pretraining: MolT5 leverages large unlabeled corpora of Internet text and publicly available molecule strings for self-supervised pretraining.The motivation is to exploit abundant single-modal data when paired molecule-description data are scarce.
  • Pretraining: The model initializes from a public T5 checkpoint and uses a replace-corrupted-spans denoising objective.Each minibatch contains both natural-language sequences and SMILES sequences.
  • Pretraining: The shared model treats molecules and text as two monolingual corpora with distinct grammars and no explicit alignment during pretraining.This design is compared conceptually with multilingual-model pretraining.
  • Finetuning: After pretraining, the model is finetuned for molecule generation or molecule captioning.Generation maps descriptions to SMILES, while captioning maps SMILES to textual descriptions.

5 Experiments and Results

Experiments evaluate MolT5 and sequence-to-sequence baselines on molecule captioning and text-based molecule generation using ChEBI-20. Pretraining generally improves cross-modal outputs, with strong qualitative performance but remaining limitations on sparse or underspecified inputs.

  • Data and setup: ChEBI-20 provides 33,010 molecule-description pairs split into 80/10/10% train, validation, and test sets for finetuning and evaluation.
  • Data and setup: MolT5 pretrains T5 on combined natural-language and SMILES data, then finetunes for molecule captioning or generation.
  • Molecule captioning: Pretrained T5 and MolT5 models generate more realistic molecule captions than the RNN and Transformer baselines.The RNN extracts relevant properties but often produces ungrammatical text, whereas the Transformer is grammatical but frequently repeats inapplicable properties.
  • Molecule captioning: MolT5 improves over T5 in captioning across traditional NLG metrics and Text2Mol for every model size.The passage describes the increase as reasonable and refers statistical-significance analysis to the appendix.
  • Text-based de novo molecule generation: Pretrained models outperform the RNN and Transformer for molecule generation, while MolT5 adds especially large gains in validity over comparable T5 models.T5-large with 770M parameters outperforms MolT5-small with 60M parameters, but MolT5 still slightly improves some generation results.
  • Text-based de novo molecule generation: MolT5 qualitatively handles molecule classes, manipulation instructions, peptides, saccharides, enzymes, hydrates, and long SMILES strings, including a 474-character exact match.It still misses the charge of a rare ruthenium atom in one example.
  • Text-based de novo molecule generation: Some underspecified descriptions yield molecules that may be interesting to investigate rather than clearly matching a single target.
  • Probing the model: Probing shows MolT5 often retrieves finetuning-data matches, sometimes adds ions, and sometimes generates molecules absent from finetuning data.For corticosteroid prompts, one output is corticosterone and another is absent from PubChem but structurally related to known steroids.

6 Related Work

Prior work provides foundations in vision-language captioning, text-guided generation, molecular representation learning, and molecule generation, but these molecule-language tasks remain distinct challenges.

  • Text2Mol applies a CLIP-style contrastive loss to cross-modal retrieval between molecule and text pairs.
  • Image captioning is well studied, including with Transformer pretraining on massive text-image corpora and biomedical applications.
  • Text-guided image generation has used GAN-based methods and later achieved strong results with DALL·E.
  • Traditional molecular fingerprints do not allow representations to be learned from data, motivating machine-learning approaches using SMILES strings.
  • Generative models have been applied to de novo molecule discovery, including GPT-style decoders, T5 models, and MegaMolBART5.

7 Conclusions and Future Work

MolT5 frames molecule captioning and text-guided molecule generation as translation between natural language and molecules, using self-supervised pretraining and varied evaluation methods.

  • MolT5 pretrains models on vast amounts of unlabeled text and molecule strings using self-supervised learning.
  • The work introduces molecule captioning and text-guided molecule generation as new tasks for translating between language and molecules.
  • MolT5-based models obtain high scores on both molecule captioning and text-guided molecule generation.

8 Broader Impacts

The paper presents MolT5 as a tool for natural-language interaction with molecular AI, while recognizing risks from model bias and misuse.

  • MolT5 could help chemistry experts interact with molecular AI in natural language for discovering new drugs.
  • Text-based molecule generation could support generating molecules with specific functions, such as taste, rather than only specified properties.
  • Large-scale Internet training may introduce biases that affect which molecules are generated for prompts about diseases.
  • The technology could be misused to discover dangerous molecules, although the authors state that synthesis requires technical knowledge and laboratory resources.

9 Limitations

The paper inherits common large-language-model limitations and uses SMILES-based sequence modeling, with validity concerns affecting alternative representations and some compounds.

  • Internet-collected training data may contain unintended biases, as with other large language models.
  • SMILES strings are used despite recent validity-guaranteed alternatives performing poorly with pretrained T5 checkpoints.
  • Some ChEBI-20 compounds can cause validity problems in the default SELFIES implementation, which the authors leave for future investigation.
  • The authors state that generated molecules should not be used medically without careful evaluation through standard clinical testing.
  • The tasks can use any sequence-to-sequence model, and the study implements RNN-GRU, vanilla Transformer, and T5 baselines.
  • Baseline training uses ChEBI-20 with SMILES as molecular input or output and beam decoding with beam size 5 during inference.
  • RNN and vanilla Transformer baselines use character-split SMILES vocabularies and SciBERT’s 31,090-token text vocabulary.

B Reproducibility Checklist

The paper reports release plans, hardware used for training, inference timings, model conversion details, and a high-validity decoding strategy for molecule generation.

  • The programs, trained models, and resources will be made publicly available.
  • Training used NVIDIA Tesla V100 GPUs for RNN and Transformer baselines and TPUs for T5-related pretraining and finetuning.
  • 2.24 seconds/query was the average inference time for MolT5-Base molecule generation on a GPU-free MacBook Pro.
  • 9.86 seconds/query was the average inference time for the large MolT5-Base molecule captioning model on the same MacBook Pro.
  • The HuggingFace-converted models produce slightly different outputs from the original t5x models because of implementation differences.
  • High-validity decoding uses diverse beam search and RDKit to select the first valid beam, with iterative beam-size reduction when memory limits are exceeded.The beam width and beam group are 30, with a diversity penalty of 0.5.

E Ablations

Ablations examine pretraining data choices for molecule captioning and generation, with normalized results clarifying trade-offs between overall similarity and validity.

  • Pretraining ablations: Pretraining on both C4 and ZINC is clearly more beneficial for molecule captioning than pretraining on either dataset alone.
  • Pretraining ablations: After normalization by validity, C4+ZINC pretraining outperforms C4-only or ZINC-only pretraining on most molecule-generation metrics.The normalized results account for whether generated molecules are syntactically valid.
  • Pretraining ablations: ZINC-only pretraining substantially increases validity but decreases generated-molecule similarity to the ground truths.
  • Evaluation setting: The molecule-captioning baseline results are reported for different models, with Rouge scores represented as F1 values.
  • Evaluation setting: The de novo generation comparison includes high-validity MolT5 variants that use the decoding process described in Appendix D.
  • Evaluation setting: The reported captioning and generation ablations use MolT5-Small on the CheBI-20 test split.

F More Examples

The paper presents additional generated molecules and captions, evaluates generation diversity through cross-modal retrieval, and reports statistical comparisons with T5 baselines and retained NLP performance.

  • More generated molecules: Figures 6–8 show more examples of interesting molecules generated by different models.
  • More generated captions: Figure 9 shows more examples of interesting captions generated by different models.
  • Generation diversity: Text2Mol retrieval treats generated molecules or descriptions as a corpus and uses their originating descriptions or molecules as queries to assess distinguishability.
  • Generation diversity: MolT5 generations are sufficiently distinct to be retrievable, whereas captioning-transformer outputs are essentially indistinguishable to the retrieval model.
  • Statistical significance: MolT5-Large significantly improves molecule captioning over T5-Large across ROUGE-1, ROUGE-2, ROUGE-L, METEOR, and Text2Mol, with all reported p-values extremely small.
  • Statistical significance: MolT5-Large also significantly improves molecule generation over T5-Large according to the reported MACCS FTS, RDK FTS, and Morgan FTS comparisons.
  • NLP capabilities: MolT5-base achieved 95.6% accuracy on SST-2, compared with 95.2% for T5-base.
Loading 2204.11817v3…