PaperlayerView exact revision on arXiv

Source-linked AI summary

Unifying Molecular and Textual Representations via Multi-task Language Modelling

Dimitrios Christofidellis, Giorgio Giannone, Jannis Born, Ole Winther, Teodoro Laino, Matteo Manica

arXiv:2301.12586v2cs.LGcs.CL

TL;DR

Chemistry language models often require task-specific models and fine-tuning, while domain shift makes natural-language and chemical representations difficult to combine. Text+Chem T5 applies multi-task learning across these domains, and the paper reports improved cross-domain translation through information sharing alongside competitive benchmark performance.

  • Problem

    Chemistry models often require task-specific development, and domain shift makes merging natural-language and chemical representations challenging.

    Source: §1 · p. 2

  • Method

    Text+Chem T5 is a multi-task transformer that handles chemical, natural-language, and cross-domain tasks without task-specific fine-tuning or separate heads.

    Source: §1 · p. 2

  • Results

    Sharing information across tasks and domains improves cross-domain translation, while the model is competitive with state-of-the-art methods specialized for single tasks.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The approach may provide a more efficient way to process, analyze, and generate chemical and textual data in life-sciences discovery.

    Source: §7 · p. 9

    The models are designed for research purposes, and generated molecules require standard clinical testing before medical or other use.

    Source: §6 · p. 9

Abstract

from arXiv · show

The recent advances in neural language models have also been successfully applied to the field of chemistry, offering generative solutions for classical problems in molecular design and synthesis planning. These new methods have the potential to fuel a new era of data-driven automation in scientific discovery. However, specialized models are still typically required for each task, leading to the need for problem-specific fine-tuning and neglecting task interrelations. The main obstacle in this field is the lack of a unified representation between natural language and chemical representations, complicating and limiting human-machine interaction. Here, we propose the first multi-domain, multi-task language model that can solve a wide range of tasks in both the chemical and natural language domains. Our model can handle chemical and natural language concurrently, without requiring expensive pre-training on single domains or task-specific models. Interestingly, sharing weights across domains remarkably improves our model when benchmarked against state-of-the-art baselines on single-domain and cross-domain tasks. In particular, sharing information across domains and tasks gives rise to large improvements in cross-domain tasks, the magnitude of which increase with scale, as measured by more than a dozen of relevant metrics. Our work suggests that such models can robustly and efficiently accelerate discovery in physical sciences by superseding problem-specific fine-tuning and enhancing human-model interactions.

1 Introduction

The introduction identifies the cost and domain shift involved in building task-specific chemistry models, especially when translating between chemistry and natural language. It proposes Text+Chem T5, which shares information across tasks and domains and performs strongly on cross-domain tasks.

1 Introduction

  • Existing chemistry models often require separate task-specific development, while domain shift makes combining chemistry and natural language challenging.The repeated pre-training and fine-tuning process is described as computationally expensive and sample-inefficient.Source: §1 · p. 2
  • Text+Chem T5 is consistently superior on cross-domain tasks and competitive with state-of-the-art methods specialized for single tasks.The cross-domain comparisons span NLP-based evaluation metrics.Source: §1 · p. 2
  • Text+Chem T5 is a multi-task transformer for translating between natural-language and chemical representations.The paper describes it as a cross-domain, multi-task chemical language model.Source: §1 · p. 2
  • The model handles chemical, natural-language, and cross-domain tasks without task-specific fine-tuning or separate heads.The introduction also reports that it can execute complex molecular discovery workflows in one model.Source: §1 · p. 2
  • Its training strategy adapts single-domain models for cross-domain tasks, avoiding costly mono-domain pre-training and task-specific fine-tuning.It improves cross-domain translation by sharing information across tasks and domains.Source: §1 · p. 2
  • The paper validates the approach on single- and cross-domain benchmarks and analyzes architecture and encoder choices.The reported comparison is against state-of-the-art methods specialized for single tasks.Source: §1 · p. 2

2 Background

Prior approaches address chemistry tasks, language tasks, or text-chemistry translation, but struggle to handle multiple tasks across domains without costly pre-training and fine-tuning. The paper positions Text+Chem T5 as a shared multi-task approach built on T5 and intended to bridge that gap.

2 Background

  • The paper uses T5 as its backbone and organizes its task scope across chemistry, text, and cross-domain tasks.The task groups include mol2mol, text2text, mol2text, and text2mol.Source: §2 · pp. 2–3
  • Chemistry systems such as Molecular Transformers and RXN require separate models for separate tasks, while T5Chem remains chemical-domain limited and uses task-specific heads.T5Chem is described as supporting reaction prediction, regression, and classification.Source: §2 · p. 3
  • MolT5 links natural language and chemistry for molecule generation and captioning, but requires costly mono-modality pre-training and per-task fine-tuning.Those requirements limit its ability to share information between tasks.Source: §2 · p. 3
  • Existing approaches struggle to handle multiple tasks across domains without costly, sample-inefficient pre-training and fine-tuning.The proposed strategy uses pre-trained transformers for each modality and a small learnable output decoder to merge modalities in later model stages.Source: §2 · p. 3

3 Method

Text+Chem T5 uses a jointly trained, shared encoder-decoder for natural-language and chemistry tasks. The method spans mono-domain and cross-domain tasks, with cross-attention described as an option for merging separate domain encoders.

3 Method

  • The mono-domain mol2mol task covers forward reaction prediction and one-step retrosynthesis, mapping precursors to products or products to precursors.Inputs may optionally include reagents or enzymes.Source: §3 · p. 4
  • Cross-domain tasks include generating text descriptions from molecular SMILES and generating a chemical representation from a textual molecular description.The latter is described as text-conditional de novo generation, with SMILES as the output representation.Source: §3 · p. 4
  • The task set also includes text2text and paragraph-to-action generation, which turns a natural-language reaction description into a stepwise execution protocol.The method describes text2text as a mono-domain natural-language task.Source: §3 · p. 4
  • The proposed Text+Chem T5 shares a jointly trained encoder across text and chemistry and uses a T5 decoder without separate task or domain heads.The model is named Text+Chem T5; its joint encoder is trained on multi-domain, multi-task data.Source: §3 · p. 4, Figure 4 · p. 5
  • Cross-attention can merge separate domain encoders by using one domain as queries and the other as keys and values before passing the result to the T5 decoder.The text-base version feeds its merged representation directly to the decoder without architectural adaptation.Source: §3 · pp. 4–5

4 Experiments

The experiments evaluate Text+Chem T5 across chemical, textual, and cross-domain tasks using task-specific metrics and comparisons with specialized and multitask baselines. Results favor Text+Chem T5 on both cross-domain tasks, with shared encoders and fine-tuning identified as the most influential architectural choices.

4 Experiments

  • The evaluation covered forward and backward reaction prediction, text-conditional molecule generation, molecule captioning, and paragraph-to-action; T5-small and T5-base were pretrained bases.Baselines included Transformer, task-finetuned T5, RXN-family models, and MolT5, with small and base versions for T5, MolT5, and Text+Chem T5.Source: §4 · pp. 5–6
  • Training used a balanced 11.5M-sample multi-task dataset, with 2.3M samples per task; an augmented version contained 33.5M samples, including 6.7M reaction pairs.Smaller task datasets were expanded by repeating existing samples.Source: §4 · pp. 5–6
  • Evaluation combined NLP-based and task-specific metrics, qualitative assessment, and comparisons with broader multitask models.Mol2text metrics included BLEU-2, BLEU-4, ROUGE-1, ROUGE-2, ROUGE-L, and METEOR; text2mol included BLEU, accuracy, Levenshtein distance, and additional measures.Source: §4 · p. 6

4.1 Results

  • 0.853 BLEU on text2mol and 0.625 on mol2text; Text+Chem T5 achieved the highest scores on both cross-domain tasks.It also performed best across the reported mol2text and text2mol metrics.Source: §4.1 · pp. 6–8, Table 3 · p. 7, Table 4 · p. 8
  • Shared encoders and fine-tuning had the greatest effect in the ablation, while the aggregation method played a smaller role.The best-performing Text+Chem T5 variant used shared encoders and fine-tuning.Source: §4.1 · p. 8, Table 5 · p. 8
  • Text+Chem T5 improved with model size, with faster gains than similarly sized T5-finetuned and MolT5 baselines; dataset augmentation also improved performance, especially for SMILES outputs.The augmented dataset added reactant-product pairs, and its results are marked with the prefix ‘augm‘ in the tables.Source: §4 · p. 6, §4.1 · p. 8
  • In a Monuron example, three Text+Chem T5 calls converted a textual description into SMILES, a precursor list, and a stepwise synthesis protocol; ChatGPT and Galactica did not complete all three steps correctly.The patent description used for the protocol was retrieved manually.Source: §4.1 · pp. 8–9

5 Related Work

Related work traces transformer advances in natural language and generative multitask learning, alongside growing chemistry applications. It identifies a remaining need for models that handle multiple tasks across domains without expensive pretraining or fine-tuning.

Transformers for Natural Language

  • Transformer language models span masked-language reconstruction, autoregressive next-token generation, and later developments in multitasking, few-shot use, and reasoning.The passage cites BERT and GPT-family approaches as prominent examples.Source: §5 · p. 9

Generative Transfer and Multi-task Learning

  • T5 frames generative transfer as multitask training across varied tasks using masking, while related work also covers multimodal generation and cross-domain translation.The passage describes T5's generalization and adaptation capabilities.Source: §5 · p. 9

Language Models in Chemistry

  • Chemistry language models address reaction prediction, retrosynthesis, property prediction, and multitask generation, but cross-domain multitasking without expensive pretraining or fine-tuning remains needed.The cited multitask chemistry work uses pretrained T5 with multiple heads for regression, classification, and generation.Source: §5 · p. 9

6 Limitations

The study shares limitations of earlier work and notes possible training-data bias. Its SMILES representation can produce invalid sequences, and generated molecules require standard clinical testing before use.

6 Limitations

  • The work remains subject to limitations of previous research on large language models in chemistry and cross-domain generation.It also notes unintended biases encoded in training data.Source: §6 · p. 9
  • SMILES can generate invalid sequences; SELFIES is mentioned as an alternative with validity guarantees.The passage presents SELFIES as a possible way to address this representation issue.Source: §6 · p. 9
  • The models were designed for research purposes, and generated molecules should undergo standard clinical testing before medical or other use.Source: §6 · p. 9

7 Conclusion

The conclusion presents Text+Chem T5 as a multi-task model spanning natural and chemical domains. It can address multiple chemistry tasks without task-specific heads or test-time adaptation, and may offer a more efficient way to process, analyze, and generate data in the field.

7 Conclusion

  • Text+Chem T5 translates between natural and chemical languages across reaction prediction, retrosynthesis, text-conditional de novo generation, and molecular captioning.Source: §7 · p. 9
  • The model handles multiple tasks without additional task-specific heads or adaptation at test time.Source: §7 · p. 9
  • The authors suggest the model could potentially accelerate discovery by providing a more efficient way to process, analyze, and generate chemical and textual data.Source: §7 · p. 9

A Additional Experiments

Additional experiments report strong cross-domain performance for Text+Chem T5 across molecule captioning, text-conditional molecule generation, and paragraph-to-actions. They also report that zero-shot T5 cannot perform cross-domain translation.

A Additional Experiments

  • T5 zero-shot, without fine-tuning, is unable to perform cross-domain translation in the Table 6 comparison.Source: Table 6 · p. 12
  • Text+Chem T5 outperforms the other baselines on both BLEU score and accuracy for paragraph-to-actions.Source: Table 7 · p. 12
  • Text+Chem T5 performs best across all six reported metrics for molecule-to-caption and improves with model size.The metrics are BLEU-2, BLEU-4, Rouge-1, Rouge-2, Rouge-L, and Meteor.Source: Figure 5 · p. 13
  • Text+Chem T5 performs best across all five reported metrics for text-conditional molecule generation and improves with model size.The metrics are Accuracy, Morgan FTS, RDK FTS, BLEU, and MACCS FTS.Source: Figure 6 · p. 13

B Comparison with recent Language Models and Workflow example

The appendix compares model outputs for molecular generation and captioning, and presents a Monuron discovery workflow. In that workflow, Text+Chem T5 produced the exact target molecule and reaction, while the competing systems had noted errors or uncertainty.

B Comparison with recent Language Models and Workflow example

  • Tables 8 and 9 compare models on conditional molecule generation and molecular captioning, using the stated prompt structures for Text+Chem T5 and ChatGPT and the authors' prompt structure for Galactica.Source: Table 8 · p. 14, Table 9 · p. 15
  • For a complex molecule described as a substituted 3,4-dimethoxyphenylethylamine, Galactica returned that simpler amine name, while ChatGPT returned a SMILES string.Source: B Comparison with recent Language… · p. 14
  • For the N-acetyltyrosine example, ChatGPT captioned the SMILES as acetaminophen, whereas Text+Chem T5 described an N-acetyltyrosine derivative.Source: B Comparison with recent Language… · p. 15
  • In the Monuron workflow, Text+Chem T5 alone produced the exact desired molecule and a reaction identical to the target reaction.RXN assigned the generated reaction confidence 1.0; ChatGPT's chemically valid but unlikely route received 0.6, while Galactica failed to give a sensible retrosynthesis output. Figure 8 shows a similar reaction differing in its substituted amine.Source: Figure 7 · p. 16, Figure 8 · p. 17

C Prompt templates

Table 10 documents the prompt templates used for five tasks in multi-task, multi-domain training.

C Prompt templates

  • Table 10 documents prompt templates used for five tasks in multi-task, multi-domain training.Source: Table 10 · p. 18

D Experimental Details

The experimental details include model sizes and relevant hyperparameters for Text+Chem T5.

D Experimental Details

Loading 2301.12586v2…