PaperlayerView exact revision on arXiv

Source-linked AI summary

Developing ChemDFM as a large language foundation model for chemistry

Zihan Zhao, Da Ma, Lu Chen, Liangtai Sun, Zihao Li, Yi Xia, Bo Chen, Hongshen Xu, Zichen Zhu, Su Zhu, Shuai Fan, Guodong Shen, Kai Yu, Xin Chen

arXiv:2401.14818v6cs.CLcs.DL

TL;DR

General-domain LLMs often lack the chemical knowledge needed for broad chemical tasks, while task-specific models are typically built for designated scenarios. ChemDFM specializes a general-domain model with chemical literature, textbooks, and instructions; the authors report strong performance across chemical tasks, including outperforming GPT-4 on most evaluated tasks.

  • Problem

    Task-specific chemical models are typically trained for designated tasks and scenarios, while general-domain LLMs often lack the chemical knowledge needed to understand and reason over chemical language and molecular notations.

    Source: §1 · p. 2

  • Method

    ChemDFM specializes a general-domain LLM through chemical domain pre-training and instruction tuning, while including general-domain data in both phases.

    Source: §1 · p. 2, §2.2 · p. 5

  • Results

    ChemDFM surpasses typical open-source LLMs and outperforms GPT-4 on most evaluated chemical tasks, with more accurate and relevant answers in unseen scenarios.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    The authors conclude that ChemDFM shows potential as a dialogue-based assistant for chemical research, including paper reading and experimental design.

    Source: §5 · p. 10

Abstract

from arXiv · show

Artificial intelligence (AI) has played an increasingly important role in chemical research. However, most models currently used in chemistry are specialist models that require training and tuning for specific tasks. A more generic and efficient solution would be an AI model that could address many tasks and support free-form dialogue in the broad field of chemistry. In its utmost form, such a generalist AI chemist could be referred to as Chemical General Intelligence. Large language models (LLMs) have recently logged tremendous success in the general domain of natural language processing, showing emerging task generalization and free-form dialogue capabilities. However, domain knowledge of chemistry is largely missing when training general-domain LLMs. The lack of such knowledge greatly hinders the performance of generalist LLMs in the field of chemistry. To this end, we develop ChemDFM, a pioneering LLM for chemistry trained on 34B tokens from chemical literature and textbooks, and fine-tuned using 2.7M instructions. As a result, it can understand and reason with chemical knowledge in free-form dialogue. Quantitative evaluations show that ChemDFM significantly surpasses most representative open-source LLMs. It outperforms GPT-4 on a great portion of chemical tasks, despite the substantial size difference. We have open-sourced the inference codes, evaluation datasets, and model weights of ChemDFM on Huggingface (https://huggingface.co/OpenDFM/ChemDFM-v1.0-13B).

1 Introduction

General-domain LLMs lack the chemical knowledge needed for broad chemical tasks, while task-specific models are often limited to their designated scenarios. ChemDFM specializes a general-domain LLM with chemical knowledge and is reported to outperform open-source models and GPT-4 across many chemical tasks.

1 Introduction

  • Task-specific chemical models are typically built for one designated task and scenario, limiting their usefulness beyond that setting.A broader system would handle diverse chemical tasks and free-form collaboration.Source: §1 · p. 2
  • General-domain LLMs often lack chemical knowledge and struggle with specialized molecular notations such as SMILES, IUPAC names, and molecular formulas.The passage distinguishes chemical meanings such as CO and Co from their ordinary-language interpretations.Source: §1 · p. 2
  • ChemDFM specializes the LLaMa-13B general-domain model through chemical domain pre-training and instruction tuning, using literature, textbooks, and chemical instructions.General-domain data is also included to maintain natural-language comprehension and reasoning.Source: §1 · p. 2
  • Figure 1 depicts using chemical domain knowledge to train a general-domain LLM as the route to an LLM for chemistry.The scheme addresses the gap between general-domain language ability and chemical knowledge.Source: Figure 1 · p. 2, §1 · p. 2
  • ChemDFM surpasses typical open-source LLMs and outperforms GPT-4 on most evaluated chemical tasks; in unseen scenarios, its answers are more accurate and relevant to the questions.The unseen-scenario examples were based on recent chemical papers to avoid possible training-data leakage.Source: §1 · pp. 2–3

2 ChemDFM

ChemDFM is specialized from a general-domain LLM in two stages: domain pre-training on chemical sources and instruction tuning with chemical language and molecular notations. The training also includes general-domain data to retain natural-language capabilities.

2 ChemDFM

  • ChemDFM is trained from a general-domain LLM through domain pre-training followed by instruction tuning.The paper describes the stages as instilling chemical domain knowledge and familiarizing the model with chemical language and patterns.Source: §2 · p. 3

2.1 Domain Pre-training

  • Domain pre-training uses chemistry textbooks and published papers, alongside selected general-domain data.The corpus includes 1.4K chemistry books and, after preprocessing and deduplication, 49M textbook tokens and 34B research-article tokens.Source: §2.1 · p. 3

2.2 Instruction Tuning

  • Instruction tuning combines chemistry question-answering data with tasks built around molecular notations, descriptions, properties, and reactions.Question-answer pairs draw on established datasets and middle-school exams; molecular tasks include description/design, property prediction, and reaction completion.Source: §2.2 · p. 4
  • Molecular Notation Alignment (MNA) trains translation among SMILES, IUPAC names, and molecular formulas, including conversion between names and SMILES.The paper identifies this translation data as MNA and illustrates it with a representative instruction-tuning question.Source: Figure 3 · p. 4
  • General-domain data makes up roughly twice the chemical-domain share of instruction-tuning data, supporting natural-language comprehension during specialization.The reported chemical-to-general data ratio is roughly 1:2.Source: §2.2 · p. 5

3 Evaluations

ChemDFM is evaluated on chemical benchmarks and in free-form research-assistant scenarios. The reported results show strengths in molecule-related and reaction tasks, paper reading, and experimental-design dialogue.

3 Evaluations

  • The evaluation combines ChemLLMBench comparisons with qualitative tests of free-form collaboration, using 100 randomly sampled instances per task unless otherwise specified.The quantitative tasks cover molecule recognition, molecule design, molecular property prediction, and reaction understanding.Source: §3 · p. 5, §3.1 · pp. 5–7

3.1 Quantitative Evaluation

  • ChemDFM outperforms open-source LLMs in name prediction and molecule captioning; most LLMs, including GPT-4, struggle with name prediction.ChemDFM’s captioning performance is comparable across zero-shot and ten-shot settings, while GPT-4 drops sharply from ten-shot to zero-shot.Source: Table 2 · p. 5, §3.1 · pp. 5–6, Table 3 · p. 6
  • Across almost all text-based molecule-design metrics, ChemDFM outperforms generalist LLMs and task-specific specialist models.The authors also evaluated ChemDFM on the complete test set to align its comparison with the specialist models.Source: Table 4 · p. 6, §3.1 · p. 6
  • With scaffold-based splits designed to prevent molecules sharing a scaffold across train and test sets, ChemDFM outperforms other LLMs on all but one property-prediction task.The split is intended to avoid information leakage from molecular similarity and make evaluation more challenging.Source: Table 5 · p. 7, §3.1 · p. 7
  • ChemDFM significantly exceeds open-source LLMs on all reaction-related tasks and outperforms GPT-4 on three of four, but trails on reagent selection.The authors attribute GPT-4’s reagent-selection advantage to directly copying candidate SMILES, whereas ChemDFM often generates new answers.Source: Table 6 · p. 7, §3.1 · p. 8

3.2 LLM-Based Research Assistant

  • On questions based on 2023 chemistry papers, ChemDFM’s answers are reported as more accurate and relevant than other LLMs’, including GPT-4’s.The authors note uncertainty about whether GPT-4 had already encountered the papers in its training corpus; ChemDFM’s training papers were published before 2022.Source: Figure 4 · p. 8, §3.2 · pp. 8–9
  • In an experimental-design dialogue, ChemDFM corrected an initial proposal that would oxidize both carbonyl groups, then suggested a protecting group and feasible reaction conditions.The exchange illustrates comprehension, error correction, and elaboration across the dialogue rounds.Source: Figure 5 · p. 8, §3.2 · p. 9

4 Related Work

Prior chemical LLM work either specializes models for particular tasks or applies generalist LLMs directly to chemical problems. The paper argues for combining general-domain and chemical knowledge.

4 Related Work

  • Multi-task training strategies can neglect LLMs’ natural-language understanding and reasoning, leaving models confined to trained tasks and less able to collaborate freely.The cited examples include Christofidellis et al. [2023], Fang et al. [2023], Cao et al. [2023a], Zheng et al. [2023], Kim et al. [2024], and Yu et al. [2024].Source: §4 · p. 9
  • Directly applying generalist LLMs to chemical tasks uses their language abilities, but most lack inherent understanding of chemical language and knowledge.The cited examples include Hatakeyama-Sato et al. [2023], Cao et al. [2023b], Boiko et al. [2023], Yoshikawa et al. [2023], M. Bran et al. [2024], and Ruan et al. [2024].Source: §4 · p. 9
  • The paper’s approach equips a general-domain LLM with chemical knowledge, aiming to retain general knowledge while adding chemistry expertise.Related domain-specific LLMs include work in biology and medicine, education, materials science, and geography, while domain-specific languages such as SMILES are often overlooked.Source: §4 · pp. 9–10

5 Conclusion

The conclusion presents ChemDFM as a chemistry-specialized LLM developed from a generalist model through pre-training and instruction tuning. It reports strong performance across chemical tasks and potential for dialogue-based research assistance.

5 Conclusion

  • ChemDFM specializes a generalist LLM through chemistry-focused pre-training and instruction tuning.Source: §5 · p. 10
  • Quantitative evaluations indicate strong molecular-notation comprehension and chemical reasoning, with reported performance across molecular design and reaction analysis.Source: §5 · p. 10
  • In paper reading and experimental design, ChemDFM shows potential to assist researchers through free-form dialogue using chemical and natural languages.Source: §5 · p. 10

A.1 Domain Pre-training

This block describes ChemDFM’s pre-training and instruction-tuning settings, then outlines molecule name-translation and captioning evaluations and their prompts.

A.1 Domain Pre-training

  • Pre-training uses Megatron-DeepSpeed with Zero-2 on LLaMa-13B, with 4M tokens per batch, a 6K maximum sequence length, and a 5e-5 peak learning rate.Training uses AdamW with (β1, β2) = (0.9, 0.95) and a cosine learning-rate scheduler.Source: §A.1 · p. 15

A.2 Instruction Tuning

  • Instruction tuning uses full-parameter updates with Deepspeed-Chat and Zero-3, and back-propagates loss only through return tokens.The global batch size is 256 and the learning rate is 1e-5; the loss is defined over the instruction-tuning dataset and return sequence.Source: §A.2 · pp. 15–16

B.1.1 Task Introduction

  • Name prediction evaluates translation among SMILES, IUPAC names, and molecular formulas, normalizing name and SMILES predictions but requiring exact formula matches.The four directions are S2I, I2S, S2MF, and I2MF.Source: §B.1.1 · p. 16
  • Molecule captioning tests whether models can recognize a SMILES molecule and describe its basic chemical nature, using the ChEBI-20 test set and BLEU, ROUGE, and METEOR.Source: §B.1.1 · p. 16

B.1.2 Prompt Format

  • The name-prediction prompt is simpler than Guo et al. [2023]’s, while molecule captioning uses the prompt from Guo et al. [2023].Figure 6 shows a name-prediction prompt example.Source: §B.1.2 · p. 16, Figure 6 · p. 16

B.2.1 Task Introduction

The molecule-design evaluation tests SMILES generation from molecular descriptions using both text-based and chemical-similarity measures, with prompts following Guo et al. [2023].

B.2.1 Task Introduction

  • Models predict a molecule’s SMILES from its description, using ChEBI-20’s test set.Source: §B.2.1 · p. 16
  • Text-based evaluation includes exact match, BLEU, and Levenshtein distance, while chemical evaluation includes SMILES validity and fingerprint Tanimoto similarity.The fingerprint measures use MACCS, RDK, and Morgan fingerprints.Source: §B.2.1 · pp. 16–17

B.2.2 Prompt Format

  • The molecule-design task uses the prompt introduced in Guo et al. [2023].Source: §B.2.2 · p. 17

B.2.3 Additional Results

On the full ChEBI-20 test set, ChemDFM’s molecule-design results exceed specialist models on several major metrics; the block also introduces molecular-property tasks and their prompts.

B.2.3 Additional Results

  • ChemDFM surpasses specialist models on exact match and all three FTS-based similarity scores, while matching or exceeding them on other major metrics.The comparison uses the full ChEBI-20 test set; the authors say these results indicate more reliable predictions from descriptions.Source: §B.2.3 · p. 17

B.3.1 Task Introduction

  • Molecular-property evaluation covers BACE, BBBP, HIV, ClinTox, and Tox21, with binary-classification tasks that vary in balance and number of endpoints.ClinTox has two unbalanced tasks and Tox21 has twenty-one; BACE and BBBP are balanced, while HIV is unbalanced.Source: §B.3.1 · p. 17

B.3.2 Prompt Format

  • Molecular-property evaluation uses the prompts introduced in Guo et al. [2023].Source: §B.3.2 · p. 17

B.3.3 Additional Results

The additional results discuss few-shot molecular-property evaluation, while the following reaction-understanding section defines four task types and its prompt format.

B.3.3 Additional Results

  • Few-shot molecular-property results are not always better than zero-shot results, possibly because scaffold-vertical splits make training exemplars less helpful for test samples.Table 8 reports AUC-ROC scores across models and settings for molecular-property tasks.Source: §B.3.3 · p. 17, Table 8 · p. 18

B.4.1 Task Introduction

  • Reaction-understanding evaluation covers yield prediction, product prediction, reagent selection, and retrosynthesis, using accuracy except for ligand selection’s top-50% accuracy.The tasks draw on HTE, USPTO-MIT, Perera et al. [2018], and USPTO-50K datasets, respectively.Source: §B.4.1 · p. 18

B.4.2 Prompt Format

  • Reaction-prediction and retrosynthesis prompts reformat Guo et al. [2023]’s prompts using reaction SMILES notation; Figure 7 gives examples.Source: §B.4.2 · p. 18, Figure 7 · p. 18

B.4.3 Additional Results

This appendix section points readers to additional benchmark results for yield prediction, reaction prediction, reagent selection, and retrosynthesis; the supplied passages identify the tables but do not provide their results.

B.4.3 Additional Results

  • The section also directs readers to Table 12 for reagent selection and Table 11 for retrosynthesis results.The listed results are described as complete in the section text; the supplied captions do not establish which models perform best.Source: §B.4.3 · p. 18
  • Additional yield-prediction results are reported in Table 9, with accuracy scores and dataset abbreviations identified in its caption.B-H and Suzuki refer to the Buchwald-Hartwig and Suzuki-Miyaura datasets, respectively; the caption also marks 8-shot results with an asterisk.Source: Table 9 · p. 19
  • Additional reaction-prediction results are reported in Table 10.The table caption notes that dagger-marked results come from Guo et al. [2023].Source: Table 10 · p. 19

C.1 Paper Reading

The paper-reading examples test ChemDFM on both established chemical knowledge and newer molecules and reactions. Across the examples, its answers often capture specific chemical points, while some errors and uneven performance remain.

Widely known knowledge

  • On established-knowledge questions, ChemDFM captures the key regioselectivity point for Catellani reactions and answers the Diels–Alder regioselectivity question that GPT-4 and LLaMa-2 miss.For the Dess-Martin periodinane question, ChemDFM, Galactica, and GPT-4 capture alcohol oxidation to aldehydes and ketones; ChemDFM and GPT-4 add properties of the reagent.Source: §C.1 · p. 19

New molecules and reactions

  • The newer-molecule and reaction examples use papers published after January 2022 to minimize data-leakage risk and test handling of unforeseen situations.The examples are shown in Figures 9 and 10.Source: §C.1 · p. 19

Paper-reading examples

  • Figures 8–10 mark correct, relevant answers in green, correct but irrelevant content in yellow, and wrong information in red.Figure 8 covers widely known knowledge; Figures 9 and 10 cover new molecules and reactions.Source: Figure 8 · p. 20, Figure 9 · p. 20, Figure 10 · p. 22

New molecules and reactions

  • ChemDFM identifies the key point and gives an almost correct mechanism for the novel reaction in Q-S4, and identifies the new molecule’s reaction site in Q-S5.In Q-S5 it gives the wrong IUPAC name; in Q-S6 it gives the most precise mechanism answer, while its mistakes again include wrong IUPAC names.Source: §C.1 · p. 21

Molecule-focused questions

  • The section also introduces questions focused more on molecules than reactions, but the supplied passage gives no outcomes for them.Source: §C.1 · p. 21

New molecules and reactions

  • For the novel catalyst-modification question, only ChemDFM proposes introducing electron-withdrawing groups on aromatic rings, a change described as potentially increasing acidity while preserving catalytic ability.Other models either offer no specific solution or propose solutions that would damage catalytic ability.Source: §C.1 · p. 21
  • On the chiral-environment question, ChemDFM gives two correct methods, including one targeting axial chirality, which is challenging to identify from SMILES alone.GPT-4’s candidate methods are not all correct; Galactica gives almost no detail, while LLaMa-2 gives one correct method among wrong ones.Source: §C.1 · pp. 21–22
  • For the palladium coordination question, GPT-4 and ChemDFM each identify one of the three coordinate sites, while Galactica and LLaMa-2 identify none.Source: §C.1 · p. 22

C.2 Dialogue-Based Human-AI Collaboration

The dialogue examples show ChemDFM responding to researcher questions about reactions and recovering from some mistakes after human correction. They also illustrate using prior dialogue context to answer a follow-up.

C.2 Dialogue-Based Human-AI Collaboration

  • In the LiCl dialogue, ChemDFM first misjudges the reaction type, then corrects its answer after the researcher points out the error and gives reaction details.It subsequently identifies the necessary post-processing procedure for obtaining the final product.Source: §C.2 · p. 22
  • In the second dialogue, ChemDFM corrects its mistaken bond-position and reaction-type assessment after human feedback, then answers a follow-up by inferring the system’s molecule composition from dialogue history.The researcher does not restate the previous reaction or clarify the current composition.Source: §C.2 · pp. 22, 24
  • The authors attribute the dialogues’ error correction and context use to ChemDFM’s comprehension of both natural and chemical language, which they say enables meaningful human–AI collaboration.Source: §C.2 · p. 24
Loading 2401.14818v6…