PaperlayerView exact revision on arXiv

Source-linked AI summary

LlaSMol: Advancing Large Language Models for Chemistry with a Large-Scale, Comprehensive, High-Quality Instruction Tuning Dataset

Botao Yu, Frazier N. Baker, Ziqi Chen, Xia Ning, Huan Sun

arXiv:2402.09391v4cs.AIcs.CEcs.CL

TL;DR

Existing LLMs show limited capabilities on chemistry tasks, while task-specific models do not readily adapt across tasks. This paper introduces SMolInstruct and fine-tunes open-source LLMs with it. LlaSMol outperforms GPT-4 across tasks and Claude 3 Opus on most tasks, while approaching task-specific state-of-the-art performance with a small fraction of parameters fine-tuned.

  • Problem

    General-purpose LLMs have limited chemistry-task capabilities, while task-specific models neglect shared chemistry knowledge and are difficult to adapt across tasks.

    Source: §1 · p. 1

  • Method

    The authors curate SMolInstruct, a 14-task, over-3M-sample instruction dataset, and use it to fine-tune open-source LLMs as LlaSMol.

    Source: §1 · p. 2

  • Results

    LlaSMol outperforms GPT-4 on all tasks and Claude 3 Opus on most tasks, and approaches task-specific SoTA with 0.58% of parameters fine-tuned.

    Source: §4.3 · pp. 7–8

  • Takeaways & Limitations

    Experiments identify SMolInstruct as crucial to performance gains, while LlaSMol remains short of task-specific SoTA overall.

    Source: §1 · p. 2, §5 · p. 9

    The study does not investigate generalization beyond the trained tasks, and its MC and MG evaluations cannot accurately assess chemical correctness.

    Source: §5 · p. 9

Abstract

from arXiv · show

Chemistry plays a crucial role in many domains, such as drug discovery and material science. While large language models (LLMs) such as GPT-4 exhibit remarkable capabilities on natural language processing tasks, existing research indicates that their performance on chemistry tasks is discouragingly low. In this paper, however, we demonstrate that our developed LLMs can achieve very strong results on a comprehensive set of chemistry tasks, outperforming the most advanced GPT-4 and Claude 3 Opus by a substantial margin. To accomplish this, we propose SMolInstruct, a large-scale, comprehensive, and high-quality dataset for instruction tuning. It contains 14 selected chemistry tasks and over three million samples, laying a solid foundation for training and evaluating LLMs for chemistry. Using SMolInstruct, we fine-tune a set of open-source LLMs, among which, we find that Mistral serves as the best base model for chemistry tasks. Our analysis further demonstrates the critical role of the proposed dataset in driving the performance improvements.

1 Introduction

The introduction frames chemistry LLMs against the limited chemistry performance of existing general-purpose models and task-specific models’ inability to share knowledge across tasks. It presents SMolInstruct and LlaSMol, reporting strong results and several training-related findings.

1 Introduction

  • Prior chemistry models are generally task-specific, limiting their ability to adapt across tasks and leverage shared chemistry knowledge.Existing LLMs also show limited chemistry capabilities, including poor performance when precise SMILES understanding is required.Source: §1 · p. 1
  • SMolInstruct contains 14 chemistry tasks and over 3M samples, and supports LlaSMol models fine-tuned from four open-source LLMs with LoRA.The base models are Galactica, Llama 2, Code Llama, and Mistral.Source: §1 · p. 2
  • 0.58% of parameters are fine-tuned as LlaSMol approaches task-specific SoTA models, though it does not yet surpass them overall.LlaSMolMistral exceeds SoTA models on PP-Clintox and PP-SIDER, but not on other tasks.Source: §1 · p. 2
  • Mistral is the strongest of four LlaSMol base models; canonicalized SMILES and SMolInstruct also improve performance over SELFIES and a previous instruction dataset.The introduction also reports that SMILES achieves sufficient validity of generated molecules.Source: §1 · p. 2

2 Related Work

Prior chemistry approaches include task-specific deep learning models and evaluations of general-purpose LLMs, while instruction-tuning datasets vary in scale and task coverage. The section positions SMolInstruct as larger and more diverse than earlier datasets.

2 Related Work

  • Task-specific models span sequence-to-sequence, graph-neural-network, and molecular 3D approaches, but cannot harness knowledge shared across chemistry tasks.Examples include Molecular Transformer, Chemformer, MolT5, GNNs, and Uni-Mol.Source: §2 · p. 3
  • Benchmark studies report that off-the-shelf LLMs perform poorly on chemistry tasks and can produce chemically implausible outputs.The cited work motivates fine-tuning efforts for chemistry tasks.Source: §2 · p. 3
  • SMolInstruct is larger and covers more diverse chemistry tasks than prior instruction datasets, with the stated aim of supporting cross-task chemistry knowledge.Earlier datasets include Mol-Instructions, Drugchat, and MolOpt-Instructions.Source: §2 · p. 3

3 SMolInstruct

SMolInstruct is a small-molecule instruction-tuning dataset with 14 chemistry tasks, constructed from established data sources through curation, leakage-aware splitting, and varied instruction templates. Its stated merits are scale, task breadth, and quality controls.

3.1 Overview of SMolInstruct

  • The 14 tasks cover name conversion, property prediction, molecule captioning and generation, and forward synthesis and retrosynthesis.The task set spans molecular representations, drug-relevant properties, molecule-text relations, and chemical reactions.Source: §3.1 · pp. 3–4
  • The dataset has 3.3M query-response samples and uses SMILES by default, while also providing SELFIES representations.The four name-conversion tasks are exceptions to the default molecular representation.Source: §3.1 · p. 4

3.2 SMolInstruct Construction

  • Data sources include PubChem, ChEBI-20 and Mol-Instructions, MoleculeNet, and USPTO-full for name conversion, molecule-text, property, and reaction tasks, respectively.USPTO-full contributes reaction data extracted from U.S. patents.Source: §3.2 · p. 4
  • Curation detects invalid SMILES, corrects or filters inaccurate records and descriptions, removes ambiguous SIDER disorders, and removes duplicates.The quality-control methods include RDKit parsing and manual checks of reaction atom mappings.Source: §3.2 · pp. 4–5
  • Splitting groups matched samples across related tasks and identical inputs together to reduce leakage, and uses scaffold splitting for property-prediction tasks.The split also keeps shared-task Mol-Instructions training examples out of the SMolInstruct test set for direct comparison.Source: §3.2 · p. 5
  • Manually crafted templates rephrased with GPT-4 create varied queries and responses, while canonicalized SMILES and type-specific tags standardize sequences and support answer extraction.Tags mark content such as SMILES, molecular formulas, and numbers.Source: §3.2 · p. 5

3.3 Merits of SMolInstruct

  • The dataset’s stated merits are 3.3M samples across 1.6M distinct molecules and 14 tasks spanning four types of chemistry tasks.The authors describe the tasks as selected to support a chemistry foundation model and real-world applications.Source: §3.3 · p. 5
  • Quality controls, careful data splitting, and SMILES canonicalization are presented as measures that make SMolInstruct a high-quality resource.The construction section describes these curation, split, and formatting choices.Source: §3.3 · p. 5

4 Experiments

The experiments evaluate LlaSMol models fine-tuned on SMolInstruct against untuned LLMs and task-specific chemistry models. They report broad advantages over other LLMs, differences among base models, and ablation findings on data representation and training choices.

4.1 Our LlaSMol Models

  • The authors fine-tune Galactica, Llama 2, Code Llama, and Mistral on SMolInstruct to create the corresponding LlaSMol models.All four use LoRA applied to self-attention and feedforward weight matrices.Source: §4.1 · p. 6

4.2 Experimental Setup

  • Comparisons include untuned and chemistry-tuned LLMs, task-specific models, and common task-appropriate metrics; GPT-4 and Claude 3 Opus are evaluated on at most 500 test samples per task.Metrics include exact match, fingerprint Tanimoto similarity, METEOR, RMSE, accuracy, and validity.Source: §4.2 · pp. 6–7

4.3 Main Results

  • Among the four LlaSMol models, Mistral consistently outperforms Llama 2, while Galactica and Code Llama also outperform Llama 2 in most or many tasks.The authors suggest chemistry-related pretraining may benefit Galactica and programming-language knowledge may synergize with molecular representations in Code Llama.Source: §4.3 · pp. 7–8
  • LlaSMol outperforms GPT-4 on all tasks and Claude 3 Opus on most, and also surpasses the compared chemistry LLMs ChemLLM and Molinst.The reported results cover name conversion and property prediction in Table 1 and molecule captioning, molecule generation, forward synthesis, and retrosynthesis in Table 2.Source: Table 1 · p. 7, §4.3 · p. 7, Table 2 · p. 8
  • LlaSMol does not outperform task-specific state-of-the-art models overall, although LlaSMolMistral surpasses them on PP-Clintox and PP-SIDER.LlaSMolMistral fine-tunes approximately 41.9M parameters, or 0.58% of its parameters; increasing trainable parameters substantially boosted performance in further experiments.Source: §4.3 · p. 8

4.4 Ablation Study

  • Without SMILES canonicalization, performance drops on most tasks, especially forward synthesis and retrosynthesis; SELFIES slightly improves validity but performs worse on most tasks.The authors attribute SELFIES’ poorer performance possibly to its typically greater length, and report that the Mol-Instructions-trained model performs much worse on shared tasks.Source: §4.4 · pp. 8–9, Table 4 · p. 9
  • Multi-task training outperforms single-task training, while adding LoRA modules and trainable parameters substantially boosts performance.The analyses also report that individual tasks generally do not heavily rely on the presence of other tasks.Source: §4.4 · p. 9

5 Conclusion

The paper presents SMolInstruct and LlaSMol as a chemistry-focused instruction-tuning resource and model series, while acknowledging limits in evaluation, generalization testing, and performance against task-specific models.

5 Conclusion

  • SMolInstruct comprises 14 chemistry tasks and over 3M curated samples, and the authors use it to develop LlaSMol models for chemistry tasks.The conclusion describes the dataset as large-scale, comprehensive, and high-quality.Source: §5 · p. 9
  • The authors report that LlaSMol outperforms existing LLMs and that SMolInstruct plays a crucial role in boosting performance.The paper also says its analytical experiments provide insights for developing chemistry LLMs.Source: §5 · p. 9
  • The evaluations cannot reliably assess chemical correctness for MC and MG, do not test generalization beyond trained tasks, and the models do not yet outperform SoTA task-specific models.The authors cite ambiguity and limited data for MC/MG assessment, and suggest small trainable-parameter ratios or suboptimal training procedures may contribute to the task-specific-model gap.Source: §5 · p. 9

Ethics Statement

This block states an ethics caveat, establishes the paper’s small-molecule and reaction background, and introduces SMolInstruct’s statistics and construction details.

Ethics Statement

  • The authors cannot guarantee SMolInstruct is free of inaccurate, incorrect, or harmful content, or prevent the models from generating such content.They advise users to engage with the dataset and models at their own discretion and uphold high ethical standards.Source: Ethics Statement · p. 10

A Preliminaries

  • The paper focuses on small molecules, typically no more than 100 atoms and under 1,500 Daltons, and describes reactions as converting reactants into products through bond changes.It also introduces molecule representations including SMILES, SELFIES, molecular formulas, and IUPAC names.Source: A Preliminaries · p. 16

B Details of SMolInstruct

B.1 The statistics of SMolInstruct

SMolInstruct is a 14-task dataset with about 3.3M distinct samples and 1.6M distinct molecules, spanning varied molecular complexity, size, and structure.

B.1 The statistics of SMolInstruct

  • About 3.3M distinct samples cover 4 types and 14 selected tasks; samples do not duplicate the same core chemical information across different language templates.The authors describe the dataset as a benchmark for training and evaluating chemistry LLMs.Source: §B.1 · p. 16
  • 1.6M distinct molecules are represented, with statistics covering Bertz complexity, atom count, molecular weight, and ring count.The figure removes the long-tail portions for a clearer presentation.Source: §B.1 · p. 16, Figure 2 · p. 17
  • The reported molecular values vary widely, indicating broad coverage in complexity, size, and structure, and greater complexity and diversity than Mol-Instructions.The authors describe this scale, complexity, and diversity as making SMolInstruct well-suited for learning chemistry LLMs.Source: §B.1 · p. 17

B.2 Details of Dataset Construction

SMolInstruct construction combines data collection, quality control, splitting, and instruction creation; the paper compares its small-molecule dataset and model experiments with Mol-Instructions.

B.2 Details of Dataset Construction

  • Dataset construction uses data collection, quality control, data splitting, and instruction creation, with SMILES/SELFIES canonicalization and verbalization described as common steps.Across tasks, invalid SMILES and duplicate samples are removed.Source: §B.2 · pp. 17–18
  • Name-conversion samples pair PubChem SMILES, IUPAC names, and formulas; property prediction uses selected MoleculeNet small-molecule datasets, while captioning and generation data are filtered for low quality.Property-prediction answers are expressed as numbers for regression or Yes/No for binary classification.Source: §B.2 · pp. 17–18
  • Forward-synthesis and retrosynthesis data come from USPTO-full and receive task-specific cleaning to remove duplicates and handle chemicals appearing on both sides of reactions.Retrosynthesis processing also relabels reactants and reagents using atom mapping and splits multiple products into separate samples.Source: §B.2 · p. 18

C Comparison with Mol-Instructions

  • For molecule-related data, SMolInstruct focuses on small molecules and offers more tasks, samples, representations, and careful curation overall, despite fewer MC and MG samples than Mol-Instructions.It includes more FS and RS samples, adds four name-conversion tasks, and excludes reagent prediction because of limited high-quality data and practicality.Source: C Comparison with Mol-Instructions · p. 18, Table 6 · p. 19
  • The authors report that LlaSMol outperforms models trained on Mol-Instructions and GPT-4 and Claude 3 Opus, and narrows the gap with SoTA task-specific models.They also report insights on multi-task training and LoRA settings.Source: C Comparison with Mol-Instructions · p. 19

D Details of Experimental Setup

This section specifies LlaSMol training and inference, the comparison systems and task-specific baselines, and the metrics used for evaluation.

D.1 LlaSMol Models

  • LlaSMol fine-tunes Galactica, Llama 2, Code Llama, and Mistral on SMolInstruct with shared training and evaluation configurations.Training uses LoRA for three epochs; inference uses beam search with task-dependent numbers of output sequences.Source: §D.1 · p. 20

D.2 Compared LLMs

  • GPT-4 and Claude 3 Opus are evaluated on 500 test samples each with task-specific prompts, while zero-shot GPT-4 performs best on most tasks.GPT-4’s 0-, 1-, 3-, and 5-shot results vary across tasks; the paper reports its zero-shot results.Source: §D.2.1 · pp. 20–21, Figure 3 · p. 20, §D.2.2 · p. 21
  • Galactica is evaluated zero-shot, whereas Llama 2, Code Llama, and Mistral use one-shot prompts; output extraction may lower reported validities.Galactica and the three base models can produce extra or varied output formats that heuristic extraction may not handle correctly.Source: §D.2.3 · p. 21, §D.2.4 · p. 21
  • Molinst uses its released checkpoint and zero-shot prompts, while ChemLLM is evaluated from its released checkpoint with the GPT-4 prompt format.Both systems’ outputs may require heuristic extraction, which can miss formats and lower reported validity.Source: §D.2.5 · p. 22, §D.2.6 · p. 22

D.3 Task-Specific Models

  • Task-specific baselines include STOUT for name conversions, RDKit for formula conversion, STOUT plus RDKit for IUPAC-to-formula conversion, Uni-Mol for property prediction, and MolT5 for molecule-language tasks.STOUT could not be retrained, so possible overlap with SMolInstruct test samples may inflate its results; Uni-Mol and MolT5 use different evaluation data from their original papers.Source: §D.3.1 · p. 22, §D.3.2 · p. 22, §D.3.3 · p. 22
  • RSMILES is retrained for forward and retrosynthesis, while Molecular Transformer is retrained for both tasks using canonical SMILES without augmentation.RSMILES forward-synthesis results are lower than its reported USPTO-MIT results because this dataset includes stereochemical information; its retrosynthesis results are comparable to those reported on USPTO-full.Source: §D.3.6 · p. 23, §D.3.7 · p. 23

D.4 Evaluation Metrics

  • Evaluation uses exact match, molecular fingerprint similarity, text similarity, regression error, accuracy, and SMILES validity, with additional top-k and classification metrics in Appendix E.Exact match uses task-specific equivalence rules, including molecular identity for SMILES outputs; top-k exact match checks the generated top-k outputs.Source: §D.4 · p. 24

E Detailed Experimental Results

  • Appendix E presents broader results across models and metrics, followed by task-by-task analysis.The section introduction says it provides performance over a wider range of metrics and further discussion.Source: E Detailed Experimental Results · p. 24

E.1 Name Conversion Tasks

On name-conversion tasks, fine-tuned LlaSMol models substantially outperform other open-source LLMs, but do not uniformly match task-specific systems. They approach those systems on three tasks and remain far behind on NC-S2I.

E.1 Name Conversion Tasks

  • Open-source Llama 2, Code Llama, and Mistral correctly predict none of the four name-conversion tasks, even when they sometimes achieve fairly good validity.Galactica produces some correct predictions but remains low-performing; the authors associate the other models’ failures with limited conversion reasoning.Source: §E.1 · pp. 24–25
  • LlaSMol substantially outperforms other open-source LLMs on name conversions, approaches task-specific systems on NC-I2F, NC-I2S, and NC-S2F, but remains far behind on NC-S2I.The authors identify NC-S2I as the hardest of the four tasks; STOUT reaches 56.5% accuracy there.Source: §E.1 · pp. 24–25, Table 7 · p. 25, Table 8 · p. 25

E.2 Property Prediction

Across property prediction, molecule description, and chemical-reaction tasks, LlaSMolMistral is generally the strongest LLM, though task-specific models remain ahead in some comparisons. The results also identify limits in the available evaluation measures and variation across tasks.

E.2 Property Prediction

  • LlaSMolMistral beats other LLMs, including GPT-4, on PP-ESOL and PP-Lipo, but trails Claude 3 Opus on PP-ESOL and Uni-Mol on both regression tasks.The cited discussion attributes Uni-Mol's advantage possibly to its 3D structure knowledge.Source: §E.2 · p. 27, Table 11 · p. 27
  • On PP-SIDER, LlaSMolMistral outperforms all LLMs and Uni-Mol; PP-HIV results are identical because models predict most samples as negative.The authors describe the PP-HIV dataset as imbalanced.Source: §E.2 · pp. 27–28, Table 12 · p. 27

E.3 Molecule Description

  • On MC, LlaSMolMistral leads LLMs but trails MolT5; the text-based metrics measure similarity to gold descriptions, not necessarily chemical correctness.The authors could not obtain correctness measures with their current resources.Source: §E.3 · p. 28, Table 13 · p. 28
  • On MG, LlaSMolMistral leads LLMs, and FTS may better reflect ability to generate because the task does not require matching the gold molecule exactly.The authors caution that the metrics measure similarity to gold molecules, not necessarily correctness.Source: §E.3 · p. 28, Table 14 · p. 29

E.4 Chemical Reaction

  • Across FS and RS, LlaSMolMistral leads LLMs but ties on validity; task-specific methods remain stronger, and untuned LLMs achieve no more than 2% accuracy on RS.The authors report that LlaSMol is closer to task-specific methods than other LLMs on FS.Source: §E.4 · p. 28, Table 15 · p. 29, Table 16 · p. 30

E.5 Other Common Findings

  • GPT-4's 0-shot setting performs best on most tasks, while Claude 3 Opus surpasses GPT-4 on all chemistry tasks.The authors suggest GPT-4 may lack sufficient chemistry knowledge to learn these tasks effectively from a few examples.Source: §E.5 · p. 30

F.1 Task Synergy

The experiments compare multi-task training with single-task and task-removal variants. Multi-task training generally performs better, while removing individual task groups usually has small, inconsistent effects, with notable exceptions when NC is removed.

F.1 Task Synergy

  • The study compares models trained on one task or task group with LlaSMolMistral, which is trained on all 14 tasks using the same training setup.The task-group models include combined NC and PP task models as well as separate MC, MG, FS, and RS models.Source: §F.1 · p. 30
  • Multi-task LlaSMolMistral outperforms single-task models on most tasks, with particularly substantial gains on PP-ESOL, PP-BBBP, PP-Clintox, MC, and MG.The authors suggest shared knowledge, such as SMILES understanding, may contribute to the gains.Source: §F.1 · p. 30, Table 17 · p. 31
  • Removing tasks usually causes small, inconsistent performance shifts, but removing NC greatly reduces performance on PP-ESOL and MG.The authors hypothesize that NC-learned chemical-structure understanding may help PP-ESOL, while IUPAC-to-SMILES conversion may help MG.Source: §F.1 · p. 31, Table 18 · p. 31

F.2 Influence of LoRA Modules and Trainable Parameters

The LoRA experiments vary which model modules are trainable and how many parameters are updated. More extensive LoRA settings improve performance on most tasks, with inconsistent gains on some property-prediction tasks.

F.2 Influence of LoRA Modules and Trainable Parameters

  • The five settings range from Lite (6.8M trainable parameters; q/v projections) to LlaSMol (41.9M; attention and FFN projections) and Plus (173.0M; trainable lm head).Attn uses all attention projections (13.6M parameters), while FFN uses all FFN projections (28.3M).Source: §F.2 · pp. 31–32
  • Across most tasks, adding LoRA modules from Lite through Attn, FFN, and LlaSMol substantially improves performance; Plus adds a further slight gain.Figure 4 covers 14 tasks; except for PP-ESOL and PP-Lipo, higher scores are better.Source: §F.2 · p. 32, Figure 4 · p. 33
  • Some property-prediction tasks do not improve consistently as additional LoRA modules are added.The authors suggest small training and test sets may reduce robustness and increase the effects of randomness.Source: §F.2 · p. 32
Loading 2402.09391v4…