PaperlayerView exact revision on arXiv

Source-linked AI summary

MolecularGPT: Open Large Language Model (LLM) for Few-Shot Molecular Property Prediction

Yuyan Liu, Sirui Ding, Sheng Zhou, Wenqi Fan, Qiaoyu Tan

arXiv:2406.12950v2q-bio.QMcs.AIcs.CEcs.CLcs.LG

TL;DR

Molecular property prediction is important in drug discovery, but existing approaches can require costly labeled data and may generalize poorly to new tasks. MolecularGPT instruction-tunes LLMs on more than 1,000 property tasks for zero- and few-shot in-context prediction. It outperforms supervised GNN methods on 4 of 7 datasets with two-shot examples and shows competitive few-shot performance across benchmarks.

  • Problem

    Supervised molecular property prediction can require costly annotation and task-specific training, while the field lacks an LLM method that combines generalization to unseen tasks with few-shot in-context learning.

    Source: §1 · pp. 1–2

  • Method

    MolecularGPT instruction-tunes LLMs on diverse molecular property tasks using structure-aware demonstrations to support zero- and few-shot in-context prediction on novel tasks.

    Source: §1 · p. 2

  • Results

    With two-shot examples, MolecularGPT surpasses supervised GNN methods on 4 of 7 datasets and improves average classification performance over leading LLM baselines by up to 15.7%.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The findings demonstrate the potential of instruction-tuned LLMs as few-shot molecular property predictors across multiple datasets.

    Source: §5 · p. 8

    SMILES omits molecular 3D geometry, the work focuses on property prediction rather than other foundational molecular tasks, and regression remains challenging relative to GNNs.

    Source: §6 · p. 9

Abstract

from arXiv · show

Molecular property prediction (MPP) is a fundamental and crucial task in drug discovery. However, prior methods are limited by the requirement for a large number of labeled molecules and their restricted ability to generalize for unseen and new tasks, both of which are essential for real-world applications. To address these challenges, we present MolecularGPT for few-shot MPP. From a perspective on instruction tuning, we fine-tune large language models (LLMs) based on curated molecular instructions spanning over 1000 property prediction tasks. This enables building a versatile and specialized LLM that can be adapted to novel MPP tasks without any fine-tuning through zero- and few-shot in-context learning (ICL). MolecularGPT exhibits competitive in-context reasoning capabilities across 10 downstream evaluation datasets, setting new benchmarks for few-shot molecular prediction tasks. More importantly, with just two-shot examples, MolecularGPT can outperform standard supervised graph neural network methods on 4 out of 7 datasets. It also excels state-of-the-art LLM baselines by up to 15.7% increase on classification accuracy and decrease of 17.9 on regression metrics (e.g., RMSE) under zero-shot. This study demonstrates the potential of LLMs as effective few-shot molecular property predictors. The code is available at https://github.com/NYUSHCS/MolecularGPT.

1 Introduction

MolecularGPT addresses the limits of label-intensive, task-specific molecular property prediction with instruction-tuned LLMs designed for zero- and few-shot prediction on unseen tasks. Its structure-aware demonstrations and hybrid instruction set support these in-context learning settings.

1 Introduction

  • Supervised GNN methods are limited by costly molecule annotation and task-specific training that can hinder adaptation to new tasks in open-world settings.The introduction notes that these constraints conflict with practical needs.Source: §1 · p. 1
  • Earlier zero-shot MPP efforts integrate language and molecular representations, but CLAMP is limited to classification, while another generative approach restricts few-shot prediction through added architectural design.The introduction identifies a gap in combining generalization to new tasks with LLM-style few-shot in-context learning.Source: §1 · pp. 1–2
  • MolecularGPT is presented as an instructionally fine-tuned LLM for zero- and few-shot property prediction on unseen tasks without task-specific fine-tuning.The introduction frames this as adapting pre-trained LLMs to molecular property prediction through in-context learning.Source: §1 · p. 2
  • Structure-aware few-shot instructions use globally retrieved similar molecules as demonstrations, combining molecular structure and SMILES in a data-centric design.The approach is contrasted with prior work that fuses graph structures and SMILES from a model-centric perspective.Source: §1 · p. 2
  • The hybrid instruction set mixes zero-shot and few-shot instructions across more than 1,000 property tasks, spanning classification and regression in biological, chemical, and quantum mechanical domains.The set comprises 3.5 GB of training tokens.Source: §1 · p. 2
  • With two-shot examples, MolecularGPT surpasses supervised GNN methods on 4 of 7 datasets and improves average classification performance over leading LLM baselines by up to 15.7%.The evaluation covers 10 molecular property benchmarks.Source: §1 · p. 2

2 Related work

Related work includes task-specific supervised graph models and molecular language models that still require downstream fine-tuning. MolecularGPT instead combines broad instruction coverage with structure-aware few-shot demonstrations.

2 Related work

  • Supervised GNNs are trained implicitly for each task, while meta-learning extensions require many related tasks and remain label-intensive.These methods also rely on molecular structure and overlook textual knowledge from wet-lab experiments.Source: §2 · p. 2
  • Pretrain-finetune molecular language models learn molecular representations and textual descriptions, but still require fine-tuning for particular downstream MPP tasks.This requirement limits their generalization to new tasks.Source: §2 · pp. 2–3
  • The MolecularGPT framework uses a hybrid of zero-shot and few-shot instructions across more than 1,000 property tasks, selecting top-k neighboring molecules as labeled demonstrations for few-shot prompts.The demonstrations are adaptively selected for each query molecule.Source: Figure 1 · p. 3
  • Instruction-tuned molecular models address graph-property alignment, but GIMLET lacks few-shot ICL and Mol-Instructions covers only three property tasks while neglecting inter-molecular correlations.MolecularGPT contrasts these approaches with a 1,000-task instruction set and structure-aware few-shot instructions.Source: §2 · p. 3

3 Method

MolecularGPT adapts LLMs to molecular property prediction through instruction tuning, combining SMILES-based prompts, similar-molecule demonstrations, and hybrid zero- and few-shot instructions.

3 Method

  • Molecular instruction tuning fits an LLM to input-output instructions built from labeled molecular graphs for prediction on disjoint test tasks or molecules.The method addresses the challenges of representing diverse molecules, incorporating graph structure, and learning when labeled examples are scarce.Source: §3 · pp. 3–4

3.1 SMILES-based Molecular Instruction Tuning: A Unified Step

  • The unified zero-shot template represents molecules as SMILES strings and maps the property question, query molecule, and label to instruction, input, and response fields.The template is specified as ### Instruction: {instruction} ### Input: {inputs} ### Response: {output}.Source: §3.1 · p. 4

3.2 Structure-Aware Molecular Instruction Tuning: Graph Structure Matters

  • Structure-aware tuning retrieves each query molecule’s top-K similar molecules using Tanimoto similarity on MACCS Keys, then orders their labeled demonstrations by similarity.The retrieved neighbors extend the zero-shot question into a few-shot prompt; the authors report that including neighbors’ property targets improves few-shot performance.Source: §3.2 · p. 4

3.3 Hybrid Molecular Instruction Tuning: Better Few-Shot Learner

  • The constructed structure-aware instruction set is used to fine-tune a pretrained LLM, initialized in practice as LLaMA2-7b-chat with QLoRA.The training set applies the structure-aware few-shot template to each molecule in the training data.Source: §3.3 · p. 4
  • Fine-tuning on zero-shot instructions favors zero-shot performance, whereas four-shot instructions favor one-shot performance, revealing a trade-off between settings.The comparison uses the CYP450 dataset with K = 0 and K = 4 instruction sets.Source: §3.3 · pp. 4–5
  • The hybrid instruction set Sh combines zero-shot template T and few-shot template Tshot to blend their strengths, addressing the trade-off: LLaMA2 fine-tuned on T’s 0-shot set performs well in zero-shot but underperforms in one-shot, while the Tshot K=4 4-shot model shows the reverse.Source: §3.3 · p. 5
  • Hybrid instruction-tuned models consistently outperform the alternatives in both zero-shot and one-shot CYP450 settings.The hybrid set combines 0-, 1-, 2-, 3-, and 4-shot templates.Source: §3.3 · p. 5, Figure 2 · p. 5

4 Experiment

Experiments evaluate MolecularGPT on molecular property benchmarks and test how instruction-set design and retrieval choices affect zero- and few-shot performance. The reported results show strong few-shot comparisons and examine design choices that influence in-context learning and generalization.

4.1 Experimental Setup

  • The evaluation addresses three questions and uses MoleculeNet and CYP450 datasets, ROC-AUC for classification, and RMSE for regression, comparing against GNN, molecular-language, and few-shot baselines.The setup describes 657 MPP tasks and lists five GNN methods, five molecular language models, two few-shot baselines, and LLaMA-chat-7B.Source: §4 · p. 5, §4.1 · p. 5

4.2 Performance Evaluation

  • With two-shot inference, MolecularGPT surpasses supervised GNNs on 4 of 7 classification tasks; under zero-shot, it improves classification by 15.7% and regression metrics by 17.9 versus LLaMA.Source: §4.2 · pp. 5–6, Table 1 · p. 6, Table 2 · p. 7

4.3 Tuning on Hybrid Instruction Set

  • Zero-shot-tuned instructions yield a 4% average classification improvement for zero-shot over two-shot inference, while four-shot tuning raises two-shot classification performance by 8.2% versus zero-shot tuning.The zero-shot-tuned model performs poorly on many tasks under two-shot inference, and four-shot tuning leaves zero-shot generalization subpar on many tasks.Source: §4.3 · p. 6
  • Mixed-shot instruction sets improve performance over pure zero- or four-shot tuning across tested scenarios, and doubling the 0&4-shot set improves performance on several tasks in both inference settings.The mixed sets combine zero- and four-shot instructions or mix zero through four shots; reported tasks include BACE, HIV, Tox21, BBBP, CYP450, and MUV.Source: §4.3 · p. 6

4.4 Hyperparameter Sensitivity Analysis

  • MolecularGPT’s instruction-response standard deviation is 0.06 versus GIMLET’s 0.08, and MACCS-ECFP4 retrieval outperforms MACCS on 7 of 10 tasks in the eight-shot setting.Source: §4.4 · p. 7, Table 3 · p. 8, Table 4 · p. 8
  • Performance improves with up to two retrieved demonstrations across datasets but does not improve further with more; the authors hypothesize added noise and a 512-token input limit constrain larger sets.The passage also states that instructions can contain at most four examples, despite the evaluation varying retrieval counts up to eight.Source: §4.4 · pp. 7–8
  • Similarity retrieval outperforms diversity retrieval across classification tasks, with diversity 2.7% lower on average in eight-shot inference; ascending order is 0.9% lower than descending order there.Source: §4.4 · p. 8, Figure 4 · p. 8

5 Conclusion

MolecularGPT is instruction-tuned on over 1000 prediction tasks to support out-of-domain zero- and few-shot ICL, and the study reports stronger few-shot performance than baseline language and supervised models.

5 Conclusion

  • MolecularGPT is instruction-tuned on over 1000 prediction tasks to support out-of-domain prediction through zero- and few-shot ICL.Source: §5 · p. 8
  • MolecularGPT consistently outperforms baseline language models in few-shot scenarios and surpasses supervised models on multiple datasets.Source: §5 · p. 8
  • Future work plans to incorporate additional molecular modalities and expand to tasks such as molecule captioning.Source: §5 · p. 8

6 Limitation

The model uses SMILES strings and focuses on property prediction, with stated limitations in representing 3D structure, broader task coverage, and regression performance.

6 Limitation

  • SMILES strings omit geometric information such as each atom’s 3D position, which the authors say hinders molecular-structure representation.Source: §6 · p. 9
  • The study covers property prediction but not foundational tasks such as molecule optimization, generation, or captioning, potentially restricting practical applications.Source: §6 · p. 9
  • Although compatible with supervised GNNs for classification, MolecularGPT still has gaps with them on regression because generating numbers remains challenging for foundational LLMs.Source: §6 · p. 9

A.1 Details of datasets

The appendix describes the downstream and instruction-tuning datasets, then outlines how property-prediction instructions pair task descriptions and retrieval examples with molecule inputs and property labels.

A.1 Details of datasets

  • Downstream datasets follow GIMLET’s selection and splitting criteria, use 0.8:0.1:0.1 splits, and are evaluated on test sets.The tuning data additionally include QM9 HOMO and LUMO properties.Source: §A.1 · p. 12
  • For the added QM9 properties, the authors wrote task descriptions using Wikipedia and chemistry papers, then used GPT-4.0 to generate varied question-framing instructions from seed examples.Source: §A.1 · p. 12

A.2 Details of instructions

  • Each instruction-tuning example combines a property description and retrieval examples with a query-molecule SMILES input and its property-label output.Source: §A.2 · p. 12
  • The examples span ChEMBL bioassay and property tasks and QM9, including an assay-effectiveness question answered with a molecule label.Source: §A.2 · p. 12

B Training Setup

Training uses QLoRA with memory and speed optimizations, a 512-token maximum input length, and specified GPU hardware for training and inference.

B Training Setup

  • The authors fine-tune LLaMA2-chat-7B with QLoRA and use DeepSpeed ZeRO stage 2, FlashAttention-2, and BFloat16 mixed precision for memory utilization and speed.Source: B Training Setup · p. 12
  • The learning rate is 3e-4 and the maximum input length is 512 tokens.Source: B Training Setup · p. 12
  • Training runs on four Tesla A800-80G GPUs, while inference uses one RTX 3090 GPU.Source: B Training Setup · p. 12

C.1 The robustness of MolecularGPT

MolecularGPT is reported to perform robustly across variations in instruction phrasing. The appendices also assess instruction-dataset mixtures and inference strategies, including demonstration ordering and retrieval diversity.

C.1 The robustness of MolecularGPT

  • MolecularGPT exhibits robust zero-shot performance across original, detailed, expanded, rewritten, and shortened instructions.The evaluation uses instruction variants constructed in GIMLET and reports ROC-AUC or RMSE standard deviations in Table 6.Source: Table 6 · p. 13, §C.1 · p. 14

C.2 The effect of instruction datasets

  • The study compares zero- and few-shot performance after fine-tuning on single-shot, mixed-shot, and doubled-scale instruction datasets.The mixtures include equal 0-shot and 4-shot proportions, and a 0–4-shot mixture weighted 0.6:0.1:0.1:0.1:0.1, with duplicate query molecules excluded and dataset scale maintained.Source: Table 7 · p. 13, Table 8 · p. 13, §C.2 · p. 14

C.3 The effect of inference strategies

  • Inference experiments examine ascending-order retrieval demonstrations and diversity-based retrieval against similarity-based retrieval.The ordering analysis reports results for two-shot and above; diversity-based retrieval is designed for classification and evaluated with even-numbered shot counts to balance classes.Source: §C.3 · p. 14, Table 12 · p. 15, Table 13 · p. 15

Related papers

Loading 2406.12950v2…