PaperlayerView exact revision on arXiv

Source-linked AI summary

What can Large Language Models do in chemistry? A comprehensive benchmark on eight tasks

Taicheng Guo, Kehan Guo, Bozhao Nan, Zhenwen Liang, Zhichun Guo, Nitesh V. Chawla, Olaf Wiest, Xiangliang Zhang

arXiv:2305.18365v3cs.CLcs.AI

TL;DR

The paper examines what LLMs can do in chemistry, where their capabilities remain unclear because systematic evaluation across practical tasks has been limited. It builds a benchmark spanning eight chemistry tasks and evaluates five LLMs under zero-shot and few-shot settings. GPT-4 performs best overall, while competitiveness varies by task format and the models show limitations with molecular representations and evaluation metrics.

  • Problem

    LLM capabilities in chemistry remain unclear because prior work has not systematically evaluated diverse practical chemistry tasks and their distinct knowledge and reasoning requirements.

  • Method

    The paper benchmarks five LLMs on eight chemistry tasks using recognized datasets, task-specific prompts, selected demonstrations, and zero-shot and few-shot in-context learning.

  • Results

    GPT-4 outperforms other evaluated models overall, while LLMs are less competitive on SMILES-intensive generative tasks and more competitive on classification or ranking tasks.

  • Takeaways & Limitations

    LLM performance in chemistry is task-dependent, with stronger results when tasks involve classification, ranking, or text in prompts than when they require deep molecular-SMILES understanding.

  • Takeaways & Limitations

    LLMs often struggle to understand molecular SMILES, producing inaccurate or inconsistent results and chemically unreasonable outputs in some tasks.

Abstract

from arXiv · show

Large Language Models (LLMs) with strong abilities in natural language processing tasks have emerged and have been applied in various kinds of areas such as science, finance and software engineering. However, the capability of LLMs to advance the field of chemistry remains unclear. In this paper, rather than pursuing state-of-the-art performance, we aim to evaluate capabilities of LLMs in a wide range of tasks across the chemistry domain. We identify three key chemistry-related capabilities including understanding, reasoning and explaining to explore in LLMs and establish a benchmark containing eight chemistry tasks. Our analysis draws on widely recognized datasets facilitating a broad exploration of the capacities of LLMs within the context of practical chemistry. Five LLMs (GPT-4, GPT-3.5, Davinci-003, Llama and Galactica) are evaluated for each chemistry task in zero-shot and few-shot in-context learning settings with carefully selected demonstration examples and specially crafted prompts. Our investigation found that GPT-4 outperformed other models and LLMs exhibit different competitive levels in eight chemistry tasks. In addition to the key findings from the comprehensive benchmark analysis, our work provides insights into the limitation of current LLMs and the impact of in-context learning settings on LLMs' performance across various chemistry tasks. The code and datasets used in this study are available at https://github.com/ChemFoundationModels/ChemLLMBench.

1 Introduction

The paper addresses the limited systematic evaluation of LLMs on practical chemistry tasks by establishing a broad, carefully controlled benchmark. It evaluates diverse tasks, models, prompts, and in-context learning settings to characterize chemistry-specific capabilities and performance.

  • Practical chemistry applications of LLMs remain underinvestigated, with prior studies tending to focus on specific case studies rather than systematic evaluation.
  • A comprehensive benchmark is needed because chemistry tasks differ in their knowledge, reasoning, and representational requirements.
  • The benchmark covers 8 practical tasks spanning name, property, yield, reaction, retrosynthesis, molecule design, captioning, and reagent selection.
  • Five LLMs are evaluated with zero-shot and few-shot in-context learning using recognized chemistry datasets, selected demonstrations, and task-specific prompts.
  • Repeated evaluations and validation-based prompt configuration selection are used to address prompt effects and output randomness.
  • The study reports that GPT-4 outperforms GPT-3 and Davinci-003 and that prompt inclusion of property-label semantics can improve property prediction.

2 Related Work

Prior LLM research has concentrated on general NLP and selected scientific applications, while chemistry has received less systematic attention. This paper addresses that gap by evaluating LLMs directly on chemical tasks and positioning the work alongside tool-using chemistry agents and downstream-task applications.

  • Existing LLM evaluation research primarily examines general NLP, healthcare, mathematics, and code generation, with less focus on specialized chemistry domains.
  • This work transforms chemical tasks for LLM processing and conducts a comprehensive evaluation of their ability to tackle chemistry-related problems.
  • Prior chemistry efforts include LLM-based agents that use expert-designed tools or planning and execution to conduct downstream tasks and chemical experiments.
  • Other studies directly apply LLMs to chemistry-related downstream tasks, but systematic evaluation of their domain capabilities has been lacking.

3 The Evaluation Process and Setting

The evaluation examines eight practical chemistry tasks through standardized zero-shot and task-specific few-shot prompts, comparing five LLMs with selected datasets and metrics. Prompt strategies, demonstration examples, retrieval methods, and repeated evaluations are used to assess performance consistently.

  • Evaluation workflow: The workflow identifies tasks with chemistry experts, designs and selects prompts, then evaluates model answers qualitatively and quantitatively.Quantitative assessment uses selected task metrics, while chemists judge usefulness in real-world scenarios.
  • Chemistry tasks: The benchmark covers eight practical chemistry tasks spanning understanding, reasoning, and explaining, with task types, datasets, and metrics summarized for evaluation.The tasks include name, property, yield, reaction, retrosynthesis, molecule design, molecule captioning, and reagent selection.
  • Models: Five LLMs—GPT-4, GPT-3.5, Davinci-003, Llama, and Galactica—are evaluated across all chemistry tasks.
  • Prompt design: The standardized zero-shot template instructs models to act as chemists and return only the task-specific output, while ICL prompts add demonstrations and task-specific instructions.The ICL template combines general instructions, task-specific explanations and restrictions, demonstrations, and the question.
  • In-context learning: ICL strategies compare randomly selected demonstrations with scaffold-based retrieval of chemically similar examples.Scaffold retrieval uses molecular structure when the input is represented as SMILES.
  • Experimental setup: The study searches demonstration counts across task-specific settings, selects representative options using a 30-example validation set, and tests them on 100 randomly sampled instances.The searched values include k in {4, 8}, {5, 20}, or {5, 10}, depending on the task group.

4 Experiment Analysis

Across eight chemistry tasks, LLM performance varies by task formulation and chemistry-specific demands. GPT-4 leads the evaluated models, while in-context learning generally improves results when examples are selected and scaled appropriately.

  • GPT models are less competitive on precise molecular-string tasks, including name prediction, reaction prediction, and retrosynthesis.The paper attributes these weaknesses partly to difficulty understanding SMILES, IUPAC names, and molecular formulas, especially when reaction transformations must also be modeled.
  • GPT models perform strongly on text-related explanation tasks and text-based molecule design, but exact-match accuracy for molecule design is below 20%.More than 89% of generated molecules are chemically valid, although exact identity with the ground truth is often absent and real-world utility remains to be assessed.
  • GPT models are competitive on classification- and ranking-formulated tasks, including property prediction, yield prediction, and reagent selection.Reagent selection reaches 40% to 50% accuracy; yield prediction remains below UAGNN on two reported datasets, at 80% versus 96% and 76% versus 96%.
  • GPT-4 shows better chemical understanding, reasoning, and explaining abilities than Davinci-003, GPT-3.5, Llama, and Galactica.
  • In-context prompting outperforms zero-shot prompting across all tasks, while scaffold-similar retrieval and larger numbers of demonstrations usually improve performance.The evaluation compares 12 variants on validation data and uses repeated evaluations because GPT outputs can vary across identical API calls.
  • SELFIES representations perform worse than SMILES across the four tested tasks, and temperature is tuned using sampled validation experiments to conserve API resources.The temperature analysis evaluates GPT-4 with scaffold sampling and k = 8 using F1 and accuracy across settings.

5 Discussion

The discussion identifies SMILES understanding and chemical hallucinations as major limitations, while noting that LLMs may offer generalized chemistry capabilities despite underperforming task-specific baselines.

  • Limitations: LLMs often misunderstand SMILES representations, producing inaccurate or inconsistent results across chemistry tasks.The issue includes difficulty interpreting implicit hydrogens and ambiguity from multiple valid SMILES representations.
  • Hallucinations: LLMs exhibit hallucinations both when interpreting SMILES inputs and when generating chemically unreasonable SMILES outputs.These failures are especially problematic because chemistry tasks require exact SMILES matching and adherence to chemical facts.
  • Performance context: LLMs underperform relative to baselines across a majority of tasks, although they use only a few examples while baselines rely on extensive task-specific training data.The comparison highlights a difference in training regime and task coverage rather than performance alone.
  • Prospects: Advanced prompting methods such as Chain-of-thought and Decomposed Prompting could potentially improve LLMs’ complex chemistry reasoning.The paper presents these methods as prospective approaches, not established solutions.
  • Safety: Generating chemically valid molecules also creates misuse risks, including the production of hazardous, toxic, or illegal substances.The authors call for robust safeguards and ethical guidelines to deter harmful applications.
  • Impacts: The benchmark offers insights and recommendations for AI researchers and chemists while helping address concerns about replacing chemists with AI.The paper frames objective evaluation as contributing to public education and a better understanding of LLM capabilities.

6 Conclusion and Future Work

The paper constructs a benchmark of five LLMs across eight chemistry tasks to characterize their capabilities. LLMs are less competitive on SMILES-intensive generative tasks, more competitive on classification or ranking tasks, and selectively competitive on text-based or explainable tasks.

  • Benchmark: The benchmark evaluates GPT-4, GPT-3.5, Davinci-003, Llama, and Galactica on eight widely used chemistry tasks.The tasks span name, property, yield, reaction, retrosynthesis, molecule design, captioning, and reagent selection.
  • Findings: LLMs are less competitive on generative tasks requiring in-depth SMILES understanding, including reaction prediction, name prediction, and retrosynthesis.These experiments indicate a need for further improvement in chemistry capabilities.
  • Findings: LLMs show competitive performance on classification or ranking tasks such as yield prediction and reagents selection.
  • Findings: LLMs are selectively competitive on property prediction, text-based molecule design, and molecule captioning.These tasks involve text in prompts or explainable outputs.

A Name Prediction

The name-prediction section tests whether GPT models understand chemical names and molecular representations across four translation tasks. Results show extremely low accuracy, with GPT-4 case studies illustrating failures to interpret molecular structure details.

  • Task setup: Four name-prediction tasks translate among SMILES, IUPAC names, and molecular formulas.The tasks include SMILES-to-IUPAC, IUPAC-to-SMILES, SMILES-to-formula, and IUPAC-to-formula translation.
  • Task setup: The ICL prompt for each name-translation task changes the task labels and corresponding input and output names.
  • Results: 0 accuracy was achieved in the smiles2iupac task, while the best accuracy was 0.014 in iupac2smiles, 0.086 in smiles2formula, and 0.118 in iupac2formula.These values are reported for the best methods under optimal prompt settings selected through validation-set grid search.
  • Results: GPT-4 produced wrong answers in all example name-prediction tasks under the Scaffold, k=20 method.In smiles2formula, the examples show failures to count carbon atoms and infer the correct number of hydrogen atoms.

B Molecule Property Prediction

The property-prediction section evaluates LLMs on binary molecular-property datasets using task-specific prompts and ICL examples. GPT models outperform the baseline on F1 for four of five datasets, with performance affected by example selection, ICL size, and label context.

  • Task setup: Property prediction uses SMILES paired with binary labels for properties including blood-brain barrier penetration, HIV activity, toxicity, and clinical-trial toxicity failure.The evaluation uses BBBP, HIV, BACE, Tox21, and ClinTox datasets.
  • Task setup: The ICL prompt explains the target property and constrains outputs to characteristics of the property-prediction task.For BBBP, the prompt explains forecasting penetration of the brain-blood barrier.
  • Results: GPT models outperform the baseline in F1 on four out of five molecular-property datasets.The reported evaluation includes F1 and accuracy metrics.
  • Results: Expanding the number of ICL instances produces a measurable enhancement in model performance.The authors report a direct relationship between ICL-data extent and predictive precision.
  • Results: Scaffold sampling outperforms random sampling on BBBP, BACE, and Tox21.The authors suggest structural resemblance between sampled and query molecules as a plausible explanation.
  • Results: GPT-4 selectively outperforms baseline models on HIV and ClinTox, likely because prompts include information related to the dataset labels.

C Yield Prediction

Yield prediction is evaluated as binary high-yield classification on two standardized high-throughput reaction datasets. GPT models remain below the trained baseline, although GPT-4 benefits from more in-context examples.

  • C Yield Prediction: Yield prediction classifies reactions as high-yielding or not using Buchwald-Hartwig and Suzuki-Miyaura HTE datasets.The datasets contain reactions and corresponding yields acquired under standardized experimental setups.
  • C Yield Prediction: The yield-prediction prompts explain reactant and product SMILES formatting and restrict outputs to valid results.Reactants are separated by ‘.’ and products by ‘>>’.
  • C Yield Prediction: GPT models perform below the UAGNN baseline, which was trained on thousands of examples.GPT-4 is the strongest GPT model for predicting reaction yields.
  • C Yield Prediction: Increasing GPT-4’s ICL examples from 4 to 8 significantly improves performance on both evaluated reaction types.The result links predictive accuracy to the quantity of contextual ICL data.

D Reaction Prediction

Reaction prediction and retrosynthesis test whether LLMs can map between reaction representations and products or precursors, while reagent selection evaluates component recommendations. GPT models lag trained baselines on reaction prediction and retrosynthesis but perform comparatively well for reagent selection.

  • D Reaction Prediction: Reaction prediction uses USPTO-MIT’s approximately 470,000 reactions, with Top-1 Accuracy and invalid-SMILES percentage reported against Chemformer.The task predicts products from reactants and reagents.
  • D Reaction Prediction: Task-specific prompts explain reaction-component formatting and constrain outputs toward chemically valid results.The prompts are adapted to reactants, reagents, and dataset characteristics.
  • D Reaction Prediction: Zero-shot GPT reaction prediction reaches only 0.004 Top-1 Accuracy and generates 17.4% invalid SMILES.The paper attributes weaker results to limited understanding of SMILES and reaction transformations.
  • D Reaction Prediction: GPT-4 and GPT-3.5 perform comparatively well on reagent-selection tasks.The study evaluates four reaction-component selection tasks from a 5,760-point Suzuki HTE screening dataset.
  • D Reaction Prediction: GPT models also perform below the baseline on retrosynthesis, which predicts available reactants by reversing product-to-precursor synthesis.The study uses USPTO-50k and the same metric and baseline as reaction prediction.

G Text-Based Molecule Design

Text-based molecule design asks models to generate molecules from descriptions and evaluates both textual and chemical similarity. GPT models can be competitive on some metrics and generate mostly chemically valid alternatives despite weaker exact matching.

  • G Text-Based Molecule Design: Text-based molecule design generates molecules from descriptions using the 33,010-pair ChEBI-20 dataset and MolT5-Large as baseline.Evaluation includes fingerprint Tanimoto similarity and Fréchet ChemNet Distance alongside textual metrics.
  • G Text-Based Molecule Design: GPT-4 and Davinci-003 are competitive with or outperform the baseline on BLEU and Levenshtein metrics.They substantially underperform on exact match and Morgan FTS.
  • G Text-Based Molecule Design: More than 89% of GPT-generated molecules are chemically valid and may serve as alternatives when they satisfy the input description.Exact identity with the ground truth is not required for a molecule to be useful in this design setting.
  • G Text-Based Molecule Design: GPT-4 with Scaffold, k=10 produces structures more similar to the ground truth than the compared GPT variants and baseline in selected case studies.GPT models outperform the baseline for metal-atom design when the baseline generates incorrect SMILES.

H Molecule Captioning

Molecule captioning evaluates whether models can describe molecular features, properties, and functional groups. GPT-4 and Davinci-003 are competitive on some text metrics, while GPT-4 case studies contain fewer chemically inaccurate descriptions than the baseline.

  • H Molecule Captioning: Molecule captioning generates textual descriptions of molecular features, properties, and functional groups from molecule inputs.The study uses ChEBI-20 and its training set as ICL candidates.
  • H Molecule Captioning: GPT-4 and Davinci-003 are competitive with or outperform the baseline on BLEU-2 and BLEU-4.The paper presents this as evidence of GPT models’ capability in molecule captioning.
  • H Molecule Captioning: GPT-4 captions contain only a few inaccurate descriptions, whereas baseline captions contain more descriptions violating chemical facts.This contrast appears in case studies covering three molecule types.
  • H Molecule Captioning: The benchmark reports model performance for molecule captioning alongside SMILES and SELFIES comparisons.Tables 15 and 19 provide the captioning results and representation comparison context.
Loading 2305.18365v3…