Source-linked AI summary
ChemLLM: A Chemical Large Language Model
Di Zhang, Wei Liu, Qian Tan, Jingdan Chen, Hang Yan, Yuliang Yan, Jiatong Li, Weiran Huang, Xiangyu Yue, Wanli Ouyang, Dongzhan Zhou, Shufei Zhang, Mao Su, Han-Sen Zhong, Yuqiang Li
TL;DR
Chemistry benchmarks are mostly built for specialist models, and style-sensitive evaluation can reward outputs with factual errors. ChemLLM pairs a chemistry instruction-tuning dataset and nine-task benchmark with a chemistry-focused LLM that puts structured knowledge into dialogue. The authors report chemistry capabilities comparable to GPT-4 and versatility beyond chemistry.
Problem
Chemistry benchmarks are mostly built for specialist models, and style-sensitive evaluation can reward outputs with factual errors.
Source: ChemBench: A large scale benchmarks… · p. 4Method
ChemLLM combines a chemistry instruction-tuning dataset and a nine-task benchmark with a chemistry-focused LLM that converts structured chemical knowledge into dialogue.
Source: Conclusion · p. 7Results
ChemLLM demonstrates chemistry capabilities comparable to GPT-4 and versatility in other fields.
Source: Conclusion · p. 7Takeaways & Limitations
ChemBench establishes standards for assessing chemical capabilities in both chemistry-focused and general LLMs.
Source: Conclusion · p. 7
Abstract
from arXiv · showhide
Large language models (LLMs) have made impressive progress in chemistry applications. However, the community lacks an LLM specifically designed for chemistry. The main challenges are two-fold: firstly, most chemical data and scientific knowledge are stored in structured databases, which limits the model's ability to sustain coherent dialogue when used directly. Secondly, there is an absence of objective and fair benchmark that encompass most chemistry tasks. Here, we introduce ChemLLM, a comprehensive framework that features the first LLM dedicated to chemistry. It also includes ChemData, a dataset specifically designed for instruction tuning, and ChemBench, a robust benchmark covering nine essential chemistry tasks. ChemLLM is adept at performing various tasks across chemical disciplines with fluid dialogue interaction. Notably, ChemLLM achieves results comparable to GPT-4 on the core chemical tasks and demonstrates competitive performance with LLMs of similar size in general scenarios. ChemLLM paves a new path for exploration in chemical studies, and our method of incorporating structured chemical knowledge into dialogue systems sets a new standard for developing LLMs in various scientific fields. Codes, Datasets, and Model weights are publicly accessible at https://hf.co/AI4Chem
Result
The results section introduces ChemData and ChemBench, reports ChemLLM’s results on chemical and general tasks, and describes its dialogue-based approach and the authors’ hopes for applying its domain-knowledge strategy to scientific LLMs.
ChemData: A large scale Instruction Tuning Dataset for chemistry
- ChemData provides 7M instruction-tuning Q&A across molecule, reaction, and other domain-specific tasks for fine-tuning chemical LLMs.Molecule tasks include name conversion, Caption2Mol, Mol2Caption, and property prediction; reaction tasks include retrosynthesis and product, yield, temperature, and solvent prediction.Source: ChemData: A large scale Instruction… · pp. 3–4
ChemBench: A large scale benchmarks for chemistry
- ChemBench establishes an objective LLM chemistry benchmark with nine molecule and reaction tasks and 4,100 multiple-choice questions, each with one correct answer.It is intended as a fair, open, reproducible basis for assessing the chemical proficiency of both chemistry-focused and general LLMs.Source: ChemBench: A large scale benchmarks… · p. 4
ChemLLM: a large language model for chemistry
- ChemLLM outscored GPT-4 on six of ChemBench’s nine tasks, ranked just below it on the other three, and surpassed GPT-3.5 across the board.The authors also report significant improvement over InternLM2-Chat-7B and stronger performance than similarly sized general LLMs.Source: Figure 4 · p. 5, ChemLLM: a large language model for… · p. 5
- 65.6 on English MMLU and 67.2 on Chinese C-Eval exceeded competing models of similar size.The results support ChemLLM’s performance in broader disciplines and multilingual scenarios despite fine-tuning primarily on chemical corpora.Source: Figure 5 · p. 6, ChemLLM: a large language model for… · p. 6
- On C-MHChem, ChemLLM scored 76.4 and surpassed GPT-4; the authors say chemical-data fine-tuning may enhance reasoning, with outcomes underscoring its capabilities in Chinese middle and high school exams.Source: ChemLLM: a large language model for… · p. 6
Conclusion
- ChemLLM is presented as a chemistry-dedicated model that handles chemical tasks through dialogue, translating structured chemical knowledge into an accessible conversational format.The conclusion also notes its versatility beyond core chemistry and performance comparable to GPT-4.Source: Conclusion · p. 7
- ChemLLM sets a precedent for scientific LLMs and paves the way for accelerating chemistry research.Source: Conclusion · p. 7
- The authors hope their professional-domain knowledge injection strategy inspires further work applying LLMs to scientific domains.Source: Conclusion · p. 7
A Instruction Construction of ChemData
ChemData instructions are constructed by converting structured chemical data into varied single-turn templates and context-rich multi-turn dialogues. The appendix illustrates template variation and dialogue prompts for chemical question answering.
A Instruction Construction of ChemData
- GPT-4 generated semantically consistent question-answer variations, with one randomly selected template used for each structured entry to create a single-turn dialogue sample.The method addresses template diversity in instruction-tuning data.Source: A Instruction Construction of ChemData · p. 11
- A “Play as Playwrights” chain-of-thought prompt turns single-turn question-answer pairs into context-rich, logically connected multi-turn dialogues.The authors say this reduces the need for dialogue rounds and manual intervention.Source: A Instruction Construction of ChemData · p. 11
A.1 Sources of Raw Data
- Table S1 is identified as listing sources of publicly available databases used for raw data.The supplied passage does not name the databases.Source: §A.1 · p. 11, Table S1 · p. 12
A.2 Seed Template Prompt technique
- The appendix identifies Figure S1 as the pipeline for the Seed Template Prompt technique.The supplied passages identify the figure but do not describe its steps.Source: §A.2 · p. 11, Figure S1 · p. 12
- The Seed Template Prompt technique includes varied phrasings for converting between IUPAC names and SMILES strings.The examples include query and answer templates, with formal, informal, and polite variations.Source: §A.2 · pp. 11, 13
A.3 Play as Playwrights Prompt technique User
- A sample prompt asks for 5 to 10 linked question-answer rounds between a chemistry teacher and a colleague or student.Each follow-up question extends the previous exchange, and answers are requested to be useful, informative, and concise.Source: §A.3 · p. 13
A.4 Play as Playwrights with answer masking User
- The answer-masking prompt has a chemistry-expert agent ask for missing chemical information as a human supplies it gradually, then answer the question.It asks that the agent's questions concern the chemical information, not the method or procedure for solving the problem.Source: §A.4 · p. 13
B. Multiple-choice question construction
ChemBench multiple-choice items are built from chemical QA data, paired with three incorrect answers, and rendered in varied natural-language templates.
B. Multiple-choice question construction
- Each multiple-choice item combines one correct answer with three wrong answers, randomly assigned to A–D after raw QA data are converted into natural-language templates.Wrong answers are sampled around ground-truth values for value-prediction tasks; for other tasks, they come from same-task answers or GPT-4 generation.Source: of ChemBench · p. 14
Settings
ChemLLM is fine-tuned on chemical and general instruction data using distributed training, LoRA, and autoregressive cross-entropy. The reported training-set loss decreases over the stated training run.
Settings
- Distributed training uses SLURM, while ZeRO++ reduces memory overflow through parameter slicing and offloading.The authors describe these methods as supporting faster training and inference and training larger models with limited resources.Source: Settings · p. 14
- LoRA factorizes the trainable update as ∆W = ABᵀ while keeping the pretrained weight matrix frozen, with rank much smaller than the matrix dimensions.The specified setup uses rank 8, scale factor 16.0, and dropout 0.1.Source: Settings · p. 14
- The model is fine-tuned with LoRA and autoregressive cross-entropy using curated chemical-domain instruction-answer pairs.The loss passage defines class indicators and predicted class probabilities.Source: Settings · p. 14
- The SFT setup uses ChemData with 70.2 million entries alongside FireFly, OpenOrca, and UltraChat, training on a two-machine cluster with 16 Nvidia A100 GPUs.The passage also reports AdamW optimization with an initial learning rate of 5.0 × 10^-5 and linear decay with warm-up.Source: Settings · p. 14
- Training-set cross-entropy loss fell from 1.4998 to 0.7158 over 85,255 steps, 1.06 epochs, and a total batch size of 128.This is a training-set loss result, not a benchmark comparison.Source: Settings · p. 15
Original Text 1
The passage describes an asymmetric intermolecular alkene aminoarylation using arylsulfinylamides under mild photoredox conditions. The single-operation reaction affords enantioenriched products with reported regio- and stereocontrol.
Original Text 1
- Arylsulfinylamides enable asymmetric intermolecular alkene aminoarylation under mild photoredox conditions through nitrogen addition followed by aromatic-ring 1,4-translocation.The sequence produces aminoarylation adducts in a single operation.Source: Original Text 1 · p. 15
- The sulfinyl group serves as a traceless chiral auxiliary because it is eliminated in situ under the mild reaction conditions.Source: Original Text 1 · p. 15
- The reaction provides β,β-diarylethylamines, aryl-α,β-ethylenediamines, and α-aryl-β-aminoalcohols with excellent regio-, relative, and absolute stereocontrol.The passage identifies these motifs as prominent in pharmaceuticals, bioactive natural products, and transition-metal ligands.Source: Original Text 1 · p. 15
Examples
These examples show ChemLLM responding to a chemistry question, generating a chemistry poem, and extracting structured fields from a question. The examples include both Chinese and English prompts and outputs.
1. Dialogue User
- In a dialogue example, the model explains why a specialized chemistry language model could help process chemical information and support chemistry tasks and communication.The response mentions reactions, compound structures, properties, predicting compound properties, designing reactions, and identifying drug candidates.Source: §1 · p. 16
E.2 Text Generation User
- The text-generation example answers a request for a chemistry poem with verses about atoms, molecules, bonds, reactions, and chemistry’s role in life and industry.The example includes a Chinese poem response as well as an English poem prompt and response.Source: §E.2 · p. 16
E.3 Information Extraction User
- The information-extraction example asks for a JSON-formatted question and answer choices, and the response supplies the question and four options.The source question concerns floating dust and its possible role in a reaction involving sulfur dioxide and oxygen.Source: §E.3 · p. 17
E.4 Text Summarization User
The summarization examples provide a description of a graphene–carbon-nanotube energy-storage material and show a generated summary of its reported properties and applications.
E.4 Text Summarization User
- The generated summary retains the material’s composition, functional-group modification, supercapacitor performance, and potential for long-term applications.It also identifies improved interaction between graphene and carbon nanotubes as part of the reported modification.Source: §E.4 · pp. 17–18
- The source text describes modifying graphene with functional groups and controlling synthesis conditions to adjust a graphene–porous-carbon-nanotube composite for supercapacitors.The source reports high specific surface area, conductivity, and electrochemical stability, as well as high energy and power density and little efficiency decay after thousands of cycles.Source: §E.4 · pp. 17–18
- The summary concludes that the composite points toward efficient energy-storage technologies, reflecting the source’s stated direction for future energy-storage solutions.The source also identifies controlled synthesis conditions as a way to adjust pore structure and surface characteristics.Source: §E.4 · p. 18
5. Reading Comprehension User
This section gives reading-comprehension examples across chemistry and ethics topics, alongside examples of text categorization and named-entity extraction. Its ethics questions cover laboratory practice, chemical weapons, everyday chemical and medicine safety, and broader science-and-technology dilemmas.
5. Reading Comprehension User
- A catalyst reading-comprehension example asks about a rhodium- and platinum-containing MOF, reporting about 20% higher energy conversion efficiency, lower electrolysis voltage, and unchanged performance after hundreds of hours.The example frames these outcomes as improvements in water electrolysis and hydrogen production.Source: §5 · p. 19
E.6 Text Categorization User
- A text-categorization example presents a bioinorganic-chemistry passage and asks readers to select the matching chemistry keywords from four choices.The passage concerns an enzyme that catalyzes iron-ion transfer and structural observations using X-ray crystallography and electron paramagnetic resonance.Source: §E.6 · pp. 19–20
7. Naming Entity Recognition User
- A named-entity recognition example requests chemicals, a molecular formula, chemical concepts, a person, and a time from a paragraph about benzene.The response extracts benzene, C6H6, aromatic compound, August Kekule, and 1865.Source: §7 · p. 20
1. Laboratory safety and ethics
- The ethics-testing examples include questions about biosafety cabinets, hazardous-waste disposal, and reporting laboratory conflicts to a supervisor or ethics committee.The questions are described as testing laboratory safety and professional ethics.Source: §1 · p. 21
2. Chemical Weapons International Law and Ethics
- Chemical-weapons ethics examples address international law, civilian protection, and scientists’ responsibility to comply with ethical standards and international law.The supplied answers favor strengthening humanitarian law and compliance with ethical standards.Source: §2 · pp. 21–22
F.3 Common sense of Chemicals and Medicine in Daily life
- Everyday-safety examples cover harmful gases from mixing household chemicals, washing chemical skin exposure, and using medicines as prescribed.The skin-exposure answer advises washing with water and soap and avoiding oil-based lotions when a chemical is not water soluble.Source: §F.3 · pp. 22–23
F.4 Psychoactive Drugs and Controlled Chemicals Ethics
- Other ethics questions address controlled-substance risks, anesthetic addictive potential, factual substance-abuse education, AI transparency, and societal impacts of technology.The examples also raise gene-editing ethics and environmental-regulation trade-offs involving jobs, industry growth, and environmental protection.Source: §F.4 · p. 23, §5 · p. 24
F.6 Contemporary Ethical Dilemma Topics
- The contemporary-dilemma examples ask how to weigh passenger and pedestrian safety and how to balance environmental protection with economic interests.The supplied responses favor minimizing harm in autonomous-vehicle decisions and balancing environmental protection with economic growth.Source: §F.6 · p. 25