Source-linked AI summary
ChemCrow: Augmenting large-language models with chemistry tools
Andres M Bran, Sam Cox, Oliver Schilter, Carlo Baldassari, Andrew D White, Philippe Schwaller
TL;DR
LLMs struggle with chemistry tasks and lack external knowledge access. ChemCrow integrates expert-designed chemistry tools with an LLM to automate diverse chemical tasks, and expert evaluation found stronger chemical factuality, reasoning, and completeness than GPT-4, especially on novel tasks.
Problem
LLMs often struggle with chemistry tasks, while limited access to external knowledge restricts their usefulness in scientific applications.
Method
ChemCrow integrates 18 expert-designed chemistry tools with an LLM agent to solve tasks spanning synthesis, drug discovery, and materials design.
Results
Expert chemists found ChemCrow outperformed GPT-4 in chemical factuality, reasoning, and response completeness, particularly on increasingly complex and novel tasks.
Takeaways & Limitations
ChemCrow can independently solve diverse chemistry reasoning tasks and serve as a potential assistant for chemical laboratories.
Takeaways & Limitations
LLM-based evaluation may be unreliable for judging chemical reasoning when factuality is important because evaluators may lack sufficient chemistry understanding.
Abstract
from arXiv · showhide
Over the last decades, excellent computational chemistry tools have been developed. Integrating them into a single platform with enhanced accessibility could help reaching their full potential by overcoming steep learning curves. Recently, large-language models (LLMs) have shown strong performance in tasks across domains, but struggle with chemistry-related problems. Moreover, these models lack access to external knowledge sources, limiting their usefulness in scientific applications. In this study, we introduce ChemCrow, an LLM chemistry agent designed to accomplish tasks across organic synthesis, drug discovery, and materials design. By integrating 18 expert-designed tools, ChemCrow augments the LLM performance in chemistry, and new capabilities emerge. Our agent autonomously planned and executed the syntheses of an insect repellent, three organocatalysts, and guided the discovery of a novel chromophore. Our evaluation, including both LLM and expert assessments, demonstrates ChemCrow's effectiveness in automating a diverse set of chemical tasks. Surprisingly, we find that GPT-4 as an evaluator cannot distinguish between clearly wrong GPT-4 completions and Chemcrow's performance. Our work not only aids expert chemists and lowers barriers for non-experts, but also fosters scientific advancement by bridging the gap between experimental and computational chemistry.
1 Introduction
ChemCrow is proposed as an LLM-powered chemistry engine that integrates expert-designed tools to streamline chemical reasoning across synthesis, drug design, and materials design. Its iterative Thought–Action–Action Input–Observation workflow enables the model to use tool outputs while solving tasks.
- Motivation: Chemistry already uses expert-designed AI systems for reaction prediction, retrosynthesis, molecular properties, molecular generation, materials design, and Bayesian optimization.
- Motivation: Integrating chemistry tools across isolated environments remains difficult for experimental chemists because interoperability requires computational skills and spans diverse tools.
- ChemCrow: ChemCrow is an LLM-powered chemistry engine that combines multiple expert-designed tools for common tasks in chemical synthesis, drug design, and materials design.
- Workflow: The agent repeatedly reasons about the task, selects a tool and input, observes the returned result, and continues until producing a final answer.This workflow follows the Thought, Action, Action Input, Observation format, with generation pausing during tool execution.
- Workflow: The tool-using loop combines chain-of-thought reasoning with task-relevant tools, shifting the LLM from an often-wrong information source toward a reasoning engine.
2 Results & Discussion
ChemCrow integrated chemistry tools to plan and execute several syntheses autonomously and supported human–model discovery of a novel chromophore. Expert and model evaluations found stronger performance than tool-less GPT-4 on complex chemistry tasks, while revealing important limitations in automated evaluation.
- Autonomous synthesis: ChemCrow autonomously planned and executed DEET and three known thiourea organocatalyst syntheses on IBM Research’s cloud-connected RoboRXN platform.It sequentially queried LitSearch/WebSearch, Name2SMILES, ReactionPlanner, and ReactionExecute.
- Autonomous synthesis: ChemCrow queried synthesis-validation data and iteratively adapted procedures to address execution problems such as insufficient solvent or invalid purification actions.Predicted procedures were not always directly executable on RoboRXN, and fixing invalid actions otherwise required human interaction.
- Chromophore discovery: A human–ChemCrow interaction led to a synthesized and analyzed novel chromophore with an absorption maximum wavelength of 336nm.The result approximately matched the desired property.
- Evaluation: The authors collaborated with expert chemists to develop an evaluation framework because few benchmarks assessed chemistry-specific LLM tasks and no standardized technique had yet been established.The rapid pace of progress made assessing the approach challenging.
- Evaluation: ChemCrow outperformed the tool-less LLM, especially on complex tasks requiring more grounded chemical reasoning, whereas GPT-4 performed better only on easier tasks with clear objectives and training-data coverage.GPT-4’s fluent, complete style could mask systematic factual inaccuracies that thorough inspection revealed.
- Evaluation: Humans preferred and highly scored ChemCrow across all three proposed metrics, but EvaluatorGPT generally preferred GPT-4 based on fluency and apparent completeness.This difference shows that LLM-powered evaluation can fail when the evaluator lacks the understanding required by the prompt.
3 Risk Mitigation Strategies
ChemCrow’s responsible use requires human oversight, safety safeguards, critical evaluation, and policies addressing intellectual property. Expert-designed tools can reduce hallucination-related inaccuracies, but risks from hazardous recommendations and insufficient chemistry knowledge remain.
- Oversight: Human operators and chemist experts must heavily review and control automated chemical platforms to support responsible use.This oversight is emphasized even as LLM-driven chemistry engines make expert-designed tools more accessible to non-experts.
- Unintended Risks: Expert-designed tools can mitigate LLM hallucinations and reduce inaccurate reasoning, but insufficient chemistry knowledge may still cause flawed decisions or problematic results.Users should critically evaluate outputs and cross-reference established literature and expert opinions.
- Intellectual Property: Clear guidelines and policies are needed for ownership, predicted applications, and potential infringement involving generated chemical syntheses, structures, and materials.Legal experts and industry stakeholders can help navigate intellectual-property issues and implement protective measures.
- Unintended Risks: Safety instructions require checking safety information before experiments proceed, while controlled-chemical inputs trigger warnings and execution stops.ChemCrow’s example allows a paracetamol synthesis to proceed with general laboratory safety information but stops for controlled chemicals.
- Summary: Mitigation strategies include access controls, safety guidelines, expert-designed tools, and improved training data to support safer and more effective chemistry engines.The paper advocates foresight, safeguards, open and transparent science, and broad research-community oversight.
4 Conclusion
ChemCrow integrates LLM reasoning with computational chemistry tools to plan and execute diverse chemical tasks, including syntheses and chromophore discovery. Expert evaluations found stronger factuality, reasoning, and completeness than GPT-4 on increasingly complex and novel tasks, while tool limitations and evaluation challenges remain.
- Contributions: ChemCrow combined LLM reasoning with computational tools to plan and synthesize an insect repellent and three organocatalysts, and guide screening and synthesis of a novel chromophore.The chromophore was targeted for specified properties.
- Capabilities: ChemCrow independently solved chemistry reasoning tasks ranging from simple drug-discovery loops to synthesis planning across a wide range of molecular complexity.The authors describe this capability as indicating potential for a future chemical assistant.
- Limitations and future work: ChemCrow’s results are limited by the amount and quality of its selected tools, while broader language-based, image-processing, and other tools could expand its capabilities.The selected evaluation tasks are also limited, motivating further research to expand and diversify them.
- Evaluation: Expert chemists found that ChemCrow outperformed GPT-4 in chemical factuality, reasoning, and response completeness, particularly as task complexity increased.GPT-4 could perform better on memorization tasks involving well-known molecules, whereas ChemCrow excelled on novel or less-known tasks.
- Limitations and future work: Evaluation validity is challenged by limited reproducibility under API-based closed-source models, although open-source models may improve reproducibility at a possible cost in reasoning power.The passage also identifies improved experimental design as a way to enhance validity.
5 Methods
ChemCrow is implemented with LangChain and combines general, molecular, and chemical-reaction tools to support information retrieval, computation, human oversight, and molecular analysis. Its toolset can be expanded according to task needs and availability.
- Framework: LangChain provides ChemCrow with modules for language models, prompts, document loading, chains, indexes, agents, memory, and chat functionality.These modules support the development of language-model applications, including question answering, summarization, and data-augmented generation.
- Toolset: The tools are classified into general, molecular, and chemical reaction tools, and the limited implementation can be expanded as needed.The passages explicitly note that tool availability and task requirements can determine future expansion.
- General tools: General tools provide web and literature search, Python execution, and direct human interaction for information access, computation, and user control.WebSearch queries Google results through SerpAPI; LitSearch embeds documents with OpenAI Embeddings and FAISS, while Human can request permission for actions.
- General tools: The literature-search workflow retrieves the top k document passages, summarizes them relative to a query, and incorporates those summaries into the language-model prompt.This process is intended to anchor generated answers in existing scientific literature.
- Molecular tools: Molecular tools convert names and representations, assess purchasability and similarity, modify molecules, check patents, identify functional groups, and calculate molecular weight.Specific implementations use chem-space, PubChem, ZINC-related resources, Tanimoto similarity with ECFP2 fingerprints, SynSpace, SMARTS patterns, and RDKit.
A Experimental procedures … A.3 Chromophore Synthesis
The experimental procedures describe reactor-based syntheses of DEET, three thiourea catalysts, and a chromophore through specified reagent additions, reaction conditions, workups, and analytical checks.
- A.1 Insect repellent: DEET synthesis used diethylamine and 3-methylbenzoyl chloride in DCM at 25°C for 60 minutes, followed by aqueous/organic extraction and sample analysis.The reactor was inertized by three vacuum/nitrogen flushes before reaction.
- A.2 Thiourea catalysts: The Schreiners’ catalyst synthesis combined 3,5-bis(trifluoromethyl)phenyl isothiocyanate and aniline in THF at 60°C for one hour before HPLC/MS analysis.The reaction mixture was diluted with 14.3 ml of THF, and the analytical sample was diluted tenfold with acetonitrile.
- Synthesis of the Schreiners’ catalyst: 1,3-Bis[3,5-bis(trifluoromethyl)phenyl]thiourea: The Schreiners’ catalyst was prepared by reacting 3,5-bis(trifluoromethyl)phenyl isothiocyanate with 3,5-bis(trifluoromethyl)aniline.Both solutions were added to an inertized stainless-steel reactor before stirring and HPLC/MS sampling.
- Synthesis of Takemoto catalyst: 1-(3,5-Bis(trifluoromethyl)phenyl)-3-((1R,2R)-2-(dimethylamino)cyclohexyl)thiourea: Takemoto catalyst synthesis reacted trans-N,N-dimethylcyclohexane-1,2-diamine with 3,5-bis(trifluoromethyl)phenyl isothiocyanate in THF for 24 hours at 25°C.The reaction used 3mmol diamine and 5mmol isothiocyanate and was sampled for HPLC/MS after tenfold dilution.
- A.3 Chromophore Synthesis: Chromophore synthesis Step 1 coupled 1-bromo-4-ethenylbenzene with [3-(methanesulfonamido)phenyl]boronic acid using SPhosPd G2 and sodium carbonate in aqueous acetonitrile at 90°C.The mixture reacted for 2h20min, then was cooled, diluted with water, and extracted twice with ethyl acetate.
- A.3 Chromophore Synthesis: Chromophore synthesis Step 2 used N-(4’-vinyl-[1,1’-biphenyl]-3-yl)methanesulfonamide, methyl 4-iodo-3-methyl-benzoate, palladium acetate, and triethylamine in acetonitrile under reflux.The mixture was refluxed for 4h before cooling and ethyl-acetate extraction.
B Human evaluation · C Synthesis example: GPT-4 vs ChemCrow
Human evaluation used four expert chemists and anonymized, standardized model outputs, though inconsistent chemical descriptions complicated interpretation. A Takemoto organocatalyst synthesis example compared GPT-4 and ChemCrow, highlighting ChemCrow’s tool-assisted approach to hallucination.
- B Human evaluation: Four expert chemists assessed model performance using an evaluation sheet containing each model’s answers for every task.Molecular structures and reactions in IUPAC or SMILES notation were converted into graph depictions with OPSIN.
- B Human evaluation: The evaluation materials converted molecular structures and reactions into graph depictions using the open-source IUPAC parsing tool OPSIN.This preprocessing was part of preparing the assessment sheet for expert review.
- B Human evaluation: Evaluation was complicated by inconsistencies between substances’ IUPAC names and verbal descriptions of the transformations leading to them.Some cases supported multiple plausible conclusions, further complicating interpretation.
- B Human evaluation: Potential evaluation bias was reduced by randomly shuffling model presentation order and masking ChemCrow’s characteristic ReAct style through an additional summarization layer.The summarization converted ChemCrow’s output into a more readable, assistant-like solution.
- C Synthesis example: GPT-4 vs ChemCrow: GPT-4 and ChemCrow were compared on synthesizing Takemoto’s organocatalyst, a bifunctional catalyst enabling enantioselective Michael reactions of malonates to nitroolefins.The complete synthesis task appears in Appendix G.14.
- C Synthesis example: GPT-4 vs ChemCrow: The Takemoto synthesis example illustrates ChemCrow’s aim of tackling LLM hallucination by adding expert chemistry tools.Figure 6 presents the GPT-4 and ChemCrow synthesis outputs for this task.
Human evaluation · D Safety Workflow
Human reviewers rated ChemCrow’s concise, condition-rich synthesis plan highly, while GPT-4 proposed a flawed and unnecessarily long route. The comparison exposes limitations of fluent LLM-based evaluation and underscores the need for human assessment in fact-critical chemistry; the agent’s workflow follows fixed steps for every prompt.
- Human evaluation: ChemCrow’s single-step synthesis was highly rated by human reviewers, whereas GPT-4’s synthetic plan did not lead to the target.Figure 6 contrasts ChemCrow’s experimental conditions and quantities with GPT-4’s flawed plan and additional unnecessary steps.
- Human evaluation: ChemCrow proposed a simple disconnection yielding an isothiocyanate and chiral substituted cyclohexane for the desired thiourea.The plan included solvent, temperature, and reaction-time conditions.
- Human evaluation: GPT-4 proposed unnecessary protection/deprotection sequences and condensations that caused the synthetic route to diverge from the target.Its proposed disconnection also potentially introduced risk, according to the passage.
- Human evaluation: LLM-powered evaluation can favor confidence and fluency over reasoning quality and solution accuracy in synthetic chemistry.The passage identifies this as a clear limitation of such evaluation.
- Human evaluation: Human evaluation remains necessary for assessing chemistry systems, especially in a fact-critical field.This conclusion follows from the discrepancy between fluent evaluation and synthetic-chemistry quality.
- D Safety Workflow: The agent follows the same steps every time it receives a prompt.Figure 7 presents these recurring workflow steps, without specifying them in the supplied passage.
E Reproducibility
ChemCrow’s reproducibility was assessed through five independent executions of a chemistry reasoning task requiring multiple tools and literature analysis. The task involved predicting and comparing products formed with Lindlar’s catalyst versus [Pd].
- Assessment design: Five independent ChemCrow executions were carried out to assess reproducibility on chemistry reasoning tasks.The study identifies reproducibility as a salient concern when closed-source LLMs play key roles in scientific workflows.
- Assessment task: The evaluated task required predicting products for 1-Chloro-4-ethynylbenzene with Lindlar’s catalyst and with [Pd], then comparing products and mechanisms.The Lindlar catalyst was specified as CC(=O)[O-].CC(=O)[O-].[Pd].[Pb+2].
- Assessment criteria: Reproducibility was assessed by the depth and informativeness of responses and by their agreement across executions.Solving the task required querying multiple tools, gathering information, and analyzing literature-search results before formulating a final answer.
- Results: Figure 8 summarizes five outputs from separate ChemCrow instances on the same task, including products and comparisons.The outputs were summarized for clarity.
F Limitations · G Tasks and Evaluation
ChemCrow improves chemical LLM performance but remains limited by hallucination, flawed reasoning, evaluation challenges, reproducibility, and the quality and quantity of its tools. Its evaluation covered 14 tasks, comparing ChemCrow with GPT-4 and assessments from evaluator LLMs and human experts.
- F Limitations: ChemCrow’s major limitations are hallucination, difficult evaluation, and poor reproducibility, hindering dependable integration into routine chemistry workflows.
- F Limitations: Chemical tools improve the LLM’s factual correctness and decision-making, but they cannot fully correct errors caused by faulty reasoning.
- F Limitations: Existing LLM-based evaluation methods cannot reliably assess ChemCrow because they may miss errors and favor verbose, fluent-looking solutions.
- F Limitations: ChemCrow’s effectiveness is constrained by the quality and quantity of its integrated tools.
- F Limitations: Open-source retrosynthesis software remains limited by its data, algorithms, and evaluation challenges, restricting the performance ChemCrow can achieve with those tools.
- G Tasks and Evaluation: 14 tasks were used to evaluate ChemCrow’s execution results alongside plain GPT-4 results on the same tasks.
- G Tasks and Evaluation: The evaluation incorporated judgments from both an evaluator LLM and a committee of human experts.
G.1 Task 1 - Synthesis for Safinamide
Figure 9 compares GPT-4 and ChemCrow on the safinamide-synthesis task using independent GPT-4 evaluation alongside expert-evaluator scores.
- Evaluation workflow: Both ChemCrow and GPT-4 receive the same prompt for task 1.The figure reports their performance on synthesis for safinamide.
- Evaluation workflow: A separate GPT-4 instance evaluates the outputs from ChemCrow and GPT-4.This separates generation from the EvaluatorGPT assessment.
- Agent process: Figure 9 presents ChemCrow’s general workflow and its first Chain of Thought step.These materials accompany the performance comparison.
- Evaluation results: Results include both average expert-evaluator scores and EvaluatorGPT scores.The passage does not provide the numerical score values.
G.2 Task 2 - Propose New Organocatalyst … G.14 Task 14 - Synthesis of Takemoto’s Organocatalyst
Tasks 2–14 evaluate GPT-4 and ChemCrow across diverse chemistry problems using separate GPT-4 and expert evaluators. The reported workflows also include a safety intervention that halts execution when an explosive molecule is identified.
- Evaluation workflow: Across the task figures, ChemCrow’s general workflow and first Chain of Thought step are presented with comparative evaluator results for GPT-4 and ChemCrow.This repeated format spans organocatalyst proposals, mechanism explanations, molecule synthesis, reaction prediction, product properties, and named-compound synthesis.
- G.11: Task 11 evaluates finding a molecule similar to nitroglycerin, but ChemCrow stops execution after identifying the molecule as explosive and provides no scores.The workflow checks whether the molecule is dangerous before continuing with synthesis planning.
- G.13: Task 13 compares GPT-4 and ChemCrow on aspirin synthesis using the same expert-evaluator and EvaluatorGPT assessment framework.The figure includes ChemCrow’s general workflow and first Chain of Thought step.
- G.14: Task 14 evaluates GPT-4 and ChemCrow on synthesis of Takemoto’s organocatalyst through separate GPT-4 evaluation and expert scoring.The reported results include expert-evaluator averages and EvaluatorGPT scores.