ChemCrow: Augmenting large-language models with chemistry tools

1 Introduction

• LLMs struggle with chemistry operations such as converting IUPAC names to molecular graphs, motivating dedicated tools that can provide exact answers.The introduction attributes these shortcomings to models’ next-word prediction design.Source: §1 · p. 1
• Chemistry automation remains limited by its experimental nature, lack of data, and limited computational tools, while steep learning curves and computational-skill demands hinder tool integration.These barriers make it difficult for experimental chemists to exploit existing tools fully.Source: §1 · p. 2
• ChemCrow prompts GPT-4 to use expert-designed chemistry tools for common tasks spanning drug and materials design and synthesis.The prompt specifies the task and desired format and supplies tool names, uses, and expected inputs and outputs.Source: §1 · p. 2
• Its iterative Thought, Action, Action Input, Observation loop lets the LLM select tools, receive their outputs, and continue until it reaches a final answer.The model is guided to consider the task’s current state and goal when planning its next step.Source: §1 · pp. 2–3
• ChemCrow implements 18 tools for literature and web search, molecule and reaction tasks, and direct execution in a physical lab.The toolset is designed to be adapted to new applications by adding a tool and describing its intended use in natural language.Source: §1 · p. 3
• The authors position ChemCrow as an assistant for expert chemists and an accessible interface for non-experts, analyzing its capabilities across 14 use cases.The use cases include target-molecule synthesis, safety controls, and searches for molecules with similar modes of action.Source: §1 · p. 3

2.1 Autonomous chemical synthesis

• ChemCrow planned and autonomously executed four syntheses—DEET and three thiourea organocatalysts—and ActionCleaner iteratively corrected platform-invalid procedures without human intervention.The agent queried synthesis-validation data and adapted procedures, such as by increasing solvent quantity, until they were valid for the RoboRXN platform.Source: §2.1 · p. 3

2.2 Human-AI collaboration

• ChemCrow processed candidate data, trained and evaluated a Random Forest, and proposed a chromophore for 369 nm; synthesis measured 336 nm, confirming a new chromophore near the target.The measured absorption maximum was approximately the desired property, though it differed from the 369 nm target.Source: §2.2 · pp. 3–4, Figure 3 · p. 4

2.3 Evaluation across diverse chemical use cases

• Expert chemists preferred ChemCrow overall, especially on complex tasks, while GPT-4 did better on easier familiar tasks and EvaluatorGPT favored GPT-4 based on fluency.Experts assessed chemistry correctness, reasoning quality, and task completion; close inspection exposed factual errors in GPT-4’s fluent, apparently complete answers.Source: §2.3 · pp. 5–6, Figure 4 · p. 5

Unintended Risks

• ChemCrow checks queried molecules for controlled-chemical status and safety information, stopping execution for controlled chemicals and otherwise providing safety and handling guidance.The guidance can accompany a requested synthesis procedure, as illustrated by the paracetamol example.Source: §3 · p. 6, Figure 5 · p. 7
• Expert-designed tools can mitigate hallucination-related inaccuracy, but limited chemistry understanding remains a risk; proposed remedies include broader training data and built-in validation or peer review.The paper suggests a review system analogous to RLHF for GPT-3.5 (69,70).Source: §3 · p. 6
• The paper flags ownership and potential-infringement questions around generated syntheses, structures, materials, and applications, recommending clearer policies and collaboration with legal experts and industry.The section also calls for open and transparent science to enable broad community oversight and feedback.Source: §3 · pp. 6–7
• Human oversight, safety guidelines, and ethical policies are presented as safeguards, with developers, users, and industry stakeholders urged to remain vigilant as risks evolve.Users can further reduce reliance on flawed reasoning by critically evaluating outputs against established literature and expert opinions.Source: §3 · pp. 6–7

4 Conclusion

• ChemCrow planned and synthesized an insect repellent and three organocatalysts, and guided screening and synthesis of a novel chromophore with target properties.It also independently solved chemistry reasoning tasks from drug-discovery loops to synthesis planning across a wide range of molecular complexity.Source: §4 · pp. 7–8
• ChemCrow’s current results are limited by the amount and quality of its tools and the limited evaluation tasks, which could be expanded and diversified.Language-based and image-processing tools could broaden its capabilities.Source: §4 · p. 8
• Expert chemists found ChemCrow stronger than GPT-4 in chemical factuality, reasoning, and completeness, especially on complex or novel tasks, while GPT-4 may do better on memorized syntheses.LLM-based evaluation tended to favor GPT-4 and may be less reliable than human assessment of chemical reasoning.Source: §4 · p. 8
• Evaluation faces reproducibility challenges with API-based closed-source models, task-selection bias, and difficulty testing chemical logic at scale.Recent open-source models76–78 may improve reproducibility, potentially at the cost of reasoning power; improved experimental design could strengthen validity.Source: §4 · p. 8

5.1 LLMs

• Experiments used GPT-4 at temperature 0.1 in a tool-using framework combining chain-of-thought41 reasoning with tools10, as in ReAct43 and MRKL53.Source: §5.1 · p. 8

5.2 LLMs application framework – LangChain

• LangChain supplied modular application infrastructure and the external-tool integration used in ChemCrow.Source: §5.2 · p. 8

5.3 Tools

• The implementation uses a limited but readily expandable toolset organized into general, molecular, and reaction tools.Source: §5.3 · p. 8

5.3.1 General tools

• General tools provide web and literature search alongside computation and human interaction.LitSearch retrieves relevant document passages and grounds answers with references, and is preferred over WebSearch; WebSearch samples first-page Google results, while Python REPL runs code and Human enables questions or permission prompts.Source: §5.3.1 · p. 9

5.3.2 Molecule tools

• Molecule tools resolve molecular identifiers and support structural comparison, local modification, functional-group analysis, molecular-weight calculation, and purchasability checks.Source: §5.3.2 · pp. 9–10

5.3.3 Safety tools

• Safety tools automatically screen synthesis requests for controlled chemicals and explosives, and provide molecule-level hazard summaries.The summary covers operational, GHS, environmental, and societal risks; when database information is missing, GPT-4 may fill gaps while stating so.Source: §5.3.3 · pp. 10–11

5.3.4 Chemical reaction tools

• Reaction tools classify named reactions, predict products, plan multistep routes, and adapt and launch robotic synthesis after user permission.The planner returns actionable conditions, additives, and solvents, while the execution loop adapts errors and warnings before requesting launch permission.Source: §5.3.4 · p. 11

Synthesis of N,N-Diethyl-m-toluamide (DEET)

• DEET was prepared from diethylamine and 3-methylbenzoyl chloride in DCM at 25°C for 60 minutes, with MS detecting m/z 192.14 [M+H].The reaction was conducted in a nitrogen-inertized stainless-steel reactor; the product was analyzed by HPLC/MS.Source: Synthesis of N,N-Diethyl-m-toluamide… · p. 18

Synthesis of the Schreiners’ catalyst: 1,3-Bis[3,5-bis(trifluoromethyl)phenyl]thiourea

• Schreiners’ catalyst was prepared from bis(trifluoromethyl)phenyl isothiocyanate and aniline in THF at 60°C for 1 hour, with MS detecting m/z 501.02 [M+H].The reaction was conducted in a nitrogen-inertized stainless-steel reactor.Source: Synthesis of the Schreiners’ catalyst:… · p. 18

Synthesis of Takemoto catalyst: 1-(3,5-Bis(trifluoromethyl)phenyl)-3-((1R,2R)-2-(dimethylamino)cyclohexyl)thiourea

• Takemoto catalyst was prepared from trans-N,N-dimethylcyclohexane-1,2-diamine and bis(trifluoromethyl)phenyl isothiocyanate in THF at 25°C for 24 hours.HPLC/MS detected m/z 413.14 [M+H].Source: Synthesis of Takemoto catalyst:… · p. 18

Synthesis of the Ricci’s thiourea catalyst: (1-(3,5-Bis(trifluoromethyl)phenyl)-3-((1R,2S)-2-hydroxy-2,3-dihydro-1H-inden-1-yl)thiourea)

• Ricci’s thiourea catalyst was prepared from amino-indanol and bis(trifluoromethyl)phenyl isothiocyanate in THF, stirred at 25°C for 24 hours.HPLC/MS detected m/z 421.08 [M+H].Source: Synthesis of the Ricci’s thiourea… · pp. 18–19

A.3 Chromophore Synthesis

• Chromophore synthesis step 1 coupled ethenylbenzene and methanesulfonamidophenyl boronic acid in aqueous acetonitrile at 90°C for 2 hours 20 minutes.The isolated vinyl-biphenyl intermediate was characterized by HPLC-MS and 1H and 13C NMR.Source: §A.3 · p. 19
• Chromophore synthesis step 2 refluxed the vinyl-biphenyl intermediate with methyl 4-iodo-3-methyl-benzoate, palladium acetate, and triethylamine for 4 hours.The final methyl (E)-vinyl benzoate product was characterized by HPLC-MS and 1H and 13C NMR.Source: §A.3 · p. 19

• Four expert chemists assessed model answers for every task using an evaluation sheet, with molecular structures and reactions rendered as graph depictions using OPSIN.Some GPT-4 responses required substantial human interpretation when preparing the sheet.Source: B Human evaluation · p. 20
• Evaluation was complicated by inconsistencies between IUPAC names and verbal transformation descriptions, as well as cases admitting multiple conclusions.These issues complicated interpretation of the outputs.Source: B Human evaluation · p. 20
• Responses were anonymized by randomly shuffling model order and summarizing ChemCrow outputs to mask its characteristic ReAct style.The summarization layer made ChemCrow’s answers more readable and more assistant-like, making them harder to distinguish from GPT-4 by style.Source: B Human evaluation · p. 20

C Synthesis example: GPT-4 vs ChemCrow

• The comparison concerns Takemoto’s organocatalyst, which enables enantioselective Michael reactions of malonates to nitroolefins59.The complete task is shown in Appendix G.14.Source: C Synthesis example: GPT-4 vs ChemCrow · p. 21
• ChemCrow’s single-step synthesis was highly rated by human reviewers, whereas GPT-4’s flawed plan diverged from the target despite receiving a higher EvaluatorGPT grade.Figure 6 contrasts ChemCrow’s plan with GPT-4’s unnecessary steps; EvaluatorGPT favored GPT-4 for its apparent attention to stereochemistry and protecting groups.Source: Figure 6 · p. 21, C Synthesis example: GPT-4 vs ChemCrow · p. 21
• ChemCrow proposes a simple disconnection to an isothiocyanate and chiral substituted cyclohexane, with solvent, temperature, and reaction-time conditions.These components form the desired thiourea.Source: C Synthesis example: GPT-4 vs ChemCrow · p. 21
• GPT-4’s route includes unnecessary protection/deprotection and condensation steps, risks the chiral center, and gives IUPAC-named molecules that may not match its described reactions.Its proposed reactions also require substantial human interpretation.Source: C Synthesis example: GPT-4 vs ChemCrow · p. 21
• The example highlights that LLM-powered evaluation can favor confident, fluent responses over accurate synthetic reasoning, so human evaluation remains necessary in fact-critical chemistry.The passage identifies this as a limitation of LLM-powered evaluation for synthetic chemistry.Source: C Synthesis example: GPT-4 vs ChemCrow · p. 21

D Safety Workflow

• The Safety Workflow figure indicates that its shown steps are followed every time the agent receives a prompt.Source: Figure 7 · p. 22

E Reproducibility

• Five independent executions of Task 6 assessed reproducibility on a chemistry reasoning task requiring multiple tools and literature-search analysis.The task asks for products with Lindlar’s catalyst and with [Pd], then a comparison of the products and reaction mechanisms.Source: E Reproducibility · p. 23
• The authors suggest advanced text–molecule multimodal models for molecular captioning, noting that further research is needed despite recent approaches [103,104].The issue arose in interpreting molecular structure, rather than obtaining the products.Source: E Reproducibility · p. 23
• In 2 of 5 runs, ChemCrow called “CCc1ccc(Cl)cc1” a trans-alkene, causing incorrect conclusions about reaction-mechanism differences.Across the executions, it systematically obtained the correct products in both catalyst cases using appropriate tools, but interpretation varied.Source: Figure 8 · p. 23, E Reproducibility · p. 23

F Limitations

• ChemCrow’s key limitations for dependable chemistry workflows are hallucination, difficulty evaluating results, and lack of reproducibility.These limitations are also discussed in recent publications on LLM applications.Source: F Limitations · p. 24
• Chemical tools improve LLM factual correctness and decision-making, but cannot fully correct faulty reasoning, which still causes occasional errors.The authors report that tool addition improves the reasoning process without eliminating flaws in the model’s reasoning.Source: F Limitations · p. 24
• LLM-based evaluators lack the knowledge to detect errors and favor verbose, fluent-looking solutions, making them insufficient for dependable assessment of ChemCrow.This leads the authors to rely heavily on human evaluation, limiting the pace and scale of performance measurement.Source: F Limitations · p. 24
• Designing experiments to show the strengths and weaknesses of LLM-based tools is field-specific and requires substantial expert oversight.This challenge further constrains the assessment of ChemCrow’s performance and reliable use across distinct tasks.Source: F Limitations · p. 24
• ChemCrow’s effectiveness depends on tool quality and quantity, and it should not be expected to outperform the retrosynthesis tools it uses.The authors identify potential improvements through different tools, stronger chemistry-specific machine learning, more chemistry-capable LLMs, and progress in agent architecture.Source: F Limitations · p. 24

G Tasks and Evaluation

• The evaluation presents 14 ChemCrow tasks and execution results alongside GPT-4 results and assessments from an evaluator LLM and a human-expert committee.Source: G Tasks and Evaluation · p. 24

G.1 Task 1 - Synthesis for Safinamide

• Figure 9 reports both expert-evaluator average and EvaluatorGPT scores, after a separate GPT-4 instance evaluates outputs from GPT-4 and ChemCrow.The prompt is given to both systems; the figure also shows ChemCrow’s general workflow and first Chain of Thought step.Source: Figure 9 · p. 25

G.2 Task 2 - Propose New Organocatalyst; G.3 Task 3 - Explain Mechanisms; G.4 Task 4 - Synthesize Insect Repellent

• Figures 10–12 compare GPT-4 and ChemCrow on tasks 2–4, showing workflows and first Chain of Thought steps; both systems receive the prompt, and expert-average and EvaluatorGPT scores are reported.Source: Figure 10 · p. 26, Figure 11 · p. 27, Figure 12 · p. 28

G.5 Task 5 - How to Make Target; G.6 Task 6 - Compare Catalyst Mechanisms; G.7 Task 7 - Synthesize Similar Molecule

• Figures 13–15 report GPT-4 and ChemCrow performance on tasks 5–7: both receive the prompt, a separate GPT-4 instance evaluates outputs, and workflows, first Chain of Thought steps, expert-evaluator averages and EvaluatorGPT scores are reported.Source: Figure 13 · p. 29, Figure 14 · p. 30, Figure 15 · p. 31

G.8 Task 8 - Synthesis Planning of Riccis’s Organocatalyst; G.9 Task 9 - Predict Success of Reaction; G.10 Task 10 - Property of Reaction Product

• For tasks 8–10, figures report GPT-4 and ChemCrow performance: both receive the prompt, a separate GPT-4 instance evaluates outputs, and results include expert-evaluator averages and EvaluatorGPT scores.Source: Figure 16 · p. 32, Figure 17 · p. 33, Figure 18 · p. 34

G.11 Task 11 - Similar Molecule to Nitroglycerin

• For task 11, ChemCrow checks whether the molecule is dangerous before synthesis planning; after finding it explosive, execution stops with a warning, and no scores are provided.Figure 20 reports results for GPT-4 and ChemCrow, including expert-evaluator average and EvaluatorGPT scores.Source: Figure 19 · p. 35, Figure 20 · p. 36

G.12 Task 12 - Synthesis and Cost of Atorvastatin; G.13 Task 13 - Synthesis of Aspirin

• Figures 20 and 21 compare GPT-4 and ChemCrow on tasks 11 and 13, showing prompts, ChemCrow workflows and first Chain of Thought steps, and reporting expert-evaluator averages and EvaluatorGPT scores.Source: Figure 20 · p. 36, Figure 21 · p. 37

G.14 Task 14 - Synthesis of Takemoto’s Organocatalyst

• Figure 22 reports GPT-4 and ChemCrow performance on task 14: both receive a prompt, a separate GPT-4 instance evaluates their outputs, and results include average expert-evaluator and EvaluatorGPT scores.Source: Figure 22 · p. 38