Source-linked AI summary
ChemToolAgent: The Impact of Tools on Language Agents for Chemistry Problem Solving
Botao Yu, Frazier N. Baker, Ziru Chen, Garrett Herb, Boyu Gou, Daniel Adu-Ampratwum, Xia Ning, Huan Sun
TL;DR
Existing evaluations of chemistry agents covered narrow task sets, leaving their performance across diverse chemistry problems unclear. ChemToolAgent broadens the evaluation with an enhanced tool set; results show that its benefits vary by task, and it does not consistently outperform base LLMs without tools.
Problem
Earlier evaluations of tool-augmented chemistry agents covered narrow task sets, leaving their performance across diverse chemistry tasks unclear.
Source: §1 · p. 1, A Related Work · p. 9Method
Results
Takeaways & Limitations
The findings indicate that tool benefits depend on task characteristics: specialized tasks can benefit from task-specific tools, while general chemistry questions require sound chemistry reasoning.
Source: §1 · p. 2The evaluated chemistry tasks may not represent the full range of real-world chemistry problems.
Source: Limitations · p. 5
Abstract
from arXiv · showhide
To enhance large language models (LLMs) for chemistry problem solving, several LLM-based agents augmented with tools have been proposed, such as ChemCrow and Coscientist. However, their evaluations are narrow in scope, leaving a large gap in understanding the benefits of tools across diverse chemistry tasks. To bridge this gap, we develop ChemToolAgent, an enhanced chemistry agent over ChemCrow, and conduct a comprehensive evaluation of its performance on both specialized chemistry tasks and general chemistry questions. Surprisingly, ChemToolAgent does not consistently outperform its base LLMs without tools. Our error analysis with a chemistry expert suggests that: For specialized chemistry tasks, such as synthesis prediction, we should augment agents with specialized tools; however, for general chemistry questions like those in exams, agents' ability to reason correctly with chemistry knowledge matters more, and tool augmentation does not always help.
1 Introduction
Existing tool-augmented chemistry agents had been evaluated on few, narrow tasks, leaving their performance across diverse chemistry problems unclear. ChemToolAgent broadens that evaluation and shows that tool benefits depend on task type.
1 Introduction
- Earlier evaluations of ChemCrow and Coscientist covered only 14 and six specific tasks, respectively, leaving performance across diverse chemistry tasks poorly understood.Source: §1 · p. 1
- ChemToolAgent uses ReAct and 29 tools, and the study evaluates it on specialized tasks and general chemistry questions.The evaluation uses SMolInstruct for specialized tasks and MMLU-Chemistry, SciBench-Chemistry, and GPQA-Chemistry for general questions.Source: §1 · p. 1
- CTA improves on specialized chemistry tasks but does not consistently outperform its base LLMs without tools, and it underperforms them on general chemistry questions.The authors report that specialized tools can help with molecular representations and operations, while general questions require fundamental knowledge and extensive reasoning.Source: §1 · pp. 1–2
- The authors conclude that tool augmentation may add complexity that hinders reasoning, so it does not always help with chemistry problem-solving.Their analysis attributes general-question underperformance mainly to intermediate-stage mistakes, including faulty logic and information oversight.Source: §1 · p. 2
2 ChemToolAgent
ChemToolAgent extends ChemCrow with reasoning and tool-grounding abilities and an expanded tool set for chemistry problem-solving. Its workflow repeatedly plans, selects a tool and input, observes the result, and returns a final answer when done.
2 ChemToolAgent
- CTA frames chemistry problem-solving around reasoning during thought generation and grounding when selecting a tool and its input.Source: §2 · p. 2
- The agent follows an iterative ReAct cycle of thought generation, action determination, and observation, continuing until task completion or conclusion.Source: Figure 1 · p. 2
- CTA expands ChemCrow’s tools, adding 16 new tools and enhancing six existing ones for broader chemistry problem-solving capabilities.Examples include PubChem information retrieval, molecular property predictors, and an enhanced web-search service.Source: §2 · p. 2
3 Experiments
The experiments compare tool-augmented agents with base LLMs across specialized chemistry tasks and general chemistry questions. CTA’s advantage depends on the task category, while its error patterns differ between those categories.
3.1 Experimental Setup
- The evaluation covers 14 molecule- and reaction-centric task types and general chemistry questions ranging from high-school and college to graduate level.The general-question datasets are MMLU-Chemistry, SciBench-Chemistry, and GPQA-Chemistry.Source: §3.1 · pp. 2–3
3.2 Overall Performance
- CTA improves over base LLMs on specialized SMolInstruct tasks but underperforms base LLMs on most general-question cases, despite outperforming ChemCrow.The reported pattern spans the general-question datasets and contrasts with the specialized-task results.Source: §3.2 · pp. 3–4, Table 2 · p. 3, Table 3 · p. 3
3.3 Error Analysis
- SMolInstruct errors were over 99.0% tool errors, whereas MMLU-Chemistry errors were over 90.0% reasoning errors.The authors attribute many specialized-task tool errors to imperfect neural-network tools; general questions demand broader knowledge and more intricate reasoning.Source: §3.3 · p. 4, Figure 2 · p. 5
- The authors hypothesize that role-switching and complex contexts increase cognitive load, while inaccurate or conflicting tool outputs may also confuse the agent.Source: §3.3 · p. 4
4 Conclusion
ChemToolAgent improves performance on specialized chemistry tasks but does not surpass base LLMs without tools on general questions. Error analysis finds tool errors predominate in specialized tasks and reasoning errors in general questions.
4 Conclusion
- ChemToolAgent improves specialized-task performance but does not surpass base LLMs without tools on general chemistry questions.The paper recommends focusing future general-question agent design on cognitive load, reasoning, and verifying information across inconsistent sources.Source: §4 · p. 5
- Tool errors predominate in specialized tasks, whereas reasoning errors are more frequent in general questions.Source: §4 · p. 5
- The authors recommend optimizing LLM cognitive load and improving reasoning and information verification for general chemistry questions.Source: §4 · p. 5
Limitations
The evaluation is limited by its model selection, task coverage, and small-scale manual error analysis. The related-work discussion identifies a gap in comprehensive evaluations, while the tools span general, molecule, and reaction tasks.
Limitations
- The evaluation uses GPT-4o and Claude-3.5-Sonnet, omits broader LLM coverage, and may not represent the range of real-world chemistry problems.Manual error analysis by a chemistry expert and two chemistry-knowledgeable PhD students is limited in scale and may miss error variations or introduce bias.Source: Limitations · p. 5
A Related Work
- Prior chemistry agents address varied applications, but the field lacks comprehensive evaluation of tool-augmented agents across chemistry problems.Source: A Related Work · p. 9
B.1 Tools
- The tool set groups broad retrieval and computation, molecular analysis and prediction, and reaction prediction or reactant suggestion.Examples include BBBPPredictor, ForwardSynthesis, and Retrosynthesis.Source: §B.1 · pp. 9–10
B.2 Improvements of Tool Set
The improved tool set is associated with ChemToolAgent consistently and substantially outperforming ChemCrow using its original tools. The changes address broken tools, usability problems, and missing chemistry functionality.
B.2 Improvements of Tool Set
- The improvements repaired non-functional tools and made tools more practical, including replacing inflexible, truncating web search with an LLM-based summarized search service.Source: §B.2 · pp. 10–11
- The expanded tool set adds PubChem information access and chemical-representation conversions that the original set lacked.Source: §B.2 · p. 11
- ChemToolAgent consistently and substantially outperforms ChemCrow using the original tool set.Source: §B.2 · p. 11
C.1.1 Dataset and Evaluation Setups
The evaluation uses SMolInstruct, a dataset of molecule- and reaction-centric tasks, and compares models on sampled test examples. The detailed results also include non-LLM and fine-tuned baselines, with prompting and answer formatting specified for other models.
C.1.1 Dataset and Evaluation Setups
- SMolInstruct contains 14 molecule- and reaction-centric tasks, illustrated with task names, abbreviations, and examples.Source: §C.1.1 · p. 11, Figure C.1 · p. 11
- The evaluation samples 50 test-set examples per task and includes SoTA non-LLM models and LlaSMol, using their own input and output formats.Source: §C.1.1 · p. 11
- Other models receive step-by-step prompts and must wrap final answers in answer tags; evaluation metrics follow Yu et al. (2024).Source: §C.1.1 · p. 12
C.1.2 Detailed Results
On SMolInstruct, ChemToolAgent performs best or near-best across tasks, with comparable performance from Claude and GPT backbones attributed to the tasks’ emphasis on tool use. The appendix also describes leakage checks and the setup for MMLU-Chemistry evaluation.
C.1.2 Detailed Results
- ChemToolAgent achieves the best performance or close on all SMolInstruct tasks, while Claude and GPT backbones are comparable because the tasks emphasize tool use.The reported results cover name conversion, property prediction, and MC, MG, FS, and RS tasks.Source: Table C.1 · p. 12, Table C.2 · p. 12, §C.1.2 · p. 12
C.1.3 Potential Data Leakage via Tools
- Self-trained tools excluded test examples from their training data, while possible leakage through IBM-backed synthesis APIs remains uncertain.The paper reports that retrained Molecular Transformer results were consistent with ChemToolAgent’s; this suggests any leakage, if present, is minimal.Source: §C.1.3 · pp. 12–13
C.2.1 Dataset and Evaluation Setups
- MMLU-Chemistry contains 70 manually reviewed questions, split evenly between high-school and college levels, selected from MMLU-Pro and MMLU-Redux.Samples answered correctly by all models in preliminary experiments were removed.Source: §C.2.1 · p. 13
- Models generated chain-of-thought solutions and were run three times per sample, with average accuracy reported.The evaluation also compares zero-shot and five-shot settings, using manually constructed demonstrations for base LLMs and step-wise demonstrations for ChemToolAgent.Source: §C.2.1 · p. 13
C.2.2 Detailed Results
On MMLU-Chemistry, ChemToolAgent leads only in the Claude five-shot configuration and trails base models in the others. The section also describes the SciBench-Chemistry evaluation and its comparison setup for ChemAgent.
C.2.2 Detailed Results
- ChemToolAgent has the highest overall performance only with Claude in the five-shot setting, and underperforms base LLMs in every other configuration across both school levels.The same pattern is reported for ChemCrow; five-shot examples yield minimal improvement for base LLMs but significantly enhance ChemToolAgent.Source: Table C.3 · p. 13, §C.2.2 · pp. 13–14
C.3.1 Dataset and Evaluation Setups
- SciBench-Chemistry evaluation uses 223 college-level questions designed to test chemistry knowledge, complex reasoning, and calculations, with average accuracy over three runs.When questions specify answer units, the unit instruction is appended to the model input.Source: §C.3.1 · p. 14
- Because issues in ChemAgent’s codebase prevented evaluation under the authors’ setup, ChemToolAgent was instead tested using ChemAgent’s setup and reported results.That comparison used gpt-4-1106-preview and the dataset version released in ChemAgent’s repository.Source: §C.3.1 · p. 14
C.3.2 Detailed Results
On SciBench-Chemistry, Claude-based ChemToolAgent achieves the highest accuracies among the compared base models and ChemCrow, while the GPT version slightly trails GPT-4o. The section also reports stronger performance than ChemAgent and introduces the GPQA-Chemistry setup.
C.3.2 Detailed Results
- ChemToolAgent with Claude achieves the highest accuracies against the base models and ChemCrow, while the GPT version slightly underperforms GPT-4o.The authors associate Claude’s result with frequent PythonREPL use for calculations, especially on numerical questions; they note tool complexity and errors may offset benefits for GPT.Source: Table C.4 · p. 14, §C.3.2 · p. 14
- ChemToolAgent demonstrates stronger performance than ChemAgent under the reported comparison setup.The comparison uses the bottom section of Table C.4.Source: Table C.4 · p. 14, §C.3.2 · p. 14
C.4 GPQA-Chemistry
- GPQA-Chemistry uses 93 expert-verified chemistry questions from the Diamond subset to assess high-level knowledge and complex reasoning, reporting average accuracy over three runs.Models generate chain-of-thought solutions for answer extraction.Source: §C.4 · p. 14
D Error Cases
The error cases illustrate failures in chemistry reasoning and tool use, including incorrect inferences, overlooked information, and mishandling or over-trusting tool outputs, alongside an example identifying a SMILES compound as paricalcitol, which PubChem classifies as hazardous.
D.1 Reasoning: Wrong Knowledge/Logic and Incomplete Reasoning
- ChemToolAgent misidentifies bromine as the limiting reactant and fails to verify whether the remaining substances exceed the measurable-amount threshold.The PythonREPL calculation is correct, but rubidium is actually limiting; the example illustrates both wrong knowledge/logic and incomplete reasoning.Source: §D.1 · p. 15
Explanation
- On an optical-isomerism question, the agent rules out option (B) in its analysis but still selects (B) as its final answer.The final answer is wrong; the passage identifies this as a wrong-final-answer reasoning error.Source: Explanation · p. 16
- An algebra error in cross-multiplication produces the wrong [DMSO]:[AN] concentration ratio despite the preceding reasoning being correct.The agent predicts 1:9, while the passage identifies 1:6 as correct.Source: Explanation · p. 17, Step 3 · p. 18
- An extra “SMILES:” prefix makes both SideEffectPredictor and CanonicalizeSMILES reject the input, and the agent’s predicted answer differs from the stated correct answer.After both tools return invalid-SMILES errors, the agent turns to AiExpert for an assessment.Source: Explanation · p. 18, Problem Solving Process of… · p. 19, Step 2 · p. 19
D.6 Tool: Wrong Tool Output
- WebSearch supplies an incorrect electron-affinity result, which leads the agent to select silicon instead of the gold answer, calcium.The agent accepts the tool’s claim that silicon has the lowest electron affinity.Source: §D.6 · p. 20, Step 2 · p. 20
D.7 Tool: Inconsistent Tool Outputs
- When ToxicityPredictor and PubchemSearchQA give conflicting toxicity assessments, the agent trusts the latter and returns the wrong answer.ToxicityPredictor estimates a 7.85% probability of toxicity, while PubchemSearchQA claims the compound is hazardous; the stated correct answer is “No.”Source: §D.7 · pp. 20–21, Step 2 · p. 21
Step 3
- PubChem classifies the SMILES-identified compound paricalcitol as hazardous, with several GHS hazard classifications, documented toxicity, and potential health risks.Source: Step 3 · p. 21
E Tool Usage Analysis
Figure E.2 summarizes tool-use rates across SMolInstruct and MMLU-Chemistry, showing different patterns across the two task categories.
E Tool Usage Analysis
- On SMolInstruct, CTA predominantly invokes task-designed tools; for NC-I2F, IUPAC2SMILES and SMILES2Formula usage is often near 1.0.Figure E.2 expresses tool use as the ratio of samples using each tool to total samples.Source: E Tool Usage Analysis · p. 22, Figure E.2 · p. 22
- For the more open-ended molecular captioning and molecular generation tasks, CTA opts for a variety of tools.Source: E Tool Usage Analysis · p. 22
- On MMLU-Chemistry, CTA typically uses general-purpose tools such as PythonREPL and WebSearch because task-specific tools are absent.Figure E.2 reports the ratios of samples in which each tool is used.Source: E Tool Usage Analysis · p. 22, Figure E.2 · p. 22