Source-linked AI summary
CACTUS: Chemistry Agent Connecting Tool-Usage to Science
Andrew D. McNaughton, Gautham Ramalaxmi, Agustin Kruel, Carter R. Knutson, Rohith A. Varikoti, Neeraj Kumar
TL;DR
LLMs have limited access to dynamic or domain-specific information and may fail at reasoning, challenges that are especially pronounced in chemistry. CACTUS combines open-source LLMs with cheminformatics tools, and its benchmark results show better performance than baseline LLMs, with Gemma-7b and Mistral-7b achieving the highest accuracy across prompting strategies.
Problem
LLMs have limited access to dynamic, real-time, or confidential information, and chemical data and drug discovery present challenges pure computational models alone cannot effectively address.
Source: §1 · p. 2Method
CACTUS is an LLM-powered agent that selects suitable cheminformatics tools and the order in which to apply them for chemical research tasks.
Source: §1 · p. 2Results
CACTUS significantly outperformed baseline LLMs on one thousand chemistry questions, with Gemma-7b and Mistral-7b achieving the highest accuracy regardless of prompting strategy.
Source: §4 · p. 10Takeaways & Limitations
The findings suggest smaller models can be deployed on consumer-grade hardware without significant accuracy loss, potentially widening access to CACTUS.
Source: §4 · p. 10Further work is needed to optimize open-source model deployment across hardware configurations and develop standardized prompt-engineering practices.
Source: §3.3 · p. 9
Abstract
from arXiv · showhide
Large language models (LLMs) have shown remarkable potential in various domains, but they often lack the ability to access and reason over domain-specific knowledge and tools. In this paper, we introduced CACTUS (Chemistry Agent Connecting Tool-Usage to Science), an LLM-based agent that integrates cheminformatics tools to enable advanced reasoning and problem-solving in chemistry and molecular discovery. We evaluate the performance of CACTUS using a diverse set of open-source LLMs, including Gemma-7b, Falcon-7b, MPT-7b, Llama2-7b, and Mistral-7b, on a benchmark of thousands of chemistry questions. Our results demonstrate that CACTUS significantly outperforms baseline LLMs, with the Gemma-7b and Mistral-7b models achieving the highest accuracy regardless of the prompting strategy used. Moreover, we explore the impact of domain-specific prompting and hardware configurations on model performance, highlighting the importance of prompt engineering and the potential for deploying smaller models on consumer-grade hardware without significant loss in accuracy. By combining the cognitive capabilities of open-source LLMs with domain-specific tools, CACTUS can assist researchers in tasks such as molecular property prediction, similarity searching, and drug-likeness assessment. Furthermore, CACTUS represents a significant milestone in the field of cheminformatics, offering an adaptable tool for researchers engaged in chemistry and molecular discovery. By integrating the strengths of open-source LLMs with domain-specific tools, CACTUS has the potential to accelerate scientific advancement and unlock new frontiers in the exploration of novel, effective, and safe therapeutic candidates, catalysts, and materials. Moreover, CACTUS's ability to integrate with automated experimentation platforms and make data-driven decisions in real time opens up new possibilities for autonomous discovery.
1 Introduction
The introduction frames chemistry-focused LLM use as constrained by reasoning limitations and gaps in access to dynamic or specialized information. CACTUS addresses this by pairing LLMs with cheminformatics tools in an agent that selects suitable tools and their sequence for chemical research.
1 Introduction
- Prior work describes tool augmentation and task-specific prompting as ways to support problem solving, with the TALM framework combining these approaches and outperforming models on configured tasks [Mialon et al., 2023, Xu et al., 2023, Qin et al., 2023; Cai et al., 2023; Parisi et al., 2022].The introduction also notes that LLMs have limited understanding of curated or parsed text [Chiesurin et al., 2023].Source: §1 · p. 1
- LLMs may produce plausible answers yet fail at human-like reasoning or subject knowledge, and learned statistical relationships can yield similar mistakes across applications [Huang et al., 2023, Kambhampati, 2024; Bommasani et al., 2021].The introduction warns that such failures could have unwanted outcomes if foundation models are integrated into important systems.Source: §1 · p. 2
- Static training datasets leave LLMs without access to dynamic, real-time, or confidential data, while chemical data and drug discovery pose challenges that pure computational models alone cannot effectively address.The authors present cheminformatics-tool integration as a promising path for domain-specific applications.Source: §1 · p. 2
- CACTUS is an LLM-powered cheminformatics agent designed to select appropriate tools and their order for workflows in de novo drug design and molecular discovery.The authors describe this as a way to combine computational linguistics with chemical science.Source: §1 · p. 2
- The authors describe the combination of human intelligence, AI, and specialized tools as having potential to make molecular discovery more efficient, accurate, and innovative.They connect intelligent agents with autonomous work spanning data analysis, experimental planning, hypothesis generation, and testing.Source: §1 · p. 2
2 Methods
CACTUS combines an LLM agent with cheminformatics tools and evaluates how its tool use responds to domain-specific prompting on a varied question benchmark.
2 Methods
- The methods frame the agent as a system built from external tools and language models, with components intended to support a range of problems and quick model swapping.The workflow proceeds through planning, action, execution, and observation.Source: §2 · p. 2, Figure 1 · p. 3
2.1 The Agent
- CACTUS uses a LangChain custom MRKL agent combining cheminformatics tools, an LLMChain prompt, and a zero-shot agent to select actions and integrate new tools.The ReAct framework selects a tool based on the user input, available tools, and their descriptions.Source: §2.1 · pp. 2–3
2.2 Cheminformatics Tools
- Ten tools built around cheminformatics libraries and databases analyze SMILES inputs for molecular descriptors, pharmacokinetic properties, and structural or toxicity alerts.Supported functions include molecular weight, LogP, TPSA, QED, synthetic accessibility, blood-brain barrier permeability, and gastrointestinal absorption.Source: §2.2 · p. 4, Table 1 · p. 4
2.3 Prompting Strategy
- The study compares a default Minimal Prompt with a chemistry-aligned Domain Prompt to test the role of domain-specific prompting.The Minimal Prompt includes model-specific tokens to avoid unfairly evaluating models against the Domain Prompt.Source: §2.3 · p. 4
2.4 Benchmarking
- The benchmark assesses qualitative, quantitative, and combined question sets, with 500 questions each in the first two sets and 1,000 in the combined set.Most tests use the combined set to assess performance across diverse tasks; Table 2 gives example questions.Source: §2.4 · pp. 4–5, Table 2 · p. 5
3 Results and Discussion
CACTUS benchmarking finds performance varies with model and prompting strategy, while smaller models can run on consumer-grade hardware. The discussion also identifies deployment and prompt-engineering challenges and outlines broader molecular-design capabilities for future development.
3.1.1 Open Source Models in Varied Settings
- Gemma-7b and Mistral-7b perform robustly across prompting strategies, whereas Falcon-7b shows a noticeable performance disparity between minimal and domain prompts.The comparison spans questions from physicochemical properties to more complex measures such as QED.Source: §3.1 · p. 5, Figure 3 · p. 7
3.1 Benchmarking and Performance Evaluation
- Domain-specific prompts improve response accuracy, particularly for qualitative questions; Figure 2 reports similar performance between prompt types on quantitative questions.The figure compares Gemma-7b with minimal and domain prompts on the full question-set benchmark.Source: §3.1 · p. 5, Figure 2 · p. 6
3.1.1 Open Source Models in Varied Settings
- Smaller-model deployment on consumer-grade hardware may widen access to CACTUS, while prompt customization offers a way to adapt open-source models to specific domains.The discussion identifies Phi2 and OLMo-1b as smaller models and notes researchers with limited computational resources as potential users.Source: §3.1 · p. 6, §3.1.1 · p. 7
3.2 Hardware Performance and Model Efficacy
- 185 minutes were required for Llama2-7b with domain prompting on the full dataset, despite accuracy similar to its minimal-prompt version.Inference time generally rose with question-set size; domain prompts improved speed and accuracy for Falcon-7b, MPT-7b, and Mistral-7b, while Phi2 on the full set was faster but less accurate with a minimal prompt.Source: §3.2 · p. 8
- 52.2% accuracy was achieved by OLMo-1b with a domain prompt on the combined question set using an RTX 2080 Ti.The minimal-prompt version returned no correct answers on that benchmark; Phi2 handled the quantitative benchmark quickly with similar performance across GPUs.Source: §3.2 · p. 8
3.3 Issues Encountered and Resolutions
- CPU-hosted models were slow, prompting a switch to vLLM, while different open-source models required specialized prompts; deployment optimization and prompt standardization remain future needs.The authors report that vLLM substantially decreased inference time, including on consumer-grade GPUs.Source: §3.3 · p. 9
3.4 Future Outlook - Molecular Design
- Future plans add physics-based models, similarity and descriptor screening, and symbolic reasoning and explainability, extending CACTUS toward drug discovery, catalysis, and materials applications.The proposed additions include molecular dynamics simulations, quantum chemistry calculations, and high-throughput virtual screening.Source: §3.4 · pp. 9–10
4 Conclusions
CACTUS combines open-source LLMs with cheminformatics tools and is benchmarked on chemistry questions. The conclusion highlights its performance, deployment challenges, and directions for expanding the platform.
4 Conclusions
- CACTUS integrates LLMs and cheminformatics tools as a user-friendly platform for exploring chemical space and identifying promising therapeutic compounds.Source: §4 · p. 10
- CACTUS significantly outperforms baseline LLMs, with Gemma-7b and Mistral-7b achieving the highest accuracy regardless of prompting strategy.The paper also examines domain-specific prompting and hardware configurations.Source: §4 · p. 10
- The work identifies inference speed and prompting as integration challenges, addressed with vLLM hosting and tailored prompts for each LLM.The authors emphasize model deployment and prompt engineering as priorities for future work.Source: §4 · p. 10
- Future development proposes 3D-scaffold models, reinforcement learning, graph neural networks, added screening tools, and stronger explainability and symbolic reasoning.The proposed tools include fragment identification, molecular-property calculation, and screening for toxic groups.Source: §4 · p. 10
- The authors envision integrating CACTUS with computational tools and autonomous discovery platforms to identify candidates for experimental validation and optimization.The stated future goal includes improving safety and efficacy in identifying and designing therapeutic drugs, catalysts, and materials.Source: §4 · p. 10
- The conclusion presents CACTUS as an adaptable cheminformatics tool for drug discovery, molecular-property prediction, and related research.It frames open-source language models and cheminformatics tools as a route toward exploring therapeutic agents, catalysts, and materials.Source: §4 · p. 10
A Benchmark Data
The benchmark table lists model, hardware node, question set, prompt, time, and accuracy for selected configurations.
A Benchmark Data
- 98.4 accuracy is reported for Gemma-7b on A100 80GB with the Quantitative question set and Full prompt.The table reports a time of 25.61667 for this configuration.Source: A Benchmark Data · p. 14
- 93.6 accuracy is reported for Phi2 on V100 with the Quantitative question set and Minimal prompt.The corresponding time is 19.93333; Phi2 on A100 80GB with the same set and prompt also has 93.6 accuracy, with a time of 12.1833.Source: A Benchmark Data · p. 14
- OLMo-1b on RTX 2080 Ti with the Full question set and Minimal prompt has 0 accuracy in the table.The table reports a time of 51.48333 for this configuration.Source: A Benchmark Data · p. 14