Source-linked AI summary
SciToolAgent: A Knowledge Graph-Driven Scientific Agent for Multi-Tool Integration
Keyan Ding, Jing Yu, Junjie Huang, Yuchen Yang, Qiang Zhang, Huajun Chen
TL;DR
Specialized scientific tools are difficult to integrate into complex workflows, while existing LLM-based approaches struggle with tool interdependencies. SciToolAgent combines a scientific tool knowledge graph with LLM-based planning, execution, summarization, and safety checking. It achieves 94% accuracy on 531 problems, surpasses state-of-the-art baselines by 10%, and demonstrates multi-tool orchestration across four scientific case-study scenarios.
Problem
Specialized scientific tools create substantial expertise barriers, and existing LLM-based frameworks struggle to integrate and orchestrate interdependent tools in complex workflows.
Method
SciToolAgent uses SciToolKG to guide dynamic tool-chain selection and combines LLM-based planning, sequential execution, summarization, and safety checking across hundreds of scientific tools.
Results
94% overall accuracy on 531 diverse scientific problems surpasses state-of-the-art baselines by 10%, with case studies demonstrating complex multi-tool workflow orchestration.
Takeaways & Limitations
SciToolAgent supports autonomous scientific workflows and aims to make advanced research tools more accessible to experts and non-experts.
Takeaways & Limitations
SciToolKG scalability is constrained by the manual effort required to curate and update tool information, while open-source models can lag behind GPT-4o in complex planning and reasoning.
Abstract
from arXiv · showhide
Scientific research increasingly relies on specialized computational tools, yet effectively utilizing these tools demands substantial domain expertise. While Large Language Models (LLMs) show promise in tool automation, they struggle to seamlessly integrate and orchestrate multiple tools for complex scientific workflows. Here, we present SciToolAgent, an LLM-powered agent that automates hundreds of scientific tools across biology, chemistry, and materials science. At its core, SciToolAgent leverages a scientific tool knowledge graph that enables intelligent tool selection and execution through graph-based retrieval-augmented generation. The agent also incorporates a comprehensive safety-checking module to ensure responsible and ethical tool usage. Extensive evaluations on a curated benchmark demonstrate that SciToolAgent significantly outperforms existing approaches. Case studies in protein engineering, chemical reactivity prediction, chemical synthesis, and metal-organic framework screening further demonstrate SciToolAgent's capability to automate complex scientific workflows, making advanced research tools accessible to both experts and non-experts.
1 Introduction
Scientific tool diversity and interdependencies make complex workflows difficult to use, especially for researchers lacking specialized expertise. SciToolAgent addresses this challenge with a knowledge-graph-guided, safety-aware workflow and reports strong benchmark and case-study performance.
- Motivation: Specialized scientific tools are increasingly essential, but their complexity and diversity create adoption barriers for researchers without substantial technical expertise.These tools support tasks including molecular simulation, property prediction, and compound design.
- Motivation: Naive in-context agent frameworks struggle with complex workflows because they overlook sequential dependencies among tools and their input-output relationships.Incorrect operational ordering can produce suboptimal solutions and reduced efficiency.
- Approach: Its workflow combines an LLM Planner for graph-based tool-chain generation, an Executor for sequential implementation, safety checking, and a Summarizer that can trigger replanning.The system also stores final answers as context for subsequent queries.
- Approach: SciToolAgent uses SciToolKG to model tool dependencies, prerequisites, compatibility, formats, functionalities, and safety levels across biology, chemistry, and materials science.The knowledge graph supports informed tool selection and combination among hundreds of tools.
- Evaluation: 94% overall accuracy on 531 diverse scientific problems surpasses state-of-the-art baselines by 10%.The benchmark spans multiple scientific domains and complexity levels.
- Evaluation: Case studies in protein design, chemical reactivity prediction, chemical synthesis, and metal-organic framework screening demonstrate autonomous orchestration of complex multi-tool workflows.The studies report maintained solution reliability and accuracy across these scenarios.
2 Results
SciToolAgent combines knowledge-graph-guided planning, multi-tool execution, and safety checks to address complex scientific workflows. It outperformed baseline agents on SciToolEval and demonstrated applications across protein design, chemical prediction and synthesis, and MOF screening.
- Overview of SciToolAgent: SciToolAgent uses SciToolKG to model tool dependencies, prerequisites, and compatibility, supporting informed tool selection and combination.Its LLM-powered workflow includes a Planner, Executor, and Summarizer, with retrieval-augmented generation using the knowledge graph.
- Evaluation setup: SciToolEval contains 531 scientific questions, and evaluation measures pass rate, tool-planning accuracy, and final-answer accuracy.The benchmark spans multiple fields and complexity levels, including single-tool Level-1 and multi-tool Level-2 tasks.
- Foundation-model comparison: OpenAI o1 performed best across metrics, while GPT-4o provided the reported accuracy–cost trade-off and remained SciToolAgent’s default model.A finetuned Qwen2.5-7B improved by 10% using data generated from SciToolKG.
- Scientific case studies: Case studies show SciToolAgent orchestrating workflows for protein design, chemical reactivity prediction, chemical synthesis and analysis, and MOF screening.The examples include protein sequence and structure analysis, machine-learning reactivity prediction, synthesis characterization and safety evaluation, and filtering MOFs by stability, adsorption, and price.
3 Discussion
SciToolAgent combines a knowledge graph and safety module to orchestrate scientific tools across domains. Its reported strengths are broad workflow automation, benchmark performance, and accessibility, while scalability and model dependence remain limitations.
- Discussion: SciToolKG captures tool dependencies, input/output formats, and application contexts, enabling dynamic tool chains tailored to scientific tasks.This supports delegation of repetitive or computationally intensive steps to the agent.
- Discussion: Manual construction and updating of SciToolKG constrains scalability, motivating automated knowledge extraction and broader tool integration.The paper also describes standardized APIs, templates, and planned GUI-based registration to support extensibility.
- Discussion: Underlying LLM capability limits performance across varied scientific tasks, with Qwen2.5-7B-FT trailing GPT-4o in complex planning and multi-step reasoning.The paper notes that domain-specific fine-tuning can partially narrow the gap between open-source and proprietary models.
- Discussion: Despite knowledge-graph scalability and proprietary-model challenges, SciToolAgent provides a foundation for automating complex scientific workflows.Future work targets automated graph maintenance, expanded tool coverage, and stronger open-source LLMs.
4 Methods
The methodology covers scientific-tool collection, construction of SciToolKG and SciToolEval, and implementation of the agent’s core components and underlying language models.
- 4 Methods: The methodology includes collecting scientific tools and constructing SciToolKG and SciToolEval from those tools.These activities form the paper’s data and knowledge-graph preparation stages.
- 4 Methods: The implementation includes Planner, Executor, and Summarizer components, together with the LLMs behind them.The section frames these components as part of the agent’s implementation details.
- 4 Methods: The tool collection and agent implementation are presented as parts of a methodology spanning scientific resources, knowledge representation, and execution components.The passage organizes the methodology around these three elements.
4.1 Collection of Scientific Tools
SciToolAgent’s tool collection uses a systematic process to assemble a comprehensive, domain-specific, functionally diverse set of scientific tools across biology, chemistry, and materials science.
- 4.1 Collection of Scientific Tools: The collection process targets biology, chemistry, and materials science as domains that would benefit from LLM integration.Tools were selected from domains identified as especially suitable for this integration.
- 4.1 Collection of Scientific Tools: The current SciToolAgent toolset includes over 500 tools spanning a wide range of functionalities.The passage describes the collection as comprehensive, domain-specific, and functionally diverse.
- 4.1 Collection of Scientific Tools: Researchers first identified key domains and then curated frequently used tools within those domains.This sequence defines the stated collection procedure.
4.2 Construction of SciToolKG
SciToolKG represents scientific tools and their operational relationships as a structured directed graph. Its construction proceeds through tool characterization, schema development, and graph population.
- 4.2 Construction of SciToolKG: SciToolKG is a directed graph G = (V, E) whose nodes represent tools or attributes and whose edges represent relations between them.The graph provides a structured representation of tool relationships, dependencies, and operational details.
- 4.2 Construction of SciToolKG: The graph’s structured representation connects tool entities with attribute and relation information needed to model dependencies and operational details.This combines the graph representation with the attributes used during construction.
- 4.2 Construction of SciToolKG: Tool characterization defines attributes including purpose, functionalities, input/output formats, categories, sources, and safety levels.These attributes are mainly derived from tool documentation.
- 4.2 Construction of SciToolKG: The knowledge graph is constructed through tool characterization, schema development, and graph population.A hierarchical schema models the tool attributes and their logical organization.
4.3 Construction of SciToolEval
SciToolEval is a curated benchmark for quantitatively assessing scientific-tool agents across diverse real-world tasks. Its construction combines generated questions, tool execution, answer generation, and expert review.
- SciToolEval provides a testing ground for quantitatively assessing agents across diverse scientific domains.The dataset was designed around real-world scientific tasks.
- Questions are generated from selected tools with GPT-4o to ensure they are actionable and relevant to tool functionality.
- Generated questions are answered through tool execution using ReACT with only question-relevant tools supplied to improve invocation accuracy.
- Three domain-specific experts manually review the questions, tool usage, and answers for correctness and significance.
- 531 questions cover single- and multi-tool problem solving, with Level-1 single-tool queries and Level-2 sequential or parallel multi-tool tasks.Level-2 tasks require basic planning, reasoning, and summarization skills.
4.4 Implementation of SciToolAgent
SciToolAgent uses LLM-based Planner, Executor, and Summarizer components coordinated through SciToolKG. The system retrieves and ranks dependent tools, generates ordered tool chains, executes them with safety checks and error handling, and synthesizes or refines the final answer.
- SciToolAgent consists of Planner, Executor, and Summarizer components, each assigned a distinct role in the problem-solving workflow.
- Planner: The Planner queries SciToolKG and uses retrieval-augmented generation to identify and sequence tools into a chain for the user’s query.
- Planner: Full-graph retrieval selects the k tools most semantically similar to the question by comparing the question with tool-function triples.
- Planner: Sub-graph exploration searches d-hop neighborhoods of retrieved tools to identify additional tools needed in conjunction with them.
- Planner: Tools are ranked using S′ = S(q, Ti)×S(q, Ti⊕Tj), favoring tools that are both relevant to the query and complementary.
- Planner: The Planner generates an ordered chain-of-tools from selected tools and SciToolKG neighborhood information, using tool dependencies and functionalities.
- Planner: Retrieval uses d = 3, k = 5, and n = 10 by default, with cosine distance between pretrained text-embedding vectors measuring semantic similarity.
- Executor: The Executor prepares inputs, invokes tools, monitors execution, logs details, captures outputs, and adapts inputs when errors occur.