Source-linked AI summary
LLM Agents Making Agent Tools
Georg Wölflein, Dyke Ferber, Daniel Truhn, Ognjen Arandjelović, Jakob Nikolas Kather
TL;DR
LLM agents are limited by their dependence on human-built tools, especially for specialised scientific workflows. ToolMaker transforms scientific papers and code repositories into executable LLM-compatible tools and evaluates them on complex multi-domain tasks. It correctly implements 80% of tasks and outperforms OpenHands, while benchmark success does not guarantee correctness in all real-world or high-stakes settings.
Problem
LLM agents depend on tools designed, implemented, and integrated by human developers, limiting their broader use in specialised scientific domains.
Method
ToolMaker autonomously transforms scientific papers and associated code repositories into executable LLM-compatible tools, including environment setup and implementation.
Results
80% accuracy: ToolMaker surpassed the state-of-the-art software engineering agent OpenHands across multiple scientific domains.
Takeaways & Limitations
ToolMaker provides a building block for future autonomous scientific workflows as more life-science research is conducted in silico.
Takeaways & Limitations
Passing TM-BENCH’s over 100 unit tests does not guarantee correctness in untested real-world scenarios or replace expert oversight in high-stakes clinical research.
Abstract
from arXiv · showhide
Tool use has turned large language models (LLMs) into powerful agents that can perform complex multi-step tasks by dynamically utilising external software components. However, these tools must be implemented in advance by human developers, hindering the applicability of LLM agents in domains demanding large numbers of highly specialised tools, like in life sciences and medicine. Motivated by the growing trend of scientific studies accompanied by public code repositories, we propose ToolMaker, an agentic framework that autonomously transforms papers with code into LLM-compatible tools. Given a GitHub URL and short task description, ToolMaker autonomously installs dependencies and generates code to perform the task, using a closed-loop self-correction mechanism for debugging. To evaluate our approach, we introduce a benchmark comprising 15 complex computational tasks spanning various domains with over 100 unit tests to assess correctness and robustness. Our method correctly implements 80% of the tasks, substantially outperforming current state-of-the-art software engineering agents. ToolMaker therefore is a step towards fully autonomous agent-based scientific workflows. Our code and benchmark are publicly available at https://github.com/KatherLab/ToolMaker.
1 Introduction
ToolMaker addresses the dependence of scientific LLM agents on human-built tools by transforming research code repositories into executable, LLM-compatible tools. The authors evaluate this approach with TM-BENCH, which covers complex workflows across diverse domains.
- Motivation: Existing agents struggle to generalise because their tools must be designed, implemented, and integrated manually before deployment.Current systems still depend heavily on manual intervention for compatibility and functionality.
- Motivation: Publicly released research code creates potential tools, but technical deployment requirements limit their use by researchers in healthcare, biology, drug development, and R&D.These requirements include setting up, installing, and adapting existing tools.
- Approach: ToolMaker autonomously generates an executable tool from a task description, scientific paper, and associated code repository.The framework is intended to bypass manual setup, installation, and adaptation by human intermediaries.
- Evaluation: TM-BENCH evaluates 15 diverse tasks across medical and non-medical domains, including downloading resources, resolving dependencies, reading codebases, and debugging.The benchmark includes over 100 unit tests for objective correctness assessment.
- Approach: ToolMaker creates a Docker execution environment and a Python function that performs the specified task.This workflow is illustrated as the framework’s output for a given task description, paper, repository, and invocation example.
2 Related work
Related work spans LLM reasoning agents, software engineering systems, medical and bioinformatics applications, and tool-creation benchmarks. The paper positions ToolMaker as addressing the gap between simple generated functions and reusable tools for complex scientific workflows.
- LLM agents: LLMs show promise in scientific question answering and reasoning but often struggle with complex problems requiring intermediate results from multiple steps.This motivates agent systems that use external tools and workflows.
- Agent limitations: Existing agentic systems remain constrained by their available tools, which can prevent effective use of domain-specific methods such as pathology foundation models.The cited pathology example contrasts a standard convolutional network with tools designed specifically for pathology.
- Tool creation: Prior tool-creation methods build simple tools from scratch and generally cannot interact with the operating system through commands or file operations.ToolMaker is presented as addressing both limitations.
- Benchmarks: Existing code-generation and tool-creation benchmarks focus on simple Python functions and cannot install dependencies or directly interact with the operating system.These benchmarks therefore differ from the repository-based, environment-dependent setting considered here.
- Benchmarks: Software engineering benchmarks evaluate particular tasks, whereas the proposed benchmark targets reusable tools that solve classes of problems.The paper combines software-engineering capabilities with tool creation for real-world multi-step scientific tasks.
3 TOOLMAKER
TOOLMAKER converts scientific papers and associated repositories into executable, LLM-compatible tools through environment setup, planning, implementation, and closed-loop correction. Its workflow represents conversation and environment state explicitly while combining LLM calls with environment interactions and agents.
- Tool definition: TOOLMAKER converts a scientific repository into an LLM-compatible Python tool from a task description, repository URL, and example input arguments.The system also sets up the execution environment rather than only generating the function implementation.
- Workflow stages: The workflow has two stages: environment setup creates a reproducible Docker snapshot, then tool implementation generates the desired Python function.The installation stage records repository setup actions so the execution environment can be reproduced.
- Workflow components: TOOLMAKER models workflow state as conversation history paired with environment state, and components transform this state while returning messages or observations.LLM calls update conversation history without modifying the environment, whereas environment interactions can update the environment and produce observations.
- Agents: Its agent chains tool-augmented LLM calls and environment interactions, appending tool calls and observations until it returns a sub-task result.The agent chooses actions or a final result, while direct execution of candidate implementations remains a separate workflow step.
- TOOLMAKER workflow: The workflow explores the repository, plans the implementation, writes candidate code, executes it in a reset environment, and iteratively assesses outputs for correction.Environment setup includes cloning repositories, reading documentation, and downloading dependencies such as models, datasets, and libraries.
4 Benchmark
TM-BENCH evaluates autonomous tool creation on diverse scientific tasks, especially in medicine and life sciences. It uses held-out invocations and unit tests that assess generalisation, outputs, files, and execution correctness.
- Benchmark scope: TM-BENCH contains 15 diverse tasks spanning pathology, radiology, omics, 3D vision, imaging, tabular analysis, and natural language processing.Tasks were curated with medicine and life-science researchers to reflect realistic scientific problems.
- Task construction: Each task is manually implemented beforehand to verify that its intended tool is meaningful, correctly specified, and feasible.The benchmark ranges from single-method calls to multi-step tasks involving function orchestration, data transformation, and GPUs.
- Scientific workflow coverage: The benchmark explicitly supports external files and complex scientific workflows, unlike code-generation benchmarks focused on simpler functions.Examples include tasks requiring datasets such as whole-slide images and clinical data tables.
- Correctness criteria: The unit tests assess return-value structure and values, produced files, and execution errors or crashes.This evaluation uses property-based checks rather than simple exact equality with reference outputs.
- Evaluation protocol: TM-BENCH includes 42 held-out test invocations and 124 unit tests across 15 tasks, requiring complete passage of a task’s tests for correctness.The test inputs differ from example invocations to evaluate generalisation and prevent hard-coding.
5 Results
TOOLMAKER substantially outperforms OpenHands on TM-BENCH while handling complex, multi-step scientific tool creation through environment setup, implementation, and self-correction. Ablations show paper summaries reduce creation effort without improving performance, while model choice affects cost and accuracy.
- Performance: 80% accuracy: TOOLMAKER correctly implemented 12/15 TM-BENCH tasks, compared with OpenHands’ 20% (3/15).A tool is correct only when all associated unit tests pass.
- Performance: OpenHands produced invalid environment definitions for nearly half of tasks, while TOOLMAKER’s setup process captured installation commands and reset environments between iterations.OpenHands’ setup failures caused installation scripts to crash before execution.
- Multi-step tools: TOOLMAKER completed the multi-step STAMP task by performing feature extraction before training a classification model.The self-correcting loop used 9 iterations and 33 actions for this task.
- Cost: TOOLMAKER averaged 21.8 actions and $0.94 per tool, versus 7.5 actions and $0.15 for OpenHands.The correctly implemented OpenHands tools were among TOOLMAKER’s cheapest and easiest tasks.
- Ablations: Including task-specific paper summaries did not increase performance but reduced creation effort for TOOLMAKER, lowering the STAMP task from 9 iterations and 33 actions to 5 iterations and 15 actions.The summaries were generated from full papers and then supplied to both agents.
- Ablations: Using o3-mini reduced cost but degraded performance for both systems, while OpenHands performed worse with Claude 3.5 Sonnet than with gpt-4o.The study evaluated both TOOLMAKER and OpenHands with o3-mini and reran OpenHands with Claude 3.5 Sonnet.
6 Conclusion
The paper presents TOOLMAKER as a framework that transforms scientific code repositories into LLM-compatible tools and evaluates it across multiple scientific domains. It reports 80% accuracy and releases TM-BENCH, while noting risks from automated implementation of complex biochemical tools.
- Conclusion: TOOLMAKER autonomously transforms scientific code repositories into LLM-compatible tools for real-world scientific tasks.The framework is intended to reduce the technical overhead of developing specialised toolsets.
- Conclusion: Across multiple scientific domains, TOOLMAKER surpassed OpenHands and achieved 80% accuracy.The paper reports this result as part of its evaluation of autonomous tool creation.
- Conclusion: The authors release TM-BENCH as a benchmark intended to support further advances in agentic tool creation.The benchmark accompanies the framework and evaluation.
- Risks: Automated tool creation in life sciences carries risks including potential misuse for harmful agents or bioweapons.The paper states that these risks require careful consideration.
Limitations
TOOLMAKER’s scope is limited by assumptions about repository quality and by the inability to automate physical experimentation. Its benchmark tests also cannot establish correctness for every real-world or high-stakes use case.
- Scope boundary: Fully autonomous scientific discovery remains constrained by physical experimentation, which TOOLMAKER does not address.
- Repository assumptions: TOOLMAKER assumes referenced repositories are reasonably well-structured, current, and documented, constraining autonomous installation when these conditions fail.The framework does not guarantee that any repository will be installable or usable as a tool.
- Evaluation limits: TM-BENCH’s unit tests cannot guarantee correctness across real-world edge cases or unexpected patterns.High-stakes clinical research additionally requires rigorous validation and oversight by domain experts.
- Reproducibility: Reproducibility may be affected by repository deletion, force-pushing, or branch renaming despite pinned repository commits.
A.1.2 Initial tool implementation
After creating an executable environment, TOOLMAKER implements a candidate tool from repository information and a generated plan, then iteratively evaluates and repairs it using execution feedback.
- Initial implementation: The initial implementation stage gathers repository information, creates a step-by-step plan, and asks the LLM to implement the tool function.Conversation history is retained between information gathering, planning, and implementation.
- Execution: TOOLMAKER resets the environment before running each candidate implementation, preventing prior write actions from contaminating execution.The candidate is executed using the example invocation specified in the tool definition.
- Assessment: TOOLMAKER assesses execution using the returned result and standard output and error streams to judge plausibility and detect errors.
- Self-correction: When execution fails, an agent diagnoses the root cause without resetting the environment, then re-implements the tool according to a repair plan.The agent can inspect intermediate files and outputs created during execution.
- Iteration: After each attempt, TOOLMAKER summarizes the diagnosis and repairs, resets the environment, and carries summaries plus current code into the next iteration.The conversation history is restored to the state immediately after the initial implementation before these summaries are appended.
B.1 Per-task ablation results
The supplementary ablation section presents extended results for different context and model settings, including paper-summary, o3-mini, and Claude 3.5 Sonnet conditions.
- Ablation results: Tables 4 to 6 provide detailed extended results for the ablations in a format similar to the main paper’s Table 2.
- Ablation results: Table 4 reports results with a paper summary in context.
- Ablation results: Tables 5 and 6 report results using o3-mini and Claude 3.5 Sonnet, respectively.
B.2 Raw unit test results
The supplementary materials provide raw unit-test results for benchmark experiments and ablations, alongside a figure showing TOOLMAKER’s transitions between tool calls and complete task definitions.
- Raw unit tests: Tables 7 and 8 provide raw unit-test results for the main experiments, while Tables 9 to 13 cover the ablations.
- Tool-call transitions: Figure 5 shows transitions between tool calls made by TOOLMAKER.
- Task definitions: The supplementary section includes complete task definitions for all TM-BENCH tasks.
C.1 Pathology
The pathology section presents ToolMaker tasks spanning feature extraction, biomarker analysis, classification, segmentation, and model training, alongside raw benchmark-result tables.
- Pathology tools: Pathology tasks include feature extraction with CONCH, UNI, MUSK, and CTransPath, producing feature vectors or saved slide features.
- Evaluation: The section includes raw-result tables for ToolMaker and OpenHands under paper-summary, model-choice, and baseline conditions.
- Pathology tools: The biomarker-verification task uses whole-slide-image probability maps, a biomarker hypothesis, and clinical data to assess prognostic significance and independence.
- Pathology tools: Other pathology tasks train biomarker-classification models from whole-slide images and clinical or slide metadata, with configured target and model outputs.
- Pathology tools: Medical-imaging tasks include MedSAM abdomen-CT segmentation and nnUNet training with selectable configurations and folds.
C.3 Omics
The omics section covers tools for generating biological representations, preparing knowledge-base inputs, masked-language prediction, imaging workflows, and tabular evaluation.
- Omics tools: The Cytopus task initializes a knowledge base and writes nested gene-set annotations organized by cell type for Spectra input.
- Omics tools: Protein modeling generates sequence representations and contact maps from esm2_t33_650M_UR50D, returning one-dimensional and L×L outputs.
- Omics tools: Additional tasks extract retinal-image features with RETFound and predict masked sentences with ModernBERT-base on CPU.
- Additional tasks: The benchmark also includes FlowMap camera-extrinsic estimation, TabPFN training and testing, and MedSSS_Policy response generation.
- ToolMaker workflow: ToolMaker’s workflow prompts cover repository installation, task-specific setup, environment use, iterative diagnosis, and code correction.