Source-linked AI summary
BioTool: A Comprehensive Tool-Calling Dataset for Enhancing Biomedical Capabilities of Large Language Models
Xin Gao, Ruiyi Zhang, Meixi Du, Peijia Qin, Pengtao Xie
TL;DR
Biomedical LLMs often cannot reliably use the specialized tools required for high-quality scientific answers, while existing in-context approaches cover too few tools. BioTool addresses this gap with a human-verified biomedical tool-calling dataset, and fine-tuning improves tool-calling and downstream answer quality. The framework remains limited to one-hop tool-calling responses and does not train an independent specialized biomedical agent.
Problem
Biomedical LLMs have limited ability to leverage specialized tools, while existing in-context approaches cover few tools and struggle with complex schemas and usage scenarios.
Method
BioTool provides 7,040 human-verified query–API call pairs from 34 tools across NCBI, Ensembl, and UniProt for instruction fine-tuning.
Results
BioTool-fine-tuned models outperform larger commercial models on tool calling, while BioTool-augmented LLMs produce higher-quality biomedical answers than the same models without tool usage.
Takeaways & Limitations
BioTool supports more grounded, informative, and scientifically accurate biomedical answers by training models to call specialized APIs.
Takeaways & Limitations
The framework currently handles only one-hop tool-calling responses and does not fine-tune an independent specialized biomedical agent.
Abstract
from arXiv · showhide
Despite the success of large language models (LLMs) on general-purpose tasks, their performance in highly specialized domains such as biomedicine remains unsatisfactory. A key limitation is the inability of LLMs to effectively leverage biomedical tools, which clinical experts and biomedical researchers rely on extensively in daily workflows. While recent general-domain tool-calling datasets have substantially improved the capabilities of LLM agents, existing efforts in the biomedical domain largely rely on in-context learning and restrict models to a small set of tools. To address this gap, we introduce BioTool, a comprehensive biomedical tool-calling dataset designed for fine-tuning LLMs. BioTool comprises 34 frequently used tools collected from the NCBI, Ensembl, and UniProt databases, along with 7,040 high-quality, human-verified query-API call pairs spanning variation, genomics, proteomics, evolution, and general biology. Fine-tuning a 4-billion-parameter LLM on BioTool yields substantial improvements in biomedical tool-calling performance, outperforming cutting-edge commercial LLMs such as GPT-5.1. Furthermore, human expert evaluations demonstrate that integrating a BioTool-fine-tuned tool caller significantly improves downstream answer quality compared to the same LLM without tool usage, highlighting the effectiveness of BioTool in enhancing the biomedical capabilities of LLMs. The full dataset and evaluation code are available at https://github.com/gxx27/BioTool
1 Introduction
Biomedical LLMs struggle to use specialized tools, motivating BioTool, a human-verified dataset for fine-tuning biomedical API calling. Fine-tuned models improve both tool-calling performance and downstream biomedical answer quality.
- Motivation: Biomedical LLMs remain limited because they struggle to leverage specialized tools and databases needed for reliable scientific answers.Without such tools, they are prone to hallucinations and imprecise generalizations.
- Motivation: Existing in-context approaches improve tool use but remain constrained by context length, limited tool coverage, and complex biomedical schemas.These approaches cannot fully represent diverse usage scenarios or reliably map questions to specialized identifiers and parameters.
- Contribution: 7,040 human-verified query–API call pairs form BioTool, covering 34 tools from NCBI, Ensembl, and UniProt across major biomedical subdomains.The dataset spans variation, genomics, proteomics, evolution, and general biology.
- Results: 15.0% higher overall API-calling quality distinguishes a BioTool-fine-tuned 4B Qwen-3 model from Claude-4.5-Sonnet.The comparison uses biomedical tool-calling quality on the test split.
- Results: 88.4% higher normalized answer quality results when GPT-5.1 receives oracle BioTool API calls, while a fine-tuned API caller yields 69% higher quality than raw GPT-5.1.These human evaluations use 1,048 test queries.
2 Related Works
General-purpose tool-calling datasets enable API invocation, but specialized biomedical use remains difficult because generalist models lack domain-specific tool knowledge.
- General tool calling: Toolformer and Gorilla established that LLMs can invoke external APIs to ground responses in retrieved data and mitigate hallucinations.ToolBench and APIGen subsequently introduced scalable pipelines for synthetic tool-calling instruction data.
- Biomedical gap: Generalist tool-calling models still struggle in biomedicine because broad datasets do not adequately capture specialized scientific tools and workflows.
3 The BioTool Dataset
BioTool is constructed from biomedical APIs through tool selection, API synthesis, automated and human verification, and query generation. Its 7,040 samples span diverse databases, tool types, biological domains, and endpoint usage patterns.
- Dataset structure: Each BioTool entry pairs a free-form biomedical query with tool information, API arguments, and observations returned by executing the endpoint.The example uses UniProt’s human_diseases tool to retrieve definitions for severe immunodeficiency disorders.
- Tool selection: NCBI, UniProt, and Ensembl provide the authoritative source platforms selected for their broad, interoperable coverage across the central dogma of biology.
- Tool selection: 34 selected tools comprise 124 API endpoints, excluding low-relevance, deprecated, or unstable tools and using official documentation for synthesis.
- API synthesis and verification: 6,391 unique API calls are produced after synthesizing biologically meaningful arguments and verifying their execution.
- Query generation and filtering: User queries are generated from API calls, documentation, and observations, with CoT prompting supporting necessity and informativeness filtering supporting sufficiency.
- Human refinement: Human evaluators refine terminology and phrasing before finalizing 7,040 high-quality samples.
- Dataset composition: The dataset balances its three source databases and covers retrieval, search, mapping, proteomics, variation, and evolutionary tools, including a long tail of specialized tools.
4 Experimental Results
Experiments evaluate BIOTOOL-trained tool callers against untuned and proprietary models, then test whether tool augmentation improves biomedical answer quality. Results show strong gains in API calling, parameter mapping, and human-judged answers.
- Results on Tool Calling Capability: 15.0%: the BIOTOOL-fine-tuned 4B model improves overall BioTool score over Claude 4.5 Sonnet, while achieving 68.9% higher performance than GPT-5.1.The comparison uses small open-source models fine-tuned on BIOTOOL against proprietary models evaluated with in-context learning.
- Human Evaluation of Answer Quality: Human evaluations compare GPT-5.1 without tools against oracle BIOTOOL augmentation and a BIOTOOL-fine-tuned Qwen3-4B-Instruct tool caller.Two annotators with bioinformatics backgrounds judge informativeness and task fulfillment across all test queries.
- Human Evaluation of Answer Quality: 94.2% and 84.5%: oracle BIOTOOL data and the fine-tuned Qwen3-4B-Instruct tool caller win against the unaugmented GPT-5.1 baseline, respectively.The win rates reflect normalized human judgments of answer quality.
- Additional Results: Fine-tuned Qwen3-4B-Instruct achieves nearly six times the Exact Match of the best proprietary model, despite proprietary models often attaining high API Success.The EM–AS divergence indicates that execution success does not ensure exact reference parameterization.
- Error Analysis: BIOTOOL fine-tuning substantially reduces semantic parameter-mapping failures involving database or program choices, identifiers, and Entrez traversal targets.These errors can produce biologically misaligned retrievals even when API calls execute successfully.
- Case Study: BIOTOOL supervision teaches implicit biomedical tool-usage rules, enabling correct mapping of natural-language concepts to required API parameters.A case study contrasts a Claude model passing literal terms into API arguments with a fine-tuned model mapping them correctly.
5 Conclusion
BIOTOOL is a comprehensive biomedical tool-calling dataset that improves API-calling performance and downstream answer quality when used to augment LLMs.
- BIOTOOL comprises 7,040 human-verified query–API call pairs spanning 124 biomedical tools.
- Fine-tuning a 4-billion-parameter LLM on BIOTOOL substantially improves API-calling performance and surpasses cutting-edge proprietary LLMs.
- Human evaluations find that BIOTOOL-augmented LLMs produce more helpful, informative, and scientifically accurate answers than the same base models without tool usage.
Limitations
BIOTOOL currently supports only one-hop tool-calling responses and does not include an independent specialized biomedical agent, limiting coverage of complex workflows.
- The framework focuses exclusively on one-hop tool-calling responses, excluding problems requiring multi-hop search or iterative reasoning across multiple tools.
- The authors did not fine-tune an independent specialized biomedical agent because raw biological observations often exceed available context-length resources.
- Future work should investigate long-context architectures and multi-step reasoning trajectories for intricate clinical and research workflows.
A Heuristic Filter Detail
BIOTOOL filters synthesized API calls with database-specific heuristics to reduce duplication and retain biologically informative, executable samples.
- Filtering logic varies across NCBI, Ensembl, and UniProt to reflect differences in API architectures and biological data.
- Database-specific filtering: UniProt filtering deduplicates calls targeting the same unique identifier, while restricted APIs retain some differing optional-parameter combinations.
- Database-specific filtering: For rich APIs, filtering removes samples with duplicate required-parameter combinations while preserving varied inputs.
- Database-specific filtering: BLAST filtering retains unique query sequences that return at least one significant alignment hit and removes matchless queries.
- Database-specific filtering: Other NCBI APIs remove identical identifier calls and verify that retrieved observations remain biologically informative.
B Dataset Scale Analysis
Performance improves substantially even with limited BIOTOOL training data, with gains evaluated against the untuned Qwen3-4B-Instruct baseline across three metrics.
- Six Qwen3-4B-Instruct training subsets range from 10% to 100% of the BIOTOOL training split.
- The untuned baseline scores 3.6 Exact Match, 73.3 API Success, and 63.1 BioTool Score.
- Even the smallest training subset yields substantial gains across Exact Match, API Success, and BioTool Score.
C Generalization Capability
The unseen-API split tests whether BioTool fine-tuning generalizes to API functions absent from training. Fine-tuned Qwen3-4B retains advantages, though the gap narrows compared with random-split evaluation.
- Unseen-API evaluation assigns all samples from each API function to one partition, ensuring every test API is absent from training.
- Qwen3-4B-Instruct retains a clear Exact Match advantage and the strongest BioTool Score against GPT-5.1 and GPT-5.1-Codex on unseen APIs.
- The performance gap is smaller on unseen APIs than under the standard random split, showing that unseen-function generalization is more challenging.
- BioTool fine-tuning provides transferable API-calling gains beyond memorization of the training API set.
D Human Evaluation Details
Human evaluation compares tool-augmented and base models using researcher judgments across 1,408 samples per setting. Structured prompts and agreement statistics support the evaluation procedure’s consistency.
- 1,408 samples per comparison setting were manually evaluated by researchers with biological backgrounds.The evaluations compared tool-augmented models with the base GPT-5.1 generator and recorded four outcome categories, including both-good and both-bad cases.
- Adjusted winning rates treat both-good and both-bad responses as ties and split those ties evenly between conditions.The adjustment follows the logic of McNemar’s test using raw preference counts.
- Both Cohen’s κ and Krippendorff’s α exceed the accepted agreement threshold of 0.667 across settings.The reported agreement also falls within the substantial range under the Landis–Koch scale.
- The informativeness rubric labels an observation true when it contains at least one relevant, non-trivial fact supporting part of the query.Partial answers remain informative, while errors, unrelated content, or observations too vague for one concrete relevant statement are false.
- The evaluation prompts provide the user query, API call, and observation to generate concise final responses.The tool-augmented prompt treats the observation as primary evidence while permitting general context when it is insufficient.
F Tool and API List
BioTool draws on NCBI, UniProt, and Ensembl APIs covering a broad range of biomedical data and analysis tasks. Its tool list spans database search, sequence analysis, comparative genomics, variation, and related resources.
- The tool inventory includes NCBI ESearch, ELink, EFetch, EInfo, and BLAST APIs.
- UniProt tools cover UniProtKB, UniRef, UniParc, GeneCentric, Proteomes, and literature citations.Listed operations include retrieval, search, streaming, membership, database links, and citation access.
- Ensembl comparative-genomics APIs provide alignments, gene trees, homology, cross-references, and information resources.
- Ensembl sequence, transcript-haplotype, VEP, variation, and GA4GH APIs support sequence and variant-related workflows.