BioAgent Bench: An AI Agent Evaluation Suite for Bioinformatics

1. Introduction

• Bioinformatics workflows chain tools, handle heterogeneous file formats, and interpret subfield-specific intermediate outputs, making rigorous evaluation difficult.Early parameter and algorithm choices can substantially influence final results, while simplified question-answering or code-generation evaluations miss nuanced agent behavior and failure modes.Source: §1 · p. 1
• Patient sequencing data and clinical metadata can be too sensitive to share with third-party model providers, motivating agents that run within secure institutional environments.The paper cites privacy and intellectual-property constraints as barriers to sharing such data.Source: §1 · p. 1
• Frontier proprietary models can solve most benchmark tasks without extensive custom scaffolding, but controlled perturbations and repeated runs reveal performance drops and inconsistencies.The authors conclude that pipeline completion alone can overestimate reliability.Source: §1 · p. 1, Figure 1 · p. 2
• BioAgent Bench provides curated practical tasks, compares closed- and open-weight models, and evaluates intermediate steps, output validity, and robustness under controlled perturbations.Its overview shows agents receiving task prompts, input data, and reference data before an LLM judge compares outputs with ground truth.Source: §1 · p. 2, Figure 1 · p. 2

2. Preliminaries

• A bioinformatics pipeline transforms raw biological data into analyzed outputs, using sample-specific input data and shared reference data such as genomes or annotation.The distinction separates the material being processed from the shared baseline used for alignment or annotation.Source: §2 · p. 2
• An evaluation task has defined inputs and a success criterion; a trial is one execution scored against ground truth, with a transcript recording actions and intermediate steps.The outcome is the final result produced by executing the entire pipeline.Source: §2 · p. 2
• An evaluation harness executes tasks, captures transcripts, applies graders, and aggregates trials, whereas an agent harness enables a model to act and use tools.In this framework, an agent consists of a model plus its agent harness.Source: §2 · p. 2
• BioAgent Bench is an evaluation suite of end-to-end bioinformatics tasks designed to measure successful and robust pipeline automation across trials and controlled perturbations.The paper defines an evaluation suite as a collection of tasks that share a broad goal.Source: §2 · p. 2

3. Benchmark Design

• The benchmark covers bulk and single-cell RNA-seq, comparative genomics, variant calling, metagenomics, viral metagenomics, transcript quantification, and experimental evolution.Tasks are framed as pipelines requiring tool orchestration, file handling, and structured reporting.Source: §3 · p. 2
• Concrete ground-truth outputs, typically CSV files, support automated evaluation and distinguish the benchmark from single-step question-answering frameworks.End-to-end framing is intended to reveal agent behavior and failure modes.Source: §3 · p. 3
• Pipeline choices can yield different or conflicting biological conclusions from the same data, especially for small effects and analyses sensitive to modeling decisions.The paper points to design, normalization, batch correction, and statistical assumptions as choices that can affect conclusions.Source: §3 · p. 3
• BioAgent Bench is designed more like a software-engineering benchmark than a biology data-analysis benchmark, targeting common tasks that can be represented as code and tool-calling pipelines.The stated purpose is to support model post-training uses such as reinforcement learning, distillation, and harness changes.Source: §3 · p. 3
• Runtime below four hours and memory at or below 48GB restrict eligible datasets, favoring smaller organisms and directly provided references.These constraints exclude large-organism workflows and tasks such as finding, downloading, and staging large references, but make large-scale benchmarking more feasible.Source: §3 · p. 3
• The ten curated tasks use Python, R, or bash; seven require tool interaction, four have binary-verifiable outputs, and each pairs a goal-and-format prompt with execution and evaluation files.Table 1 summarizes the tasks by implementation language, tool-call requirement, and verifiability.Source: §3 · p. 3, Table 1 · p. 3

4. Experimental Setup

• Each evaluation run is sandboxed and receives a system prompt, input files, and task instructions specifying a goal and expected output format; generated artifacts and the outcome file go to the grader.The system prompt asks agents to produce artifacts at pipeline steps such as quality control, trimming, and assembly.Source: §4 · p. 3

4.1. Model Settings

• The study interprets performance as the combined capability of a model and its harness, not as a direct comparison of individual LLMs.It evaluates frontier open- and closed-weight models in Claude Code, Codex CLI, and OpenCode, with internet access and high reasoning effort when available.Source: §4 · pp. 3–4, §4.1 · p. 4

4.2. Grader Logic

• An LLM grader scores outputs and pipeline steps against a rubric, accommodating multiple valid workflows and variable step counts that would make hard-coded grading impractical.Its inputs include expected and agent outcome tables and a trace of folders and file paths; reported measures include steps completed, estimated steps to completion, final artifact reached, results match, and applicable F1 score.Source: §4.2 · p. 4

4.3. Evaluation Suite

• The suite tests multiple trials, prompt bloat, corrupted inputs, and decoy data; its primary completion-rate metric is the percentage of required steps that pass checks for completion and requested artifact format.The perturbations assess distraction and instruction overload, detection of corrupted files, and whether agents ignore decoys.Source: §4.3 · p. 4

Single-run pipeline completion

• 100% completion for Claude Opus 4.5 and above 90% for Gemini 3 Pro, GPT-5.2, and Sonnet 4.5; GLM-4.7 led open-weight models at 82.5%.The top results used Codex CLI; other open-weight models ranged down to 65%.Source: §5 · p. 4, Figure 2 · p. 5

Planning ability evaluation

• Plan quality correlated with overall pipeline completion (Pearson R = 0.61), but the relationship was not deterministic and some weaker-planning models still completed pipelines successfully.Gemini-Pro-3 is one example; the authors suggest domain knowledge is needed, while agentic capability remains a bottleneck for open-weight models.Source: §5 · p. 5, Figure 3 · p. 5

Qualitative failure modes

• Manual inspection found some open-weight models stuck in repeated error-correction loops or terminating early, while frontier models more often recovered and completed pipelines.The authors leave systematic analysis of these behaviors to future work.Source: §5 · p. 5

Robustness across multiple runs

• Across four GPT-5.2 Codex CLI trials, mean Jaccard overlap was 0.43 and Pearson correlation was 0.73, indicating considerable variability in final results.Open-weight models were excluded because low pass@1 left too few completed trials for reliable stability metrics; the authors hypothesize tool nondeterminism and differing pipeline decisions contribute to variability.Source: §5 · p. 5

Perturbation outcomes

• Agents identified corrupted inputs in 7/10 tasks, erroneously used decoy files in 2/10, and completed 28% fewer steps under prompt bloat than without perturbation.Table 2 combines perturbation outcomes with run-to-run stability measures.Source: §6 · pp. 5–6, Table 2 · p. 6

Corrupted data

• Some agents failed to detect corruption and continued analysis, while others flagged it but tried alternative analyses or downloads and continued the pipeline.Examples included differential-expression analysis on corrupted data and attempts to route around flagged corruption.Source: §6 · p. 6
• Corruption sometimes caused early exits, appropriately or through cascading failures; superficially valid but semantically corrupted inputs could instead yield completed, scientifically invalid analyses.In the single-cell task, the agent correctly identified corruption and stopped; behavior depended on the corruption type and severity.Source: §6 · p. 6

Decoy files

• Decoy failures included suffix-based file selection that admitted an unintended organism and use of a viral instead of the required bacterial reference database.Both cases were consistent with relying on surface cues rather than configuring tools for the biological context; other decoys were discarded correctly.Source: §6 · p. 6

Prompt bloat

• Under prompt bloat, some agents repeatedly restated or superficially reformulated instructions, then terminated without producing substantive intermediate artifacts.These symptoms were most evident in tasks with complete completion degradation.Source: §6 · p. 6

7. Discussion

• High completion rates show agents can interpret goals, choose tools, manage environments, and recover from routine errors, yet pipeline completion can substantially overestimate reliability.The authors describe frontier agents as effective workflow assistants for common analyses without substantial custom engineering.Source: §7 · p. 7
• Open-weight models remain behind frontier closed models, with greater variability and more frequent failures to produce requested final artifacts.Plan ratings correlate positively with completion, but the relationship is not deterministic.Source: §7 · p. 7
• Correct high-level pipeline construction does not guarantee correct step-level reasoning, making completion a necessary but insufficient criterion, especially for clinical use.For sensitive applications, the discussion asks whether agents can detect when not to proceed and justify choices using data and context.Source: §7 · p. 7
• Locally deployable agents can be preferable for sensitive workflows even when they underperform on aggregate benchmarks, because they support stronger governance and reduce organizational risk.The cited settings include patient-derived sequencing data, proprietary references, and unpublished intellectual property.Source: §7 · p. 7

8. Limitations

• LLM grading can vary with rubric wording, trace verbosity, and artifact presentation, potentially rewarding plausible-looking trials with incorrect step-level reasoning.Grading can also introduce run-to-run inconsistency in the number of accepted steps.Source: §8 · p. 7
• Runtime below 4 hours and RAM capped at 48GB improve reproducibility but narrow tasks and datasets, favoring smaller organisms and well-packaged inputs.Large genomes, heterogeneous cohorts, messy metadata, long-running pipelines, and sourcing external references are underrepresented or excluded.Source: §8 · p. 7
• Robustness perturbations use one trial per task and condition, limiting uncertainty and variance comparisons, while their coverage remains narrow relative to practice.Some corruptions are obvious malformed files; more realistic, safety-relevant cases may remain syntactically valid but biologically implausible.Source: §8 · pp. 7–8

9. Related Work

• General agent benchmarks cover interactive environments, tool and API use, and software engineering; BioAgent Bench’s closest relation is SWE-bench (Deng et al., 2025).Examples include AgentBench (Liu et al., 2024), a 73-API tool-use benchmark, and ToolBench (Qin et al., 2024b;a; Guo et al., 2024).Source: §9 · p. 8
• Biomedical tasks are harder to evaluate automatically because candidate solutions may require wet-lab or clinical validation, and biological noise can make multiple pipelines reasonable.Success may be better characterized by decision quality under constraints than by one correct output.Source: §9 · p. 8
• Biomedical benchmarks span multi-domain prediction, practical biology skills, single-cell analysis, computational-biology scenarios, and open-ended data-driven research (Miller et al., 2025; Laurent et al., 2024; Liu et al., 2026; Mitchener et al., 2025; Panigrahi et al., 2026).The cited examples include BioML-bench, LAB-Bench, BixBench, and HeurekaBench.Source: §9 · p. 8
• Unlike benchmarks centered on data analysis or broad research text responses, BioAgent Bench emphasizes multi-step pipelines across bioinformatics domains and computational environments.Source: §9 · p. 8

10. Conclusion

• BioAgent Bench evaluates realistic workflows requiring tool orchestration, artifact production, and structured outputs, while exposing shallow file selection, weak input validation, and distraction sensitivity.The conclusion reframes evaluation around reliable, well-grounded decisions throughout the pipeline, not correct outputs alone.Source: §10 · p. 8
• Frontier agents complete canonical pipelines with high success rates without heavy scaffolding, but that success coexists with brittle step-level behavior.Source: §10 · p. 8
• Future work will expand task and dataset diversity, add external-reference sourcing tasks, and strengthen robustness evaluation and automated scoring.The authors aim to integrate robustness into primary metrics and expect the suite to accelerate locally deployable agents for privacy-constrained workflows.Source: §10 · p. 8

• BioAgent Bench is intended to improve evaluation for practical bioinformatics use cases.It is also intended to support private, locally deployable agentic systems using open-weight models.Source: Impact Statement · p. 8
• Standardized benchmarking can track the capabilities gap between open-weight models and the closed frontier.Source: Impact Statement · pp. 8–9
• The benchmark may support improvement of agentic behavior through fine-tuning, distillation, and reinforcement learning on verifiable multi-step tasks.Source: Impact Statement · p. 9

A.1. System prompt

• The system prompt directs agents to use the designated mamba environment, inspect files, organize outputs by stage, and stop after saving final results.Source: §A.1 · p. 12

A.2. Task prompts

• Task prompts specify end-to-end analyses and concrete output formats for tasks including differential expression, variant identification, and metagenomics.Source: §A.2 · pp. 12–13

A.3. Task descriptions

• Task descriptions cover varied datasets, including simulated family variant data, Candida expression samples, evolved E. coli lines, and dolphin viral sequencing reads.Source: §A.3 · p. 13

A.4. Grading logic prompt

• The grading prompt evaluates pipeline progress and final-result validity against the expected artifacts, prompt, and truth data.It prioritizes pipeline completion, counts steps only when expected artifacts are present, and returns completion and result-match metrics.Source: §A.4 · p. 14

A.5. Harness evaluation

• Table 3 reports completion rates by model and task across different harnesses.Source: §A.5 · p. 14, Table 3 · p. 15

A.6. Robustness metrics

• Robustness metrics use Jaccard overlap for categorical results and Pearson correlation for numerical values across trials.Source: Table 4 · p. 15, §A.6 · p. 15

A.7. Robustness case study

• Four transcript-quant trials used the same Salmon indexing-and-quantification workflow but differed in optional flags, with no consistent configuration across runs.Two runs were baseline; one enabled --keepDuplicates during indexing, and one enabled --gcBias and --seqBias during quantification.Source: §A.7 · p. 15

A.8. Perturbation settings

• Perturbation tests use prompt bloat, corrupted inputs, and decoy files; examples include replacing many FASTQ bases with N and inserting unrelated sequences or references.Source: §A.8 · pp. 16–18