Source-linked AI summary
BixBench3: Benchmarking AI agents on research-study-scale computational biology tasks
Zane Koch, Asmamaw T. Wassie, Javier Valdes-Aleman, Jason Lee, Michaela M. Hinks, Samuel G. Rodriques, Andrew D. White, Jon M. Laurent
TL;DR
AI agents’ ability to execute complete computational biology studies has not been systematically evaluated. BixBench3 benchmarks end-to-end analysis from raw data to publication-matched artifacts, finding that the best model reproduced 48% of requested artifacts while performance varied sharply with task demands.
Problem
AI agents increasingly perform parts of scientific workflows, but their ability to execute complete computational biology studies remains uncertain.
Method
BixBench3 evaluates agents on 20 published-study tasks spanning 17 assay types and 9 domains by comparing generated artifacts with original-study artifacts.
Results
Across 13 frontier models, GPT 5.6 Sol scored highest at 0.48, with performance varying by scientific domain, data size, and analysis depth.
Takeaways & Limitations
Top-performing agents approached research-study-scale biology but reproduced only about half of requested artifacts, struggling particularly with large datasets and long analysis chains.
Takeaways & Limitations
Because tasks prescribe the methods and analyses, BixBench3 does not test whether agents can choose which biological questions or analyses are worth pursuing.
Abstract
from arXiv · showhide
Artificial intelligence (AI) promises to accelerate biological research by automating computational analyses. Yet the ability of AI agents to carry out computational biology at the scale of complete research studies has not been systematically evaluated. Here we introduce BixBench3, a benchmark that measures the capacity of AI agents to process raw biological data through to scientific results. We designed BixBench3 tasks to mirror the delegation of work from a scientist to an agent: the scientist chooses the research question and high-level methods, then delegates implementation of all analyses to the agent. In each task, an agent receives a research objective, methodological guidance, and raw data derived from a published scientific study, and must execute a sequence of analyses to achieve the research objective. The data artifacts resulting from these analyses - such as peak call matrices or differential expression tables - are programmatically graded against the corresponding artifacts generated and reported in the original study. Across 20 BixBench3 tasks encompassing the generation of 138 unique artifacts, we find that 13 frontier models achieve scores ranging from 0.00 for Gemini 3.1 Flash Lite to 0.48 for GPT 5.6 Sol. Agents perform worse on tasks with larger raw datasets (0.36 on tasks with <100 GB versus 0.10 on tasks with >100 GB) and on analyses requiring more sequential steps (0.36 at 1-2 steps vs 0.24 at 3+). On average, agents use 6.8 hours, 102 million tokens, and $43 to complete each task, with the longest attempts consuming 24 hours, 1.07 billion tokens, and $525. Notably, the highest-scoring agents used fewer tokens and were cheaper than less performant options. These results reveal that LLMs vary substantially in their ability to (1) execute multiple sequential analysis steps coherently, (2) manage large quantities of raw data, and (3) work across scientific domains.
1 Introduction
BixBench3 evaluates whether AI agents can execute complete, multi-step computational biology studies from raw data to research artifacts. It combines long-horizon tool use with biology-specific analytical demands and grades outputs against original-study artifacts.
- Benchmark design: BixBench3 gives agents a research objective, methodological guidance from a published biology paper, and the study’s raw data.Agents must achieve the objective by analyzing the provided data.
- Benchmark design: 20 studies spanning 17 assay types and 9 scientific domains yield 138 total graded artifacts.Performance is assessed by comparing agent-produced artifacts with artifacts from the original publication.
- Benchmark design: BixBench3 tests agents’ capability to construct and execute long analysis pipelines following specific instructions.This distinguishes it from benchmarks focused on biological question answering or isolated bioinformatic analyses.
- Relation to prior benchmarks: BixBench3 combines the extended, multi-step challenge of long-horizon agent benchmarks with the domain specificity of biology and bioinformatics benchmarks.It therefore sits at the intersection of tool-based planning and execution benchmarks and scientific analytical-competence benchmarks.
2 Results
BixBench3 evaluates 13 frontier models across 20 research-scale computational-biology tasks and finds substantial variation in artifact-level performance, with accuracy declining for deeper analyses and larger datasets. Performance also varies across scientific domains and entails large differences in cost and computational efficiency.
- Overall performance: 48% was the highest average score across tasks, achieved by GPT 5.6 Sol, followed by Kimi K3 at 0.47 and GLM 5.2 and Claude Opus 4.8 at 0.46.The evaluation comprised 13 models, 260 completed runs, and 1,794 graded artifact evaluations.
- Analysis depth: 0.24 was the mean binary artifact pass score for depth-3+ artifacts, compared with 0.30 for depth-1 and 0.44 for depth-2 artifacts.Six models exhibited a significant decrease in performance at greater analysis depths.
- Task difficulty: 0.10 was the average score for tasks with more than 100 GB of raw input data, compared with 0.34 for 50–100 GB and 0.37 for less than 50 GB.Models scored highest on microbiology, evolution and ecology, and immunology tasks, while struggling on neuroscience and cell biology tasks.
- Cost and efficiency: 367-fold was the difference in average cost per task across models, ranging from $0.35 to $129.14.On average, each task attempt used 102 million tokens, 6.8 hours, and 695 model turns; highest performance occurred at intermediate levels of these measures.
- Failure modes: ρ = −0.92 was the correlation between total failure-mode tags and mean task score across models.The three highest-scoring models received 31–51 total tags, whereas the two lowest-scoring models received 102–117.
3 Discussion
BixBench3 shows that frontier AI agents are approaching research-study-scale biology tasks, but performance varies sharply and declines with larger datasets and longer analysis chains. Higher performance does not require higher computational cost, while programmatic grading introduces method-specific limitations.
- Agent capabilities: 13 frontier models approached research-study-scale biology tasks, with the best models scoring approximately 0.5 and performance varying sharply between tasks.BixBench3 is described as the biology benchmark with the longest reported horizon by run time and token usage.
- Failure modes: Agents scored lowest on tasks with the largest raw datasets and on analyses at depth 3+ in the dependency chain.Large datasets and long analysis chains may extend the horizon over which errors introduce challenges.
- Cost and performance: Kimi K3 cost approximately 58% less than other models with comparable performance while ranking as the second-highest-performing model.This demonstrates that maximum performance did not necessitate maximum cost.
- Cost and performance: Models scoring at least 0.40 used 28–179M tokens and approximately 271–1,149 model turns, compared with 8–285M tokens and approximately 107–1,894 turns for models scoring below 0.40.Greater computational expenditure did not lead to better performance.
- Benchmark limitations: Programmatic grading requires each task prompt to specify the method, because scientifically valid alternatives may receive low artifact scores when their outputs differ from the published artifact.Differences may involve format, scale, or other respects.
4 Methods
BixBench3 tasks give agents structured research objectives, method guidance, raw data, and exact artifact requirements, then evaluate outputs against published results. Artifact-level metrics are combined into pass/fail outcomes that determine task scores, with eligibility requiring accessible data, sufficient ground truth, and feasible execution.
- Task design: Each task prompt specifies workspace instructions, a biological objective and raw data, method parameters and filtering rules, and required output paths and formats.Method guidance mirrors the original paper without revealing its results.
- Task design: Artifacts form a dependency graph from raw data to downstream analyses, and agent outputs are programmatically compared with corresponding published artifacts.The resulting artifact scores are aggregated into task-level performance.
- Artifact grading: Four artifact metrics assess identifier recovery, numerical agreement, categorical-label agreement, and genomic-interval recovery, depending on artifact format and biological content.These metrics include row or column F1, Lin’s CCC, macro F1, and overlap F1.
- Artifact grading: Artifact scores multiply the applicable metrics, while missing, empty, or unreadable files receive zero.For example, RNA-seq count matrices combine gene recovery, sample recovery, and expression-value agreement.
- Artifact grading: Artifacts pass at ≥0.8, and a task’s score is the proportion of artifacts that pass.The 0.80 threshold was based on domain-expert review linking scores to preservation of the artifact’s main biological meaning.
- Task selection: Papers qualified when raw data were publicly accessible, at least four ground-truth artifacts were available, and the analysis fit the benchmark’s 32-CPU, 500-GB, 24-hour budget.An agentic pipeline screened papers before human review.
Statement of Contributions
The project was conceived and designed by Z.K., A.T.W., and J.M.L., with J.M.L. supervising the project. Z.K. led benchmark task and infrastructure design, while collaborators coordinated expert evaluation, prepared the initial manuscript materials, and all authors contributed to the final manuscript.
- Z.K., A.T.W., and J.M.L. conceived and designed the overall project, with J.M.L. supervising it.
- Z.K. designed the benchmark tasks and infrastructure, while J.V.-A., J.L., and Z.K. coordinated with scientific domain experts for evaluation.
- Z.K. and J.M.L. prepared the initial manuscript, figures, and tables, and all authors contributed to the final manuscript version.
- M.M.H., S.G.R., and A.D.W. supervise research at Edison Scientific, Inc.
Supplementary Material
The supplementary material details artifact-level performance, expert-rating comparisons, model task-score similarities, task performance across data and assay types, and token usage. These analyses extend evaluation across the 20 BixBench3 tasks and 13 models.
- Artifact performance: Supplementary Figure 1 reports programmatic artifact scores for each artifact across all 20 BixBench3 tasks and 13 evaluated models.Colored points show model-specific scores, while grey boxplots summarize score distributions across models within each task.
- Expert comparison: Supplementary Figure 2 compares programmatic artifact scores with domain-expert reproduction ratings for 25 artifacts produced by GLM 5.2.Artifacts are grouped by expert reproduction rating, with boxes, medians, whiskers, and individual points showing score distributions.
- Model similarity: Supplementary Figure 3 shows pairwise Spearman correlations between models’ per-task scores across the 20 BixBench3 tasks.Rows and columns are ordered by overall mean task score; asterisks mark two-sided significance levels, and dashes indicate undefined correlations.
- Data and assay types: Supplementary Figure 4 presents mean task scores by model across broad data types and specific assay types.Parenthetical values indicate task counts, and tasks may contribute to multiple categories.
- Token usage: Supplementary Figure 5 summarizes mean total processed tokens and token-category usage across models.Total processed tokens include full-rate input, provider-reported cache-write input, cache-read input, and output tokens; both panels use logarithmic axes.
A Appendix · A.1 Source papers
Appendix A.1 identifies the source papers underlying the 20 tasks in the BixBench3 benchmark.
- A.1 Source papers: 20 BixBench3 tasks are linked to source papers in Appendix A.1.The provided passage is the caption for Table 1, which lists the source papers for the benchmark’s 20 tasks.
A.2 Claude Opus 5 output-format errors
Claude Opus 5 ranked seventh overall, largely because it violated specified artifact-output formats on Tasks 5, 19, and 20. On Task 5, altered column names caused the grader to treat required columns as missing.
- Overall impact: 7th overall (score 0.406) was Claude Opus 5’s ranking and score, with format violations contributing substantially.The violations occurred on Tasks 5, 19, and 20.
- Format errors: On Tasks 5, 19, and 20, Claude Opus 5 violated the specified artifact-output formats.These were instruction-following errors that most other models did not make.
- Task 5 example: On Task 5, replacing required names such as col0_gene_id and col0_tpm with Col-0 and Ct-1 caused the grader to treat those columns as missing.The alternative labels did not satisfy the artifact output contract.
A.3 Expert review of tasks
Domain experts reviewed 25 artifacts produced by GLM 5.2 to validate the grading and task structure. They rated each artifact from 1 to 5 according to how faithfully it recovered the biological meaning of the corresponding published artifact.
- Expert review: 25 artifacts were selected for domain-expert review from 171 artifacts produced by GLM 5.2, spanning the range of artifact scores.The review was intended to validate the grading and task structure.
- Expert review: 1 to 5 was the integer reproduction-rating scale assigned by experts to each reviewed artifact.Ratings reflected how faithfully an agent-produced artifact recovered the biological meaning of its corresponding published artifact.
- Expert review: Table 2 defined the criteria used by domain experts to assign reproduction ratings.The criteria applied to reviewed artifacts and were expressed as integer ratings from 1 to 5.
A.4 Process-judge failure modes
The process judge assigns closed-vocabulary failure-mode tags to agents’ runs. The supplied definitions cover unusable environments, misinterpreted inputs, synthetic or placeholder outputs, and premature termination.
- Environment setup failure means required tools, packages, references, or paths remained unusable and prevented or materially blocked the intended analysis.
- Input misinterpretation covers misread staged inputs, confused sample identities, or applying an analysis intended for a different data type.It concerns incorrect interpretation of supplied data rather than later analysis-parameter choices.
- Synthetic or placeholder output covers dummy, empty, formulaic, or fabricated values not derived from real staged data, while premature termination leaves at least one required artifact unwritten.Synthetic output is tagged whether fabrication is explicit or conceals a failed or incomplete analysis; termination can follow budget or time exhaustion or a premature completion declaration.
A.5 Example task prompt
The example task prompt directs an agent to perform an end-to-end computational biology analysis of raw sequencing data, following specified methods and producing reproducible, precisely formatted outputs. The task investigates how TCER-1 loss affects small-RNA regulation and mRNA expression in Caenorhabditis elegans, emphasizing WAGO-associated 22G-RNAs and target transcripts.
- The agent must complete the entire computational biology task end to end and produce the requested output.
- All work must use ./data, ./work, and ./outputs, with input data kept read-only, modified copies placed under ./work, and analysis outputs written under ./outputs.
- 0.00 seconds is not the budget: the task allows a maximum wall-clock budget of 86400 seconds before termination.
- The agent must follow named tools, parameters, thresholds, and databases, document unspecified defaults, and favor scripted, reproducible steps with pinned software versions when possible.
- Required artifacts must use exact paths and formats, headers and column names where specified, row keys first, one row per entity, and no commentary rows or notes in data files.
- The biological objective is to characterize how TCER-1 loss changes endogenous small-RNA regulation and mRNA expression in Caenorhabditis elegans, emphasizing WAGO-associated 22G-RNAs and target transcripts.