Source-linked AI summary
BixBench: a Comprehensive Benchmark for LLM-based Agents in Computational Biology
Ludovico Mitchener, Jon M Laurent, Alex Andonian, Benjamin Tenmann, Siddharth Narayanan, Geemi P Wellawatte, Andrew White, Lorenzo Sani, Samuel G Rodriques
TL;DR
BixBench addresses the lack of extensive benchmarks for open-ended bioinformatics analysis by combining real-world analytical scenarios with an agent evaluation framework. It evaluates frontier models on multi-step data exploration and interpretation, finding low open-answer accuracy and near-random multiple-choice performance.
Problem
Benchmarks for fully open-ended scientific exploration and rigorous bioinformatics analysis remain limited.
Method
BixBench combines 61 real-world analytical scenarios and 205 associated questions with a notebook-based agent framework supporting open-answer and multiple-choice evaluation.
Results
21% overall accuracy was achieved in open-answer evaluation, while multiple-choice performance remained only marginally above random when refusal was allowed.
Takeaways & Limitations
BixBench provides a resource for measuring and guiding development of AI capabilities for rigorous bioinformatics research.
Takeaways & Limitations
BixBench omits many important bioinformatics workflows, pipelines, statistical approaches, data types, and other parameters.
Abstract
from arXiv · showhide
Large Language Models (LLMs) and LLM-based agents show great promise in accelerating scientific research. Existing benchmarks for measuring this potential and guiding future development continue to evolve from pure recall and rote knowledge tasks, towards more practical work such as literature review and experimental planning. Bioinformatics is a domain where fully autonomous AI-driven discovery may be near, but no extensive benchmarks for measuring progress have been introduced to date. We therefore present the Bioinformatics Benchmark (BixBench), a dataset comprising over 50 real-world scenarios of practical biological data analysis with nearly 300 associated open-answer questions designed to measure the ability of LLM-based agents to explore biological datasets, perform long, multi-step analytical trajectories, and interpret the nuanced results of those analyses. We evaluate the performance of two frontier LLMs (GPT-4o and Claude 3.5 Sonnet) using a custom agent framework we open source. We find that even the latest frontier models only achieve 17% accuracy in the open-answer regime, and no better than random in a multiple-choice setting. By exposing the current limitations of frontier models, we hope BixBench can spur the development of agents capable of conducting rigorous bioinformatic analysis and accelerate scientific discovery.
1 Introduction
BixBench addresses the lack of benchmarks for open-ended scientific exploration by evaluating agents on realistic, multi-step bioinformatics analysis. It combines expert-curated scenarios, an evaluation framework, and an open-source coding environment, revealing substantial limitations in current frontier models.
- Open-ended scientific exploration and benchmarks for assessing it remain elusive despite progress on narrower research activities.
- BixBench comprises 61 real-world analytical scenarios pairing guiding questions with heterogeneous input data.Success requires understanding research questions, exploring data, performing multi-step analyses, and interpreting results.
- 21% accuracy was the best performance in the primary open-answer evaluation, while majority-voted multiple-choice performance was no better than random guessing.
- Primary Contributions: The benchmark contributes an expert-curated set of 61 scenarios and 205 open-answer questions for bioinformatics and computational biology.
- Primary Contributions: BixBench includes a framework for assessing AI agents in biological data analysis, including key metrics and calibrations.
- Primary Contributions: The project also provides an open-source framework for writing and executing Python, R, and bash code in a Jupyter notebook environment.
2 Related Work
Related benchmarks increasingly target practical scientific tasks, but existing systems and evaluations remain limited for ambiguous, open-ended biological analysis. BixBench is positioned to assess multi-step analytical capabilities without constraining tasks to fixed environments or tools.
- Scientific benchmarks now cover chemistry, biology research, reproducibility, code generation, and function calling across increasingly practical tasks.
- Existing benchmarks include literature review, DNA manipulation, protocol troubleshooting, recapitulation of published analyses, and bioinformatics coding.
- BixBench was developed for complex, ambiguous research tasks requiring challenging multi-step analytical capabilities without artificial constraints on environments, tools, or tasks.
3.1 Benchmark creation
BixBench was built from expert-generated analytical trajectories, standardized as reviewed capsules containing hypotheses, data, code, results, and metadata. LLM-generated questions were then human-reviewed and duplicate-filtered to produce benchmark tasks.
- Expert analysts assembled diverse bioinformatics analytical trajectories and reviewed capsules to produce 61 approved analytical scenarios.
- Capsule Creation: Each analysis capsule contains a hypothesis or research question, input data, analysis code, a result, a hypothesis-support assessment, and metadata.
- Capsule Creation: Analysts used Jupyter notebooks with Python, R, or bash and could install packages needed for their analytical trajectories.
- Task Generation: An LLM generated eight draft multiple-choice questions per capsule across two rounds, using the notebook, hypothesis, and result as context.
- Task Generation: Human experts reviewed and could approve, reject, or edit generated questions while consulting the original capsules.
- Task Generation: Duplicate filtering was repeated with LLM flagging and manual verification until no duplicates remained, yielding 205 questions across 61 capsules.
3.2 Benchmark Evaluation
BixBench evaluates agents in reproducible notebook environments where they iteratively inspect data, execute code, and submit answers. Performance is measured through open-answer judging or secondary-model multiple-choice selection, with refusal and voting variants.
- Evaluation Setup: Agents receive an empty Jupyter notebook, input data files, and questions, then freely plan and execute analyses in a standardized environment.
- Evaluation Setup: The Aviary-based framework supports controlled tool use and multi-step reasoning with reproducible software environments and access.
- Agent Tools: The agent can edit and execute notebook cells, inspect the workspace recursively, and submit an open answer.
- Agent Tools: Each code modification reruns the notebook, exposing tabular or plotted results and debugging tracebacks for iterative analysis.
- Evaluation: In open-answer evaluation, a judge LLM compares the submitted answer with the ground truth; in multiple-choice evaluation, a second LLM selects among answer options.
- Evaluation: Multiple-choice evaluation includes an optional “Insufficient information” refusal and majority voting across five runs.
4 Results
BixBench evaluates frontier agents on heterogeneous biological data requiring workspace navigation, coding, and multi-step analysis. The benchmark is challenging: Claude 3.5 Sonnet reached 21% open-answer accuracy versus 15% for GPT-4o, while MCQ voting remained near random under refusal conditions.
- Benchmark and evaluation: 61 analytical capsules contain heterogeneous data files and require agents to navigate workspaces, select libraries, and perform open-ended analyses.The benchmark includes 205 associated open-answer questions across these scenarios.
- Benchmark and evaluation: 1,220 trajectories were collected from five parallel iterations across two models and two image-generation modalities.Performance was calculated primarily as the fraction of correct answers across parallel runs and questions.
- Open-answer results: 21% versus 15% accuracy: Claude 3.5 Sonnet outperformed GPT-4o in the open-answer regime.Overall results showed that BixBench was highly challenging even with a custom agent scaffold and tools.
- Multiple-choice results: MCQ performance was close to random when models had access to a refusal option.Without refusal, performance was higher and above random, while majority voting produced no significant accumulation effect across five trajectories.
- Ablations: Image-generation restrictions did not significantly affect benchmark results.The vision ablation prevented agents from producing images or plots, whereas the refusal ablation allowed them to use them.
5 Discussion
BixBench provides a real-world benchmark for autonomous bioinformatics analysis, but current frontier models remain far from autonomous-bioinformatician performance. The benchmark is broad yet incomplete, and lacks a human baseline comparison.
- Contributions: BixBench contains 61 real-world analytical scenarios and 205 questions, supporting both open-answer and multiple-choice evaluation.The benchmark is intended to measure progress toward agents for scientific discovery through biological data analysis.
- Discussion of results: 21% overall open-answer accuracy indicates substantial development remains necessary to achieve an autonomous bioinformatician.Multiple-choice performance increased but remained only marginally above random, and removing abstention increased performance further.
- Implications: BixBench highlights a fundamental shortcoming in current LLM capabilities for rigorous bioinformatics research.The authors position the benchmark as a resource for guiding capability development and accelerating discovery across biology.
- Limitations: The benchmark omits important workflows, pipelines, statistical approaches, data types, and other parameters needed for comprehensive bioinformatics coverage.The authors describe a truly comprehensive benchmark as extraordinarily difficult to build.
- Limitations: A human baseline comparison was not prioritized, although the authors anticipate additional human experts would outperform the reported agents.The seed capsules were generated de novo by human experts.
A Appendix
The appendix provides supplementary benchmark visualizations and the agent prompt used to create analysis notebooks. The workflow directs agents to inspect files, plan and execute analyses, document results, and submit answers in JSON.
- Supplementary figures: Figure 6 depicts an analysis capsule containing expert-generated notebook code, input files, metadata, and reviewed question tasks.Questions are initially generated by an LLM and then reviewed and edited by experts.
- Supplementary figures: Figure 7 reports average accuracy per capsule across replicates, while Figure 8 reports average accuracy per individual question across replicates.These figures provide capsule-level and question-level views of benchmark performance.
- Agent workflow: The agent prompt instructs a staged workflow: list directory contents, load data, develop an analysis plan, execute it, and submit answers.The prompt emphasizes documenting code, checking outputs, interpreting results, and recording limitations or uncertainties.
- Answer submission: Agents are required to answer all four example questions in JSON using the submit_answer tool.The example prompt requests concise answers keyed by question number and precision to two decimal places when a question asks for a number.
- Implementation guidance: The prompt permits Python, R, or bash cells but advises avoiding plots in favor of tables and printed outputs.It also recommends small to medium-sized cells and checking each cell for successful execution.