Source-linked AI summary
BABE: Biology Arena BEnchmark
Junting Zhou, Jin Chen, Linfeng Hao, Denghui Cao, Zheyu Wang, Qiguang Chen, Chaoyou Fu, Jiaze Chen, Yuchen Wu, Ge Zhang, Mingxuan Wang, Wenhao Huang, Tong Yang
TL;DR
Biology benchmarks often omit the integrated reasoning researchers use to connect experimental results with contextual knowledge. BABE addresses this gap with research-derived, structured tasks that evaluate experimental reasoning across correlation types. The benchmark provides a diagnostic framework for identifying reasoning failure modes in biological AI systems.
Problem
Existing biology benchmarks rarely assess integrating experimental results with contextual background, despite its importance for biological researchers.
Method
BABE constructs benchmark tasks from peer-reviewed research papers and real-world biological studies, organizing them into strong- and weak-correlation question triplets.
Results
BABE provides a fine-grained diagnostic framework for assessing sequential multi-hop reasoning and parallel information extraction within single source documents.
Takeaways & Limitations
BABE moves biological AI evaluation beyond task-level competence toward research-level reasoning grounded in experimental evidence and context.
Abstract
from arXiv · showhide
The rapid evolution of large language models (LLMs) has expanded their capabilities from basic dialogue to advanced scientific reasoning. However, existing benchmarks in biology often fail to assess a critical skill required of researchers: the ability to integrate experimental results with contextual knowledge to derive meaningful conclusions. To address this gap, we introduce BABE(Biology Arena BEnchmark), a comprehensive benchmark designed to evaluate the experimental reasoning capabilities of biological AI systems. BABE is uniquely constructed from peer-reviewed research papers and real-world biological studies, ensuring that tasks reflect the complexity and interdisciplinary nature of actual scientific inquiry. BABE challenges models to perform causal reasoning and cross-scale inference. Our benchmark provides a robust framework for assessing how well AI systems can reason like practicing scientists, offering a more authentic measure of their potential to contribute to biological research.
1 Introduction
BABE addresses the underdeveloped evaluation of biological AI systems’ ability to integrate experimental results with contextual knowledge. It uses research-derived tasks spanning biological domains to assess experimental, causal, and cross-scale reasoning.
- Existing biology benchmarks rarely test integrated reasoning from experimental results and contextual background, instead emphasizing isolated tasks such as sequence classification or structure prediction.
- 45% of BABE questions have strong correlations and 55% have weak correlations.
- BABE derives tasks from peer-reviewed papers and real-world biological studies to reflect the complexity of biological research.
- BABE centers on integrating experimental results with contextual background to derive biological conclusions.
- BABE tasks demand causal reasoning and cross-scale inference.
- BABE covers major biological domains through diverse subfield studies, enabling evaluation across real-world biological research areas.
2 Related Work
Prior scientific and biology benchmarks assess domain knowledge, factual problem solving, or specialized tasks, but rarely capture integrated experimental reasoning over complex literature. BABE is positioned as a research-derived, multimodal benchmark addressing these gaps across biological domains.
- Scientific benchmarks evaluate domain knowledge and difficult problem solving, including expert-authored factual questions and advanced STEM or multimodal tasks.
- Biology-specific benchmarks commonly focus on sequence or structure prediction and generally do not require interpreting experimental data related to structures.
- Existing benchmarks often use simplified or summarized data rather than figures and datasets from published papers.
- Existing tasks generally do not require linking experimental results to contextual background.
- Biology benchmarks are often limited to single subfields rather than spanning multiple biological domains.
- BABE addresses these gaps with research-derived, multimodal tasks requiring reasoning similar to that of practicing biologists.
3 Approach
BABE structures questions from scientific sources into correlation-defined triplets that diagnose sequential and parallel reasoning. Its multi-stage pipeline combines expert item development with review, revision, and correctness verification.
- Problem Formulation: BABE evaluates compositional reasoning and quantitative content in scientific literature over a single source research document.
- Problem Formulation: Strong correlation represents sequential multi-hop reasoning in which a preceding answer is necessary input for a subsequent question.
- Problem Formulation: Weak correlation represents parallel independent extraction from largely distinct contexts, testing simultaneous maintenance of multiple contexts.
- Problem Formulation: The two correlation types diagnose error propagation and reasoning drift versus semantic interference during simultaneous retrieval.
- Data Construction: The benchmark pipeline integrates literature curation, expert-driven item development, and structured quality assurance.
- Data Construction: Senior experts reviewed relevance, factual fidelity, logical coherence, and answer correctness, with revision and second-round review required before inclusion.
4 Experiments
BABE performance varies across correlation settings and reasoning behaviors, while multi-trial inference improves scores with diminishing returns. Strong performance is associated with sustained deep reasoning, whereas excessive self-reflection and early reasoning bursts are linked to weaker outcomes.
- Overall Performance: 52.31 average score: OpenAI-GPT-5.1-high performs best overall, with 51.79 under strong correlation and 52.86 under weak correlation.
- Strong vs. Weak Correlation: 55.16 under weak correlation versus 49.05 under strong correlation: Gemini-3-Pro-Preview-Exp shows the clearest correlation-setting difference.
- Reasoning Behavior: Higher-performing models devote more inference steps to deep reasoning, indicating that BABE rewards resolving implicit or non-trivial dependencies rather than shallow pattern matching.
- Reasoning Behavior: Frequent self-reflection without corresponding deep-reasoning progress is associated with worse outcomes, consuming inference budget and increasing drift from relevant evidence.
- Reasoning Behavior: Sustained, relatively uniform deep reasoning is needed because solving BABE examples often requires revisiting premises, integrating implications, and maintaining multi-step constraints.
- Multi-Trial Inference: Multi-trial inference produces monotonic gains with diminishing marginal returns; frontier models converge around 30 points, while some mid-tier models exceed 35 points.
- Multi-Trial Inference: At least 4–6 inference trials are generally required for frontier models and 8+ for most non-frontier models.
5 Conclusion
BABE is presented as a rigorous benchmark for evaluating biology-focused LLM experimental reasoning. Its research-derived, structured design supports fine-grained diagnosis and aims to assess research-level reasoning.
- BABE evaluates whether LLMs can integrate experimental results with contextual background to derive scientifically meaningful biological conclusions.
- The benchmark is constructed entirely from peer-reviewed research papers and real-world biological studies, preserving the complexity and interdisciplinarity of scientific inquiry.
- Structured question triplets with strong and weak correlation relationships diagnose sequential multi-hop reasoning and parallel information extraction within one source document.
- BABE is intended as a testbed for biologically grounded LLMs, deep research agents, and multimodal scientific reasoning systems.
A Example Questions of BABE
BABE includes example questions spanning diverse biological systems, methods, measurements, and experimental contexts. These examples require interpreting source-specific evidence and biological concepts across multiple domains.
- Another example presents a CRISPR-Csm application that targets RNA for temporary gene-expression regulation and aims to delete RNA fragments.
- A separate example examines how structural features between linked enzyme channels can guide linker design for artificial multi-enzyme complexes.
- The examples also cover X-chromosome inactivation, colony-level biocomputing, dopamine-neuron activity during choices, and gene-expression dynamics.
- One example asks readers to infer replication fork directionality from Okazaki-fragment sequencing data.