PaperlayerView exact revision on arXiv

Source-linked AI summary

BIABench: Evaluating AI agents on real-world bioimage analysis tasks

Zixuan Pan, Davide Panzeri, Lukas Johanns, Marilin Moor, Yu Zhou, Hedi Peterson, Yiyu Shi, Jianxu Chen

arXiv:2609.34274v1cs.AIcs.CL

TL;DR

Bioimage analysis agents have not been systematically tested on complete real-world analyses, which require handling large images and choosing methods and tools. BIABench reconstructs published studies as benchmark tasks and evaluates outputs and process; agents handled routine tasks better than some tasks involving added dimensions or time, and repeated-run reliability remained a concern.

  • Problem

    No benchmark systematically tests whether agents can perform real-world bioimage analyses end to end, despite the need to handle complex data and select an analysis strategy.

    Source: §1 · p. 2

  • Method

    BIABench reconstructs 16 published-study tasks and scores required outputs against ground truth alongside rubric-based judgments of analysis process.

    Source: §1 · p. 2

  • Results

    Current agents complete routine analyses but remain unreliable on more complex tasks, and neither specialization, stronger models nor detailed instructions reliably close the gap.

    Source: §6 · p. 16

  • Takeaways & Limitations

    The study identifies data complexity, reliability and self-checking as gaps in current agents’ progress toward expert-level bioimage analysis.

    Source: §1 · p. 2, §6 · p. 16

Abstract

from arXiv · show

Artificial-intelligence (AI) agents hold promise for automating bioimage analysis, yet no benchmark evaluates whether they can carry out real-world analyses end to end. Such analyses are hard for agents because 2D images, 3D volumes and time-lapse sequences are often too large to read as context, so an agent must choose and run an analysis through code, specialized software and rendered views. Published studies make this capability testable, because each pairs raw images with a peer-reviewed result. We introduce BIABench, a benchmark of 16 tasks reconstructed from published biological studies that retain their scientific questions, imaging data and ground truth. The tasks span eleven analysis subtasks and modalities from H&E histology to single-molecule localization microscopy. Each submission receives an outcome score, which compares the output files with the ground truth using field-standard metrics, and a process score, in which a vision-language model judges method choice and quality control against an expert-written rubric. We evaluated general-purpose and biology-specific agents across several language models, with repeated runs of every task. Routine two-dimensional tasks were solved well, but on some tasks that added a third dimension or a time axis no agent scored above 0.19. Neither biological specialization, stronger models nor detailed expert instructions closed this gap. The agents were also unreliable, with scores varying more between repeated runs of one agent than between different agents, and without ground truth a correct run could not be told from a wrong one by its process score or by the time spent. Released openly with its data and code, BIABench provides a verifiable framework for evaluating, and eventually training, agents for reliable long-horizon bioimage analysis.

1 Introduction

End-to-end bioimage analysis remains untested in agent benchmarks despite its reliance on specialized tools and handling large, complex images. BIABench reconstructs published studies as tasks and evaluates both the results agents produce and the process they use.

1 Introduction

  • Current benchmarks test individual analysis steps, leaving untested whether agents can choose and execute a complete analysis from raw images to a study’s conclusion.Agents must work through computation, specialized software and rendered views because bioimage data can be too large or structurally complex to supply directly as context.Source: §1 · p. 2
  • BIABench turns published studies into tasks retaining their scientific questions, imaging data and reported outputs as ground truth.Each task pairs raw images with a biologist’s request for the study’s key readout, a recipe that can be extended to new studies.Source: §1 · p. 2
  • The 16-task benchmark spans eleven subtasks and 2D, 3D and time-lapse data, with outcome scores for required files and rubric-based process scores for supporting artifacts.Its workflow withholds ground truth from agents; the suite includes modalities from H&E histology to single-molecule localization microscopy.Source: §1 · p. 2, Figure 1 · p. 3
  • The study evaluates general-purpose and biology-specific agents across language models and instruction levels to characterize gaps in data complexity, reliability and self-checking.The benchmark lets agents choose methods and tools, including GUI software, through an interface covering six agents of three kinds.Source: §1 · p. 2

2 Related work

Prior benchmarks assess individual bioimage-analysis steps, agent reasoning, or other scientific workflows, but do not test agents choosing tools and producing a study’s physical measurement from real microscopy. BIABench combines study-derived tasks, ground-truth outcome metrics and a separate process score.

2 Related work

  • General-purpose agent benchmarks score task outputs or actions in controlled environments, and BIABench adopts shared interfaces and grading of required files while targeting real microscopy measurements.Examples include repository tests, verifiable answers, tool use and computer-use tasks.Source: §2 · p. 4
  • Bio-focused agents span hypothesis generation, biomedical analysis and bioimage software interfaces, but BIABench evaluates both biology-specific and general-purpose agents across tool-use, command-line and GUI classes.The evaluated bioimage agents include systems for napari and Fiji/ImageJ, segmentation routing, model-zoo tools and microscopy skills.Source: §2 · p. 4
  • Task-specific bioimage benchmarks score single steps such as segmentation, tracking, in-silico labeling or quality control, whereas BIABench scores the complete analysis at the study endpoint.Its process rubric covers stages from loading raw data through reporting.Source: §2 · p. 4
  • Existing biological-agent benchmarks score text answers, computational analyses, or trained-model predictions; cited end-to-end bioinformatics artifact benchmarks still lack microscopy tasks.ReX-MLE, BioXArena and BioML-bench ask agents to train models and score held-out predictions.Source: §2 · pp. 4–5
  • BixBench3 grades artifacts regenerated from published omics studies but prescribes methods, while BIABench leaves analysis choice to the agent and repeats each configuration–task pair three times.BixBench3 reports that frontier models reproduced fewer than half of the artifacts; its authors note that prescribed methods do not test which analysis to run.Source: §2 · p. 5
  • BiomniBench uses judge-derived process scores for tabular omics, while BIABench separately reports ground-truth outcome metrics and judged process scores for executed microscopy analyses.BIABench also includes GUI and domain-specialized agents, unlike BiomniBench’s coding-harness tasks.Source: §2 · p. 5

3 The BIABench benchmark

BIABench reconstructs published-study tasks, specifies agent instructions and execution conditions, and scores both output correctness against ground truth and process quality using a rubric-based judge.

3.1 Tasks

  • The 16 tasks inherit published studies’ measurement goals and ground truth, spanning eleven analysis subtasks, varied modalities and 2D, 3D and time-lapse acquisitions.Difficulty ranges from standard 2D pipelines to specialized methods and multi-subtask or demanding time-lapse analyses; tool choice is left to the agent.Source: §3.1 · pp. 5–6
  • Each task has brief and detailed instructions: a biologist’s request and an expert protocol, respectively, with exact output formats and a prohibition on retrieving the source study.The detailed protocol sketches recommended preprocessing, detectors or segmenters and parameter hints while leaving implementation to the agent.Source: §3.1 · p. 6

3.2 Agent integration

  • Agents run through wrappers with read-only inputs, writable outputs and a four-hour external wall-clock limit, across six agents in tool-use, coding command-line and GUI classes.Runs reaching the limit are scored on files already written; native per-step and per-tool policies remain part of the agent under test.Source: §3.2 · p. 6

3.3 Scoring

  • Task-specific outcome metrics compare required outputs with ground truth on a 0–1 scale, and submissions without output files score zero.The metrics include Dice, tracking measures, spot-detection measures and task-specific statistics; ground-truth replay reaches approximately 1.0 on tasks with ground-truth files.Source: §3.3 · p. 7
  • The process score is a severity-weighted fraction of rubric items passed across the pipeline, based on saved artifacts and reported alongside—but not used for—rankings.Unknown items are excluded from the denominator, and the undecidable fraction is reported separately as an auditability measure.Source: §3.3 · p. 7
  • On 922 rubric items jointly assessed by an expert and the adopted judge, agreement was 87% (Cohen’s κ = 0.67, 95% CI 0.61–0.73).Expert-derived process scores correlated with the judge’s at r = 0.54 and with outcome scores at r = −0.03.Source: §3.3 · p. 7

4 Experimental setup

The evaluation varies agents, models and instruction detail across repeated task runs, then summarizes outcome, process, cost and run-level measurements with documented exclusions and scoring conventions.

4.1 Configurations

  • The study first held the model constant across six agents, then varied models within DeepSeek Harness and Claude Code, and retested selected configurations with detailed instructions.Some configurations were omitted after incompatible API responses, and wound-healing runs exceeding a provider image-size limit were rerun until three scored runs existed.Source: §4.1 · pp. 7–8
  • Runs used fixed declared reasoning settings when available; repeated infrastructure failures were excluded, while agent-related crashes, timeouts and empty submissions were retained and scored as observed.Each task ran on a single GPU with a read-only input mount, and run metadata recorded software, hardware and model identifiers.Source: §4.1 · p. 8

4.2 Measurement and statistics

  • Efficiency measures include token counts, runtime, tool calls and monetary cost, with unavailable usage signals distinguished from zero and cost calculated from configuration-specific billing or token rates.Runtime was available for every agent; agent usage was counted separately from judge usage.Source: §4.2 · p. 8
  • Outcome summaries use three-run means, while variability is reported across runs and experimental factors; runtime, token and tool-call figures use medians over scored runs.The study reports process–outcome Pearson correlation and expert–judge agreement; provider safety-filter refusals are excluded from outcome means.Source: §4.2 · p. 8

5 Results

Routine two-dimensional tasks were tractable, but adding dimensions or time left severe failures that stronger models and detailed instructions did not resolve. Scores also varied substantially between repeated runs, while process scores and elapsed time did not reliably signal correctness.

5.1 Performance across tasks and agents

  • 0.85–0.96 on routine tasks, with every agent at least 0.56; on nuclear-pore kinetics and 3D puncta quantification, scores fell to 0.19 and 0.05, respectively.General-purpose agents achieved or tied the highest mean outcome on 14 of 16 tasks; biological specialization gave no overall advantage.Source: §5.1 · p. 9, Figure 2 · p. 10

5.2 Models, cost and instructions

  • 0.50–0.65 was the mean-outcome range across six harnesses on GPT-5.6 Sol and six models on DeepSeek Harness; DeepSeek-V4-Flash achieved 80% of the strongest configuration’s score for 2% of its cost.Opus 5 improved DeepSeek Harness by 0.07 but barely changed Claude Code; Codex cost five times as much as the Pareto-front configuration for a gain of 0.01.Source: §5.2 · p. 9, Figure 2 · p. 10
  • No model-and-instruction combination exceeded 0.25 on nuclear-pore assembly kinetics or 3D puncta quantification, despite stronger models and detailed instructions.Detailed instructions changed mean outcomes by +0.01 on V4-Flash and less than 0.01 on GPT-5.6 Sol; task-level changes averaged 0.21 and 0.11 in absolute value.Source: §5.2 · pp. 9, 13, Figure 2 · p. 10

5.3 Run-to-run variability and indicators of correctness

  • 40% of agent–task pairs shifted by more than 0.2 across three identical-task attempts, while best-of-three scores brought five of six agents within 0.04 of one another.Worst attempts differed by 0.18, indicating that agent differences primarily reflected reliability rather than best-case capability.Source: Figure 3 · p. 11, §5.3 · p. 13
  • Process and outcome scores correlated weakly (r = 0.29, falling to 0.09 above near-zero outcomes), and the longest run was best in only 20 of 70 disagreeing pairs.Expert process scores correlated with outcome at r = −0.03; runs scoring below 0.1 took twice as long as runs scoring at least 0.5 (21 versus 10 minutes).Source: §5.3 · p. 13, Figure 5 · p. 14, Figure 6 · p. 15

6 Discussion

BIABench finds that current agents handle routine bioimage analyses but remain unreliable as complexity grows, with some 3D and time-lapse tasks out of reach. Plausible-looking analyses can still be wrong, and neither specialization, stronger models, nor detailed instructions reliably close the gap.

6 Discussion

  • Some tasks adding a third dimension or time axis remained out of reach for every agent, while routine analyses could be completed.BIABench assumes experts could achieve nearly perfect scores in theory because tasks are scored at each study’s own endpoint.Source: §6 · p. 16
  • Across repeated runs, five of six agents came within 0.04 of one another at their best, yet failing runs rarely looked like failures.Process score and runtime did not distinguish correct runs from wrong ones; failures included contradictory tables, unchecked thresholds, and promises to finish.Source: §6 · p. 16
  • Biological specialization, stronger models, and detailed expert instructions did not reliably close the performance gap.The authors locate the remaining gap in data complexity, reliability, and self-checking.Source: §6 · p. 16
  • BIABench provides an open evaluation benchmark and a proposed community strategy for building large-scale, multistep image-to-insight datasets.The authors envision expert-level autonomous bioimage analysis as a long-term goal for the community.Source: §6 · p. 16

Competing interests

The authors disclose that they co-developed Agentic-J.

Competing interests

  • The authors co-developed Agentic-J.The disclosure identifies a competing interest related to the named agent.Source: Competing interests · p. 16
  • Agentic-J is the disclosed connection between the authors and the benchmark’s agent landscape.The passage states co-development, but does not specify its effects on the evaluation.Source: Competing interests · p. 16
  • The competing-interest statement reports co-development of Agentic-J without further qualification.No additional relationship is stated in the supplied passage.Source: Competing interests · p. 16

Appendix A Task specifications and provenance

The appendix specifies the suite’s task inputs and provenance: each task inherits a published study’s question and ground truth, with input sizes ranging from 4 MB to 5.1 GB.

Appendix A Task specifications and provenance

  • 4 MB to 5.1 GB is the input-data range across tasks, and the five largest inputs are all 3D or time-lapse data.Task descriptions name the source study, biological target, requested measurement, and repository materials for data access and reconstruction.Source: Appendix A · p. 23
  • Each task is rebuilt from a published study and inherits that study’s question and ground truth.The suite descriptions are grouped by hand-assigned difficulty level: easy, medium, and hard.Source: Appendix A · p. 23
  • The benchmark repository includes source accessions and scripts to reconstruct each task’s data locally.The per-task descriptions specify the biological target and the measurement requested from the agent.Source: Appendix A · p. 23

A.1 Easy tasks

The appendix describes tasks spanning easy, medium, and hard settings, with imaging modalities, dimensions, time structure, subtasks, required outputs, and scoring metrics specified across the suite.

A.1 Easy tasks

  • Easy tasks include 2D cell counting, compartment segmentation, histology nucleus segmentation, and dose-response quantification, with outputs scored against task-specific targets.For example, HeLa segmentation is scored by mean Dice across compartments, while NF-κB quantification uses dose monotonicity, replicate consistency, and untreated-versus-stimulated separation.Source: §A.1 · p. 23
  • The suite matrix distinguishes standard two-dimensional pipelines from specialized medium tasks and harder multistep or demanding analyses.Tasks are mapped to eleven subtasks, and each task’s output files and scoring metrics are specified.Source: Table A1, Table A2

A.2 Medium tasks

  • Medium tasks include DNA-PAINT localization, wound-healing speed kymographs, and synthetic filament segmentation.DNA-PAINT pools 300 frames into one localization list; wound-healing analysis produces a time-by-distance speed kymograph; microtubule segmentation uses synthetic images with filament signal decaying toward the ends.Source: §A.2 · p. 26

A.3 Hard tasks

  • Hard tasks include long phase-contrast bacterial tracking, where segmentation and lineage links must handle snapping division and increasing division frequency.The task uses an 800-frame sequence and scores 0.5 SEG + 0.5 TRA.Source: §A.3 · p. 27
  • Other hard tasks combine multichannel 3D tile stitching and cell-feature extraction, or track DNA-repair foci through time to compare normalized colocalization onset distributions.The stitching task preserves z while registering a 64-tile mosaic; the foci task masks a burnt-in timestamp before detection.Source: §A.3 · pp. 27–28

Appendix B Example task instructions

The appendix reproduces both instruction levels for the HeLa nucleus–cytoplasm task and shows how prompts combine task guidance with a shared output and policy footer. The detailed version recommends channel-specific segmentation and source-study annotation conventions while leaving implementation choices to the agent.

Appendix B Example task instructions

  • The shared prompt footer specifies deliverables, source-study policy, compute environment and run paths, while task-specific bodies provide brief or detailed guidance.Across 16 tasks, brief bodies are 127–398 words and detailed bodies 233–903 words; the footer adds 272–530 words.Source: Appendix B · p. 28
  • The brief HeLa instruction asks for separate nucleus and cytoplasm outlines, visualizations, and analysis of irregular, touching or overlapping cells.It identifies DAPI and phalloidin staining and gives the image pixel size as 0.645 µm.Source: Appendix B · p. 29
  • The detailed instruction assigns blue DAPI to nuclei and red phalloidin to cytoplasm, then recommends illumination correction, thresholding and watershed separation of touching nuclei.It presents alternative tools and segmentation approaches rather than prescribing one implementation.Source: Appendix B · p. 29
  • The detailed prompt requires matched nucleus–cytoplasm instance IDs, retained border cells, specified mask files and representative overlays and plots for quality control.Its source-study policy bars retrieving the source publication or its reported values, while permitting methods literature and software documentation.Source: Appendix B · p. 29

Appendix C Agent integration and run limits

The appendix describes integrations for six agents spanning tool-use, command-line coding and Fiji/ImageJ GUI approaches. Run limits varied by agent, with a four-hour external budget and two-hour sessions for most listed model exchanges.

Appendix C Agent integration and run limits

  • Six agents use distinct integrations: Biomni executes Python, coding CLIs expose JSON event streams, and GUI agents operate Fiji/ImageJ through desktop bridges.The coding interfaces parse native token usage; Biomni captures provider usage at model invocation.Source: Appendix C · p. 30
  • Biomni alone has a per-step execution timeout; coding CLIs and GUI agents have no uniform turn cap, while turn and tool-call counts are logged.The timeout is intended to let heavy steps finish without allowing a hung step to consume the whole wall-clock budget.Source: Appendix C · pp. 30–31
  • The external wall-clock budget was 4 h, while most Kimi K2.6, GLM-5.1 and Claude Opus 5 sessions ran for 2 h before timeout.Those timed-out runs stop at 120 min.Source: Appendix C · p. 31

Appendix D Baseline and reference submissions

The appendix checks that reference submissions can reach the top outcome score and that empty or report-only submissions score zero. It also describes a special outcome metric for the NF-κB task, which lacks a ground-truth file.

Appendix D Baseline and reference submissions

  • ≈1.0: feeding each task’s ground truth back through evaluation reaches approximately 1.0 wherever a ground-truth file exists, confirming the metric can attain a perfect score.No metric is capped below 1, and the declared output format admits a perfect submission.Source: Appendix D · p. 31
  • The NF-κB task cannot use ground-truth replay, so its outcome score evaluates dose response, replicate agreement and difference from untreated control.Each property is normalized to [0, 1], allowing the metric to reach 1.Source: Appendix D · p. 31
  • The empty-output baseline scores zero on every task, and a report-only submission with no output files receives no process score and zero outcome score.These synthetic submissions bound the score earned without an analysis.Source: Appendix D · p. 31

Appendix E Process rubric and judge validation

The appendix describes a severity-weighted process rubric for rendered submission outputs and validates the vision–language judge against expert review, including agreement, over-crediting, commonly failed rubric items, and agent-usage signals. It reports substantial judge–expert agreement but weak alignment between process and outcome scores.

The rubric

  • The rubric contains 105 yes/no items weighted by severity, with each run scored on 49–75 applicable items using the weighted fraction passed among decided items.It combines general items with task-specific subsections; the process score is applied to rendered outputs across pipeline stages from loading through reporting.Source: Appendix E · p. 31, Table E1 · p. 34

Expert review and judge agreement

  • The adopted judge agreed with the expert on 87% of 922 jointly decided items (Cohen’s κ = 0.67, 95% CI 0.61–0.73), but passed 27% of expert-failed items versus failing 7% of expert-passed items.The expert reviewed 20 runs spanning all six agents and 16 tasks. The judge abstained on 20% of items, compared with 15% for the expert.Source: Appendix E · p. 33

Agreement by rubric stratum

  • Agreement varied by rubric stratum, from κ = 0.87 for input understanding to κ = 0.47 for tool choice and use, where the judge passed half of expert-failed items.Agreement on critical items was 97%, but κ was 0.52; only 8 of 221 critical items were failed by the expert, and the judge caught four.Source: Appendix E · p. 33

Run-level process scores

  • The expert’s process score showed no predictive relationship with outcome score (r = −0.03, 95% CI −0.48 to 0.31), and its correlation with the judge’s score was r = 0.54.The expert’s process scores for the five runs with outcome below 0.1 ranged from 0.74 to 0.82, and the weak process–outcome relationship persisted under expert scoring.Source: Appendix E · p. 34

Rubric discrimination

  • Expert review found that critical items were rarely failed, while minor items accounted for more weighted failures, and re-weighting rubric items did not recover outcome information.Runs failed 4% of critical items, 31% of major items and 44% of minor items; scoring critical items alone correlated with outcome at r = 0.08.Source: Appendix E · p. 34

Judge validation

  • Expert validation reviewed 20 runs from six harnesses across all 16 tasks, with the expert deciding 1,085 of 1,273 rubric items.The expert was blind to judge decisions; agreement used items decided by both, with 95% cluster-bootstrap intervals over runs.Source: Table E2 · p. 38

Failed rubric items

  • Table E3 covers the 12 rubric items with the highest expert-reviewed fail rates among items decided in at least eight of 20 reviewed runs.It defines judge agreement against the expert verdict and classifies severity as critical, major, or minor, weighted 3, 2, or 1.Source: Table E3 · p. 38

Agent usage signals

  • Table F1 reports usage signals for six agents on GPT-5.6 Sol under brief instructions, with wall-clock runtime available for every run.Other signals are included only when exposed by a harness; medians use runs exposing each signal, and token medians include cached input and output.Source: Table F1 · p. 39

Appendix F Token, runtime and cost accounting

The appendix documents how token usage and costs were measured and qualified, including the sources used for billing and the reasoning-effort tier applied across configurations.

Appendix F Token, runtime and cost accounting

  • Token counts are reported so harness–model configurations can be repriced when vendors, tiers, routing layers, or prices change.Provider billing supplies Claude Code and DeepSeek-V4-Pro costs because multi-session harness runs make metered client-side counts incomplete.Source: Appendix F · p. 39
  • Only Opus 5 on DeepSeek Harness is underpriced for cache writes, by at most $0.25 per run on average and $0.94 for the heaviest run.Fresh and cache-write tokens are 5% of input; the provider's 1.25× cache-write rate sets the surcharge bound.Source: Appendix F · p. 39
  • Table F1 reports signal exposure, token-count sources and medians per harness; input-token totals include cached reads for comparability across agents.Providers bill cached reads differently.Source: Appendix F · p. 39
  • All harness–model configurations ran at medium reasoning effort, with harness-specific controls or logs recording the tier requested or applied.CopilotJ has no native setting; the driver injects the field and records whether the route accepted it.Source: Appendix F · p. 39
  • Table F2 lists OpenRouter model slugs and provider list prices per million tokens; judge-model usage is accounted and priced separately from agent usage.The two DeepSeek models used 20260423 preview snapshots; DeepSeek-V4-Pro and Claude Code costs in the paper come from provider billing.Source: Table F2 · p. 40

Appendix G Failure case studies

Three archived runs show that procedural plausibility did not guarantee a correct result or a completed deliverable: failures involved erroneous files, miscalibrated detection and a missing output.

Appendix G Failure case studies

  • 0.00 outcome despite 0.71 process: CopilotJ reported 1,609 nuclei but delivered 266 against 2,062 in the reference, reversing the condition-level puncta ordering.Its validation counted a different intermediate than the final file; 27 QC overlays were written but never inspected.Source: Appendix G · p. 40
  • 0.02 outcome despite 0.87 process: Codex preserved condition ordering but undercounted wild-type puncta by a factor of thirty.Per-image counts correlated with the reference at r = 0.85, but the detector discarded dense clusters and its thresholds were never checked against images.Source: Appendix G · p. 40
  • 0 outcome with a 0.80 process score: DeepSeek Harness wrote a 211-line NF-κB pipeline but never produced the required per-well summary file.The harness exited normally, and no process continued after the agent's final turn.Source: Appendix G · p. 41

Related papers

Loading 2609.34274v1…