PaperlayerView exact revision on arXiv

Source-linked AI summary

BenchBench-Protocol: Evaluating Real-World Wet-Lab Protocol Reasoning and Modification

Aditya Sivakumar, Ashu Singhal, Nicholas Larus-Stone, Nithin Parsan

arXiv:2608.23898v1cs.AI

TL;DR

Wet-lab scientists routinely adapt protocols, but correct modifications require tracking prior choices and downstream effects. BenchBench-Protocol reconstructs such tasks from real protocol changes and evaluates nine models, with Claude Opus 5 scoring highest while repeated sampling leaves performance unsaturated.

  • Problem

    Correctly adapting wet-lab protocols requires reasoning about prior choices and downstream steps, while existing tasks are typically elicited rather than reconstructed from real modifications.

  • Method

    BenchBench-Protocol contains 149 open-ended, weighted-rubric tasks recovered from scientist modifications to published protocols, reviewed by domain experts and evaluated across nine models.

  • Results

    Claude Opus 5 scores highest at 59.2% normalized rubric score, other models range from 34.1% to 47.1%, and best-of-ten sampling remains unsaturated.

  • Takeaways & Limitations

    Grounding evaluation in modifications scientists actually made offers a direct way to measure progress on realistic wet-lab protocol reasoning.

  • Takeaways & Limitations

    The benchmark does not directly measure whether models improve experimental outcomes and may not reflect routine adjustments because its tasks were selected to be difficult.

Abstract

from arXiv · show

We introduce BenchBench-Protocol, a benchmark for large language models of 149 protocol-modification tasks recovered from modifications that scientists made to published protocols during real experimental work. Adapting a published protocol to a new experiment is a routine task for a wet-lab scientist, and a correct modification requires accounting for prior choices and downstream steps. Recent life-science benchmarks have moved toward open-ended, rubric-graded tasks, but tasks are typically elicited from experts rather than reconstructed from real-world modifications. BenchBench-Protocol tasks are derived from differences between a published protocol and a version a scientist modified, which provides the basis for the query and the weighted rubric elements for a correct response. The benchmark draws from 96 source protocols across nine domains of wet-lab biology and only includes tasks rated highly after review by domain experts. We evaluate nine closed and open models; Claude Opus 5 scores highest at 59.2% normalized rubric score, with other models between 34.1% and 47.1%, and the benchmark remains unsaturated when taking the best of ten attempts. As models are increasingly helpful in life-sciences research, evaluating them on routine wet-lab tasks becomes correspondingly important. We present BenchBench-Protocol as both a grounded assessment of wet-lab reasoning and evidence for the utility of real-world experiments to construct benchmark tasks.

1 Introduction

BenchBench-Protocol addresses the gap between expert-authored benchmark tasks and the constrained protocol modifications scientists make during real wet-lab work. It derives realistic tasks and weighted rubrics from scientist adaptations of published protocols, retaining them after expert review.

  • Motivation: Protocol adaptation is routine wet-lab work requiring attention to prior choices and downstream experimental steps.Correct modification involves more than factual recall or retrieval.
  • Motivation: Expert-elicited benchmark tasks may differ from real-world modifications constrained by reagents, instruments, and prior results.
  • Benchmark contribution: BenchBench-Protocol recovers open-ended protocol-modification tasks from scientists’ changes to published protocols rather than creating them from scratch.Each task and rubric was independently reviewed by multiple experienced scientists.
  • Benchmark contribution: 149 expert-reviewed tasks span nine wet-lab biology domains and are phrased as scientists would pose questions to language models.
  • Benchmark contribution: Weighted rubrics score responses against the specific downstream dependencies required by the scientist’s modification.

2 Related Work

Existing life-science benchmarks increasingly evaluate open-ended research assistance, but many remain computational or rely on expert-authored protocol tasks. BenchBench-Protocol complements this landscape by grounding protocol-modification tasks in scientists’ real experimental changes.

  • Existing benchmarks: Recent benchmarks broaden evaluation beyond biological fact recall toward open-ended and rubric-graded research tasks.
  • Existing benchmarks: Agentic benchmarks assess code execution, bioinformatics reasoning, and broader multi-stage scientific workflows.
  • Existing benchmarks: Many benchmark tasks remain limited to computational biology.
  • Closest related benchmark: LifeSciBench is the closest analogous benchmark, using expert-authored rubric-graded free-response tasks that include protocol design and optimization.
  • Positioning: BenchBench-Protocol extends these evaluations by deriving tasks from protocol modifications made during real experimentation.

3 Benchmark Design and Construction

The benchmark is constructed from scientist-modified versions of published protocols, with differences defining task ground truth and downstream rubric requirements. Domain experts then filter and assess the resulting task corpus for scientific quality and relevance.

  • Task format: Each task presents a naturally phrased scientist query, a published protocol, and a requested free-text modification with rationale.
  • Corpus coverage: 149 evaluation tasks come from 96 published protocols across nine wet-lab biology domains.
  • Benchmark construction: Comparing published and scientist-modified protocols provides ground truth for the modification and its downstream consequences.Questions and rubrics are drafted from these differences, then ambiguous or unanswerable items are removed.
  • Task characterization: Primary intent labels enable performance analysis by the kind of judgment demanded, in addition to biological domain.
  • Expert review: Only tasks rated Very Strong or Excellent for relevance, scientific soundness, rubric accuracy, and overall quality were retained.Each task was reviewed by at least two relevant PhD-level scientists with three or more years of experience.

4 Experimental Setup

Nine models were evaluated on repeated free-response attempts using weighted normalized rubric scores, with confidence intervals accounting for shared source protocols. The setup also reports expected best-of-k performance from repeated attempts.

  • Models and sampling: Nine models were evaluated across all tasks, with results aggregated over ten attempts per task.Invalid responses caused by formatting problems or refusals received zero scores.
  • Model conditions: Models received the query, published protocol, and configured internet access, while LLM graders had no web tools.
  • Metrics: Normalized rubric scores divide weighted criterion points earned by total available weighted points, with criteria scored 0, 1, or 2.Bootstrap confidence intervals resample at the source-protocol level because derived tasks are not independent.
  • Metrics: Expected best-of-k samples k of ten attempts per question, takes the maximum score, and averages across resamples.
  • Reporting: Figure 3 displays mean normalized rubric scores across 149 tasks with 95% confidence intervals.

5 Results

BenchBench-Protocol reveals substantial variation in model performance across tasks and domains, with Claude Opus 5 leading overall. Repeated sampling can change rankings, while the benchmark remains unsaturated and exposes differences in reasoning demands and model behavior.

  • 59.2% normalized score: Claude Opus 5 is the strongest overall model, while other evaluated models range from 34.1% to 47.1%.The benchmark retains at least 40% headroom of unearned weighted rubric credit across model families.
  • Performance Under Repeated Sampling: Model rankings are not stable across k: GPT-5.6 Sol and Kimi K3 are statistically indistinguishable at k = 1, but Kimi K3 performs better at k = 10.Expected best-of-k performance is reported as an upper bound on single-model performance because scientists can compare candidate answers.
  • Performance Under Repeated Sampling: At ten attempts, Claude Opus 5 captures 39.5% of available headroom and Kimi K3 captures 31.5%, while GPT-5.6 models capture 19.6%–22.5%.Under oracle selection, the strongest model still leaves roughly one quarter of rubric credit unearned.
  • Performance by Task Intent: Models perform comparatively well when reasoning from experimental evidence to supported conclusions, but worse when tasks require tacit knowledge or expert intuition.This pattern may account for the discrepancy between troubleshooting and adaptation performance.
  • Performance by Source-Protocol Domain: Claude Opus and GPT-series models outperform open-source models on staining and imaging tasks, possibly reflecting stronger multimodal reasoning.Figure 6 summarizes these results, with complete values in Table E1.
  • Resource Use: Grok 4.5 consumes the most input tokens, likely because frequent web searches ingest large volumes of content.At maximum reasoning settings, models perform frequent searches, increasing input and output token consumption.

6 Limitations and Future Work

The benchmark’s scope and evaluation design limit what its scores can establish about wet-lab assistance, while future work should better reflect interactive laboratory use.

  • The benchmark measures reasoning about realistic protocol modifications, not whether models improve experimental outcomes at the bench.
  • Its 149 tasks span nine wet-lab biology domains, but cannot cover the full range of modifications arising in laboratory work.
  • Because tasks were selected for difficulty, scores may not represent performance on routine adjustments common in day-to-day laboratory work.
  • The single-turn evaluation omits scientists’ clarifications about reagent, instrument, and time constraints, and performance may improve with clarifying questions.
  • Using maximum reasoning settings increases token usage and cost, affecting the performance–cost frontier; results may differ in other harnesses or multi-agent setups.
  • Human expert baselines were not directly assessed, and model grading remains imperfect despite a coarse rubric scale.
  • Future work can examine multi-turn collaboration, follow-up planning from prior experiments, troubleshooting failures, and underspecified user intent.

7 Conclusion

BenchBench-Protocol grounds wet-lab protocol-modification evaluation in changes scientists actually made and scores answers against downstream dependencies. Across its tasks, current models remain imperfect, with performance differences largely reflecting overall level rather than distinct domain profiles.

  • Conclusion: BenchBench-Protocol contains 149 expert-reviewed protocol-modification tasks spanning nine wet-lab biology domains and recovered from real experimental work.
  • Conclusion: Its weighted rubrics are grounded in actual modifications and score whether responses address the specific downstream dependencies required for correctness.
  • Conclusion: 59.2% is Claude Opus 5’s highest score, while other models range from 34.1% to 47.1%; repeated sampling captures between one fifth and two fifths of unearned credit.
  • Conclusion: At ten attempts, Kimi K3 approaches and then exceeds GPT-5.6 Sol in expected score@k because its heavier upper tail yields a higher expected best-of-k score.
  • Conclusion: Models differ mainly in overall level rather than profile, so current systems are not differentially suited to particular areas of wet-lab reasoning.

A Protocol Domain and Task Intent Definitions

The benchmark organizes protocols into nine wet-lab domains and classifies tasks by their primary reasoning intent. Domains distinguish biological procedures, while intent labels capture the judgment demanded by each task.

  • Protocol domains: Each source protocol receives exactly one domain, assigned according to the step containing the task’s modification.
  • Protocol domains: Protein biochemistry and purification includes protein expression, extraction, purification, buffer exchange, concentration, and biochemical characterization.
  • Protocol domains: Cell culture and cell-based assays include cultured-cell maintenance, manipulation, treatment, and population-level readouts, excluding primary tissue and animal work.
  • Protocol domains: The nine domains cover sequencing, microbiology, reagent preparation, in vivo and primary tissue work, chemical synthesis, protein purification, cell culture, cloning, and imaging or cytometry.
  • Task intents: Task intent is assigned primarily according to the rubric criteria carrying the largest share of total weight.
  • Task intents: The intent taxonomy includes plan audit, result interpretation, troubleshooting, recordkeeping, procedure adaptation, quantitative formulation, and follow-up experiments.
  • Evaluation setup: Evaluation used provider-hosted or configured search access, highest available reasoning settings, repeated attempts, and standardized LLM grading with reported agreement checks.

E Performance by Source-Protocol Domain and Expected best-of-k Metrics

This section reports performance across source-protocol domains and the expected benefit of allowing multiple attempts. The accompanying tables define domain-level scores and best-of-k gains.

  • Performance by source-protocol domain: Performance by source-protocol domain is reported as mean normalized rubric score with 95% confidence intervals.
  • Expected best-of-k metrics: Expected best-of-k metrics report scores at k = 1, 2, 5, and 10, together with the gain from k = 1 to k = 10.

F Additional Sample Tasks

The paper provides sample tasks from microbiology, molecular biology, and protein biochemistry, alongside cost accounting details for model inference.

  • Sample tasks: Examples F1 and F2 show microbiology tasks with the scientist query, source protocol, and weighted rubric criteria.
  • Sample tasks: Example F3 presents a molecular-biology-and-cloning task with its scientist query, source protocol, and weighted rubric criteria.
  • Sample tasks: Example F4 presents a protein-biochemistry-and-purification task with its scientist query, source protocol, and weighted rubric criteria.
  • Cost accounting: Average inference cost uses input and output token counts with API pricing collected on August 13, 2026, excluding web search and hosted-tool usage.

H Performance–Cost Frontier

The performance–cost analysis identifies models on the Pareto frontier while reporting invalid-response rates that affect evaluated scores. Claude Opus 5 has the highest reported score among the frontier comparison.

  • Performance–cost frontier: Claude Opus 5 achieves the highest score at 59.2% while costing less than GPT-5.6 Sol; Kimi K3 slightly outperforms GPT-5.6 Terra at lower cost.GPT-5.6 Luna, Kimi K3, and Claude Opus 5 lie on the performance–cost Pareto frontier.
  • Invalid responses: GLM 5.2 produced misformatted responses on 181 of 1,490 attempts (12.1%), while Gemini 3.6 Flash refused 106 requests (7.1%).Unsuccessful attempts from refusals or misformatting were assigned a score of zero.
Loading 2608.23898v1…