PaperlayerView exact revision on arXiv

Source-linked AI summary

LabAgent: Customize Any Research Hubs for Scientific Discoveries Using AI Agents

Lei Liu, Yikun Zhang, Jialin Chen, Wanjia Zhao, Rex Ying, Wengong Jin, Hua Xu, James Zou, Tianyu Liu, Hongyu Zhao

arXiv:2609.13437v1cs.AIq-bio.BM

TL;DR

Published computational methods are difficult to reuse when laboratory know-how remains implicit and can erode as researchers leave. LabAgent turns laboratory artifacts into executable skills, verifies reproductions against published metrics, and records fixes for later runs. The paper concludes that it is useful for reusing released laboratory code and asking biological questions beyond published protocols.

  • Problem

    Important reproduction knowledge, such as preprocessing steps, package versions, and hand-chosen parameters, is often implicit and can erode as laboratory members leave.

    Source: §1 · p. 1

  • Method

    LabAgent builds laboratory-specific executable skills from public artifacts, verifies runs against reported metrics, and stores reproduction logs, diagnoses, and fixes in persistent memory.

    Source: §1 · p. 2

  • Results

    LabAgent landed within the published standard deviation on 32 of 65 entries and achieved a best mean rank of 2.62 among the three agents compared.

    Source: §2 · p. 5

  • Takeaways & Limitations

    LabAgent is useful for reusing published computational methods from a laboratory’s released code and asking biological questions beyond published protocols.

    Source: §3 · p. 16

    For settings that name the method to reproduce, a larger evaluation set is needed before the gain from this route can be distinguished.

    Source: §3 · p. 15

Abstract

from arXiv · show

Scientific research is a continuous process that emphasizes inheritance. Methods developed by predecessors are often expanded upon by new researchers to explore more novel and in-depth scientific questions. However, the change of lab staff, such as student graduation, leads to a lack of personnel capable of replicating methods. Methods that have been developed with significant effort and resources cannot be continued. To address these limitations, we propose LabAgent, a reproduce and discovery harness tailored for a lab's continuous work. LabAgent employs two mechanisms to guarantee that all skills can be executed and verified and to record the corrective methods and experiences, allowing for direct correction or avoidance of similar errors. We applied LabAgent to drug property prediction, biomedical problem analysis, protein variant effect prediction, and statistical genetics in life science domains. LabAgent ranks first over commercial generalist agents in every domain, and demonstrates accurate reproduction of a published figure. Overall, these results demonstrate that LabAgent can effectively integrate and reasonably expand laboratory knowledge.

1. Introduction

Published computational methods are difficult to reuse because important laboratory know-how remains implicit and can erode as researchers leave. LabAgent turns laboratory artifacts into reusable skills, verifies reproductions against reported results, and carries fixes forward to support further discovery.

1. Introduction

  • Unwritten preprocessing, unpinned package versions, and hand-chosen parameters make published computational work hard to reproduce, while laboratory know-how can erode as members leave.The knowledge may persist only in repository details, issue discussions, and researchers’ memories.Source: §1 · p. 1
  • Existing scientific agents address fragments of the need, but do not package a laboratory’s validated methods for reuse across reproduction and discovery.Discovery agents draw on general literature, while engineering agents handle tasks in isolation and do not retain reusable laboratory knowledge.Source: §1 · p. 2
  • Prior automatically built skill libraries draw on field-wide resources and validate executability or utility, but do not verify reproduction against a study’s reported results or retain fixes across reproductions.They also stop short of using a laboratory’s validated methods to propose and test new hypotheses.Source: §1 · p. 2
  • LabAgent’s explore agent grounds methods in laboratory papers, code, tutorials, and issue discussions, then synthesizes them into executable skills that capture otherwise unstated run details.Its laboratory-specific skills include preprocessing, pinned package versions, and reported failure modes.Source: §1 · pp. 2–3
  • The reproduce agent retrieves earlier protocols and fixes, checks outputs against the original study’s metrics, repairs the failing skill when needed, and writes logs and fixes to persistent memory.Validated skills are then used to propose and test new scientific hypotheses.Source: §1 · p. 3

2. Results

LabAgent applies its explore-and-reproduce framework across drug, biomedical, protein, and statistical-genetics tasks. It outperformed comparison agents on several reported benchmarks, while the fine-mapping reconstruction also exposed consequential departures from the published setup.

Reproducing drug property leaderboards from laboratory code

  • 32 of 65 entries landed within the published standard deviation; LabAgent had the best mean rank (2.62) and ranked first on 13 entries, at US$4.56 per entry.It had more entries within the error bar than either commercial harness, but paid the most per entry.Source: §2 · p. 5, Figure 2 · p. 6

Analysing biomedical data beyond any published protocol

  • 74.4 mean rubric score led Claude Code by 2.1 points and Codex by 8.7; removing the explore agent scored 71.6, while the full system scored 2.8 higher (SE 1.7).LabAgent led on 20 of 50 tasks, trailed on 17 and tied on 13; its gains included mutation analysis and pathway enrichment, while Claude Code led on clustering and cell composition. The Codex gap combines harness and base-model differences.Source: §2 · pp. 5, 7, Figure 3 · p. 8
  • −0.561 was LabAgent’s CXCL10 effect estimate against a published −0.600, within the grader’s 0.05 tolerance; each system fell outside tolerance when the donor group was withheld.The 24 donors formed two contamination groups associated with genotype dosage; restoring the group term brought each system back inside tolerance.Source: §2 · p. 7, Figure 4 · p. 9

Predicting protein variant effects without a named method

  • 0.443 mean Spearman correlation led the three comparison arms across 13 assays; LabAgent won 7 assays, never fell below 0.253, and cost US$7.93 per assay.On the spike assay, LabAgent reached 0.560 while Claude Code reached 0.021; the reported gap was 0.539, attributed almost entirely to method choice rather than execution.Source: §2 · pp. 10–11, Figure 5 · p. 12

Rebuilding a published figure in statistical genetics

  • SuSiE2 ranked above every other method on power in all ten grid cells, raising detection over single-trait SuSiE by 11.1% to 45.9%.At the lowest heritability, SuSiE2 raised power from 0.333 to 0.485; at the highest, from 0.680 to 0.770. The reconstruction used 50 rather than the article’s 100 replicates per cell.Source: §2 · p. 13, Figure 6 · p. 14
  • The fine-mapping reconstruction was not exact: fastPAINTOR’s two-variant enumeration cap bounded power, and two other method lines were tuned surrogates rather than reproductions.The rubric awarded 88.1 of 100 but did not identify the surrogate lines or these departures; credible-set sizes were also smaller than reported.Source: §2 · p. 13, Figure 6 · p. 14

3. Discussion

LabAgent combines validated laboratory skills with memory that carries fixes across reproductions, and it outscored commercial harnesses in every compared setting. The discussion also describes biological insights from reproductions while identifying limits in evaluation and biological interpretation.

3. Discussion

  • LabAgent outscored three commercial harnesses in every setting with a comparison across three benchmarks and two case studies.The authors conclude it can help reproduce published methods and explore biological questions.Source: §3 · p. 15
  • Named-method settings require larger evaluation sets to distinguish skill-route gains; the systems perform similarly on ADMET, while method choice matters more than execution error on ProteinGym.Withholding the method opens a performance gap, while LabAgent scored highest on BiomniBench-DA and GeneBench-Pro.Source: §3 · p. 15
  • Reproduction memory records failures, diagnoses, and repair commands, but current accumulated experience describes environments and tooling rather than analytical judgments.The authors identify a reusable form for analysis judgments as a future need.Source: §3 · p. 15
  • Reproduction can reveal assay-constrained protein regions and recover a donor-level variable that determines whether a published single-cell estimate is reached.The authors propose using reproductions to examine assay biology and expose variables an analysis may mistake.Source: §3 · p. 15
  • Biological interpretation remains limited because technical variation may carry biological signal, figure-generation paths are unrecoverable, and assay-specific method fit lacks an established measure.Current protein-assay evaluation also reports one whole-assay correlation coefficient without distinguishing residues.Source: §3 · p. 16

4. Methods

LabAgent turns laboratory artifacts into executable skills and uses those skills, a persistent memory, and a bounded reproduction target to reproduce reported metrics. Its explore agent grounds skills in repositories, checks quantitative instructions, validates execution, and organizes procedures for stepwise retrieval.

4. Methods

  • A reproduction agent executes a validated skill against a target and persistent memory, returning its metric value and updated memory.The target includes the paper, repository, dataset, reported metric, and run budget.Source: §4 · p. 16
  • Runs are scored after termination against a reference value and allowed deviation, with benchmark-specific grading and per-split values and dispersion when repeated splits apply.The scoring procedure reads only the agent's output files and returns nothing to the agent.Source: §4 · pp. 16–17

4.1. The Explore Agent

  • For open analysis tasks, the explore agent searches literature and code using expert-perspective rewritings; for named-method tasks, it starts from the specified laboratory repository.The search is bounded by round caps, reflections, and stop criteria.Source: §4.1 · p. 17
  • Skills are drafted from a repository-derived API snapshot, which constrains identifiers and dependencies to those the code actually exposes.The snapshot records importable modules, public functions and classes, examples, declared dependencies, and repository metadata.Source: §4.1 · p. 17
  • A reflection pass requires checks at quantitative steps and bars unsupported prescriptions of specific methods, thresholds, packages, or parameters.Checks compute and print quantities before choosing a course; instructions simply name the course.Source: §4.1 · pp. 17–18
  • Skills enter the library only after clearing three tiers of checks, culminating in installing dependencies and executing the quick-start in a temporary environment.The tiers check the skill file, resolve dependencies and calls, then run the quick-start end to end.Source: §4.1 · p. 18
  • Each skill is split into a roughly 50-line root document and four to seven step-specific loci that the agent retrieves by name.A run is refused if staged loci are empty.Source: §4.1 · p. 18

4.2. The Reproduce Agent

The reproduce agent selects a method, adapts its execution to the available machine, and works through failures before an external benchmark checks the output. Its memory stores actionable fixes from prior runs and returns them to later runs when relevant failures arise.

4.2. The Reproduce Agent

  • The agent selects exactly one skill in a deterministic, temperature-zero call, and an unmatched skill name raises an error rather than silently falling back.This makes selection repeatable and distinguishes selection failure from poor execution.Source: §4.2 · p. 19
  • Execution proceeds through machine probing and environment setup, data checks, code writing, running, verification, and analyst-led recovery, using one shell command per turn.Probe results determine installation commands, while an isolated virtual environment prevents package changes in one run from reaching another.Source: §4.2 · p. 19
  • At least one command failed in 72% and 71% of runs requiring live environment builds, versus 27% and 20% on tasks with staged data.Across 140 runs, recovery followed 116 of 145 failing turns; context-window limits ended runs early in 9% of cases.Source: §4.2 · pp. 19–20
  • The benchmark comparison occurs only after the agent stops, and its scoring procedure reads the output files without returning feedback to the running agent.The run itself is bounded by turn, cost, individual-command time, and total wall-clock limits.Source: §4.2 · p. 20

4.3. The Reproduction Memory

  • The memory stores skills, their grounding evidence, run records, and diagnoses, but only diagnoses are automatically read during execution.New candidate lessons pair recognized failure signatures with a successful next command and are added to persistent memory after the run.Source: §4.3 · pp. 20–21
  • A run first retrieves applicable lessons, then matches command output to failure triggers so relevant fixes can be shown at the point of failure.Lessons preserve the exact successful command and track later successes and failures; operational lessons transfer across domains, while recipe lessons are task-specific.Source: §4.3 · p. 20

4.4. Benchmarks and datasets

The benchmarks span five computational biology abilities, from reproducing published methods to rebuilding an experiment behind a figure. Their evaluations use task-specific scoring procedures and report cross-task summaries as counts rather than averaging unlike metrics.

4.4. Benchmarks and datasets

  • Five tasks cover published-method execution, open-ended analysis, artifact detection, method selection, and figure reproduction across drug discovery, genomics, single-cell analysis, protein science, and statistical genetics.The suite uses TDC ADMET, BiomniBench-DA, GeneBench-Pro, ProteinGym, and a fine-mapping study from the authors’ laboratory.Source: §4.4 · p. 21
  • ADMET reproduction uses 65 reachable leaderboard targets from 22 datasets, represented by 11 repository-based skills and evaluated against each entry’s published standard deviation and a 5% ratio-scale band.Each target uses five seed splits; one target is excluded because its repository was deleted upstream.Source: §4.4 · p. 21, §4.5 · p. 23
  • BiomniBench-DA contributes 50 public analysis tasks across 16 task types, scored by criterion-level overlap between the trace and the fully correct rubric wording.The task requires both a written trace and an answer file; this deterministic scoring route replaces the benchmark’s language-model rubric judgment.Source: §4.4 · pp. 21–22, §4.5 · p. 23
  • GeneBench-Pro evaluation uses 10 public multi-stage problems, with a binary verdict against a tolerance-calibrated target and a partial score over requested quantities.The remaining 119 of its 129 problems are held out by the benchmark authors.Source: §4.4 · p. 22, §4.5 · pp. 23–24
  • ProteinGym tests method selection on 10 stratified assays using a 26-skill library, with measured effects withheld and predictions scored by Spearman correlation.Sampling rules were fixed before runs and did not consult method performance; task instructions state the required direction identically for each system.Source: §4.4 · p. 22, §4.5 · p. 24

4.5. Evaluations

  • Each task is scored in its own metric and units, and cross-task summaries use counts rather than averages because task metrics have incompatible scales and directions.Scoring runs after termination and reads only files written by the agent.Source: §4.5 · p. 23
  • The fine-mapping target is a published simulation over 10,000 Europeans, with five methods compared across two causal architectures and five heritability settings.The rubric scores asserted relationships, reported quantities, and experiment completeness; its largest weight, 30 of 100 points, is whether the introduced method leads on power.Source: §4.4 · pp. 22–23, §4.5 · p. 24

4.6. Baselines

The study compares LabAgent with three complete systems under shared task text, staged data, and budget ceilings, while noting that these comparisons do not isolate individual design choices. It also specifies run-level statistical reporting and identifies data and code availability.

4.6. Baselines

  • The three baselines are Claude Code CLI, Codex CLI, and Claude Science, but comparisons among complete systems cannot attribute outcomes to a single design choice.LabAgent and Claude Code share a base model, while Codex uses GPT-5.5; the proposed attribution arm would compare LabAgent with and without its explore agent.Source: §4.6 · pp. 24–25

4.7. Statistics and Reproducibility

  • Each target is attempted once, and paired t-tests compare systems over jointly completed tasks with pair counts, mean differences, standard deviations, test statistics, and p-values reported.ADMET target values average five seed fits and retain their own dispersion.Source: §4.7 · p. 25
  • ProteinGym assay selection was performance-blind, and measured effects were withheld from staged data for every system.One ADMET target with a deleted upstream repository was excluded throughout.Source: §4.7 · p. 25

4.8. Data Availability

  • Generated skill libraries and reproduction trajectories are provided as Supplementary Data, while individual-level UK Biobank data require approved access and cannot be redistributed.The staged genotype panel was assembled from UK Biobank data for an earlier study from the authors’ laboratory.Source: §4.8 · p. 25

4.9. Code Availability

  • LabAgent code is available under the MIT license, and the experiments used Yale High Performance Center resources.The paper gives the code repository as https://github.com/fpxlei/LabAgent.Source: §4.9 · p. 25

A. Prompts and the acceptance rubric

This section presents prompts for building research code and discovering and refining research skills, alongside an acceptance rubric for making skill instructions concrete and verifiable. Together, these mechanisms emphasize fixed answer contracts, internal consistency, and safeguards checkable against data.

A. Prompts and the acceptance rubric

  • The two prompt pairs serve TDC ADMET and ProteinGym code builds, and BiomniBench-DA, GeneBench-Pro, and fine-mapping tasks using skill loci and deterministic checking.The caller can replace the answer contract; the fine-mapping run uses a metrics table because its scorer does not read a JSON answer.Source: A. Prompts and the acceptance rubric · p. 31
  • The first system prompt requires an initial machine probe to select install commands, followed by dependency setup, data validation, experiment execution, and result evaluation.The prompt ties GPU-dependent package installation to detected CUDA and GCC versions.Source: A. Prompts and the acceptance rubric · pp. 31–33
  • The second prompt grades quantitative analyses against exact numeric or exact-match keys and requires QC checks before writing and verifying the answer JSON.It instructs the agent to emit the completion sentinel only after the answer file exists and parses.Source: A. Prompts and the acceptance rubric · pp. 34–35
  • The second system prompt makes the agent retrieve skill loci on demand in workflow order instead of reading every locus upfront.The root skill document points to separate loci containing data schemas, method derivations, runnable code, and quality-control procedures.Source: A. Prompts and the acceptance rubric · pp. 34–35
  • The explore agent moves from a research brief and capability manifest through parallel search, reflection, and skill critique; scouts use assigned search perspectives and supervisors check stop criteria.Its reflection tool requires supervisors to name artefacts found, track met and open criteria, detect diminishing search returns, and choose a new query or stop. The final critique rewrites the skill as a short root plus separate loci.Source: A. Prompts and the acceptance rubric · pp. 36, 38, 39
  • The acceptance rubric requires hard rules, prescriptive reporting templates, count-printing code, a self-check checklist, real tools, and runnable code in each capability section.The hard rules cover intermediate counts, named entities, method justifications, citations, limitations, and not looking up the dataset’s source paper.Source: A. Prompts and the acceptance rubric · pp. 39–40
  • The critique must derive task-specific peer-review expectations from sample tasks and instructions, then close gaps in the draft skill without listing those expectations as output.For deterministic grading, every quantitative step must include a runnable check against staged data and a decision rule, rather than an unsupported directive.Source: A. Prompts and the acceptance rubric · p. 40
  • Skills should verify groupings and cutoffs from data, report observed and extrapolated estimates when targets exceed the observed range, and avoid combining features without direction checks.For derived groupings, check feature directions and exclude the outcome from the score defining groups; for cutoffs, use a histogram or sorted-gap scan to place the threshold between modes. For extrapolated estimands, report both the observed-range and extrapolated values plus the extrapolation distance.Source: A. Prompts and the acceptance rubric · pp. 40–41
  • When estimators disagree, skills should identify shared assumptions, include a check that does not share them, and select from data rather than averaging; answer keys require literal interpretation of each key’s qualifiers.For each answer key, state the implied population, subset, conditioning, and normalization before computing.Source: A. Prompts and the acceptance rubric · p. 41
  • For multi-key answers, agents should assert required algebraic or logical relationships before submission, since violations show that at least one jointly graded key is wrong.Catching such inconsistencies before submission is free.Source: A. Prompts and the acceptance rubric · p. 41
  • The rubric requires checking index comparability and join counts, auditing values against stated units and plausibility, and avoiding unsupported prescriptions in favor of data-visible selection criteria.Specific methods, estimators, thresholds, packages, or parameter settings should be named as correct only when supplied materials provide checkable evidence.Source: A. Prompts and the acceptance rubric · p. 41

B. Sources of the skill libraries

The three skill libraries draw on different named sources: benchmark repositories, ProteinGym acknowledgements, and benchmark task definitions. Their inclusion rules preserve runnable coverage and, for protein methods, avoid preselecting only high-ranked methods.

B. Sources of the skill libraries

  • Each library begins from a named source, and the three libraries differ in what that source is.Source: B. Sources of the skill libraries · p. 41
  • The drug property library covers 12 public repositories across 22 processor-runnable leaderboards, with each repository serving between 1 and 21 leaderboards.The registry specifies the processor variant for MapLight-TDC because reproductions run without an accelerator.Source: B. Sources of the skill libraries · p. 42, Table 2 · p. 42
  • All 24 ProteinGym methods with GitHub code were retained as skills, while three methods without a repository were excluded from the repository-based library.Median rank and last-place count were recorded for post-run scoring rather than used to filter the library in advance.Source: B. Sources of the skill libraries · p. 42
  • GeneBench-Pro receives 10 open-analysis skills, one per problem, alongside a directory of shared loci..Source: B. Sources of the skill libraries · p. 42

Drug property prediction

Figure 2 samples one leaderboard from each of the five ADMET categories, while the supplementary figures show all 22 leaderboards.

Drug property prediction

Genomic and single-cell analysis

The main text reports a mean over GeneBench-Pro’s 10 problems, and Table 3 exposes their individual spread. A supplementary figure separately presents all 22 absorption leaderboards in their own metrics.

Genomic and single-cell analysis

  • The main text reports the mean over GeneBench-Pro’s 10 problems, while Table 3 shows the problems individually and makes the spread visible.Source: Genomic and single-cell analysis · p. 42
  • Supplementary Fig. 1 presents each of the 22 absorption leaderboards in its own metric, without normalizing or averaging across panels.AUROC, AUPRC, and Spearman improve upward, while MAE improves downward.Source: Figure 1 · p. 43
  • The supplementary figure’s grey bars show published values and deviations; other arms show spread across five training seeds, not agent variability.Each entry was attempted once, so the figure does not capture variation on a second run.Source: Figure 1 · p. 43

Protein variant effect prediction

The supplementary tables provide assay results for every protein-benchmark arm and burial measurements for the marked positions.

Protein variant effect prediction

Statistical genetics

Table 6 provides every cell behind main-text Figure 6, allowing readers to check the numerical basis for its asserted orderings.

Statistical genetics

  • Table 6 lists every cell behind Figure 6 so readers can verify each asserted ordering against its underlying number.The table supplies a numerical check on the figure’s claims.Source: Statistical genetics · p. 43

D. Protocol, limits and resources

The supplement documents run limits, reproducibility checks, scoring rules, and data resources. It also clarifies that the reported error bars do not establish run-to-run stability.

D. Protocol, limits and resources

  • Runs start in fresh virtual environments, and success requires both a completion sentinel and a SUCCESS status line.Checking only tool and assistant messages prevents the system prompt’s sentinel instruction from counting as evidence of completion.Source: D. Protocol, limits and resources · pp. 43–44
  • The lesson store contains 104 lessons, while runs and individual command outputs receive capped subsets of matched lessons.Eleven regular expressions classify failures into nine classes.Source: D. Protocol, limits and resources · p. 44
  • Supplementary Table 3 reports GeneBench-Pro partial credit and binary verdicts for each of the ten problems.Partial credit ranges from 0 to 100; the mean of 31.4 lies between outright recoveries and problems that score nothing.Source: Table 3 · p. 46
  • Each entry was attempted once, so the reported seed-based error bars do not measure run-to-run stability.The authors draw no conclusion about whether a second attempt would land in the same place.Source: D. Protocol, limits and resources · pp. 45–46
  • Supplementary Tables 4–6 report all protein-benchmark assay scores, burial measurements for marked positions, and every fine-mapping grid cell.The fine-mapping rows average 50 replicates against the article’s 100.Source: Table 4 · p. 47, Table 5 · p. 47, Table 6 · p. 48
  • Each run ends when its first turn, cost, command-time, or wall-clock cap binds, with task-specific ceilings listed in Table 7.The fine-mapping run raised both turn and cost limits after seven earlier attempts ended against a ceiling.Source: D. Protocol, limits and resources · p. 43, Table 7 · p. 49
  • The fine-mapping rubric assigns 100 points across eight claims, weighting relations, quantities, and whether the work exists.Relations and quantities receive 40 points each; existence receives 20, with fixed tests for claim-level credit.Source: D. Protocol, limits and resources · pp. 44–45, Table 8 · p. 49
  • The data-availability table lists datasets, releasing projects, versions, and terms; its UK Biobank genotype panel has no accession number.Individual-level UK Biobank data cannot be redistributed and are available to approved researchers through the Access Management System.Source: Table 9 · p. 50
Loading 2609.13437v1…