Source-linked AI summary
Evaluating and improving AI agents for sequencing protocol understanding
Chi-Lam Poon, Michael Goneos, Zezheng Song, Xi Chen
TL;DR
Sequencing-protocol benchmarks rarely test whether agents can reconstruct exact library structures and molecular workflows, a gap relevant to customized assays. LibStructBench evaluates document-grounded reconstruction, and agents often approximate final libraries but remain unreliable at exact molecular-state and workflow reconstruction.
Problem
Existing scientific-agent benchmarks rarely assess exact reconstruction of sequencing-library structures and generation workflows, despite their relevance to analyzing customized assays.
Source: §1 · pp. 1–2Method
LibStructBench evaluates agents on document-grounded reconstruction of final-library structures, workflow-linked oligo families, and molecular-state transition graphs.
Source: §1 · p. 2Results
Takeaways & Limitations
The findings point to explicitly grounding and verifying molecular structures and transformations against protocol-specific evidence.
Source: §1 · p. 2Canonical-reference agreement does not independently establish biological correctness or downstream usability, and chemically valid representations may be penalized when they differ from canonical graphs.
Source: §4 · p. 9
Abstract
from arXiv · showhide
Existing scientific-agent benchmarks rarely evaluate whether agents can reconstruct the exact molecular structures and generation workflows specified by sequencing protocols, a capability critical for analyzing customized sequencing assays. We introduce LibStructBench, a document-grounded benchmark for reconstructing sequencing-library structures and molecular workflows. Our experiments show that frontier agents often produce approximately correct reconstructions but remain unreliable at exact reconstruction, particularly in molecular-state representation and workflow connectivity. Switching the underlying language model produced larger average performance differences than switching agent harnesses, with a statistically significant difference only for molecular-state reconstruction. Additional knowledge improved oligo-family recovery in a small pilot without improving mean molecular-workflow scores. Autonomous accumulation of procedural rules and cross-protocol examples showed non-monotonic validation performance and no mean held-out molecular-workflow improvement in the tested trajectory. These findings highlight the need for more controlled and verifiable approaches to scientific reasoning about molecular processes in complex experimental protocols.
1 Introduction
Sequencing-read processing depends on knowing the library’s molecular structure, especially for customized assays, but establishing that structure and tracing molecular workflows can require substantial manual interpretation. LibStructBench evaluates whether agents can reconstruct library structures and workflows from protocol evidence; experiments found approximate reconstructions common but exact reconstruction unreliable.
1 Introduction
- Adapter, barcode, UMI, position, and strand-orientation details determine how raw reads are extracted, trimmed, demultiplexed, and quantified.Customized single-cell assays may differ substantially from standard commercial protocols.Source: §1 · p. 1
- Establishing structures for new assays can require manual experimental-design interpretation and custom code, while diagnosing failed runs also requires molecular intermediates and operation steps.Agents could potentially reconstruct libraries, generate preprocessing logic, inspect raw data, and trace quality-control failures to candidate experimental steps.Source: §1 · p. 1
- LibStructBench divides protocol reconstruction into final-library structure, workflow-linked oligo families, and a graph of molecular states and transitions, with human-curated records audited against protocol evidence.Relevant evidence may be distributed across protocol text, supplementary files, tables, and figures.Source: §1 · p. 2
- Frontier LLMs often produce approximately correct final-library sequences but remain unreliable at exact reconstruction, with discrepancies in molecular-state assembly and workflow connectivity.Protocol-name-only controls also showed substantial reconstruction capability without supplied documents, underscoring the importance of checking protocol-specific information against primary evidence.Source: §1 · p. 2
2 Benchmark construction and validation
LibStructBench uses curated protocols and frozen source-document bundles to evaluate three linked reconstructions of sequencing libraries and their generation workflows. Its ground truth was audited against source evidence, with human experts adjudicating substantive corrections.
2 Benchmark construction and validation
- The benchmark began with more than 70 curated commercial and academic protocols spanning gene expression, chromatin accessibility, DNA methylation, genomic DNA, and multi-omic assays.Evidence bundles included available protocol or technical documentation, primary articles, supplementary files, oligo tables, and molecular figures; agents could not search the web or retrieve external protocol-specific information.Source: §2 · p. 2
2.1 Benchmark tasks and output representations
- Task 1 measures reconstruction of each defined final library’s canonical 5′ →3′ structure using normalized sequence similarity based on Levenshtein distance.Fixed nucleotide segments and typed variable regions are normalized before scoring.Source: §2.1 · p. 2
- Task 2 evaluates workflow-linked oligo families through their sequences and metadata, matching predicted families to required, source-recoverable families while neutralizing eligible optional or non-recoverable matches.Records include roles, molecular kinds, orientations, ordered components, and modifications; the weighted similarity includes sequence, modification, kind, orientation, and role.Source: §2.1 · pp. 3–4
- Task 3 represents library generation as a directed state-transition graph, recording molecular strands and architecture, operations, linked oligos, products, and carried-forward or discarded products.Shared upstream steps can branch toward modality-specific or alternative terminal states; workflow states and transitions are aligned through partial one-to-one matching.Source: §2.1 · p. 4
2.2 Ground-truth construction, audit, and adjudication
- Ground-truth auditing converted legacy records to the benchmark schema, compared candidate records against frozen sources, and had human experts review and adjudicate substantive findings before approved changes were applied.Of 69 confirmed findings, 54 (78.3%) were categorized as legacy-curation errors; this describes error causes, not the origin of accepted corrections.Source: §2.2 · pp. 4–5
3 Experiments and results
Across interface controls, baseline comparisons, and improvement experiments, agents reconstructed library components more reliably than complete molecular workflows. Model differences were most evident for molecular-state reconstruction, while added knowledge and cumulative learning did not consistently improve workflow scores.
3.1 Interface and PDF-conversion controls
- Native coding agents showed no statistically detectable Task 1 accuracy improvement over direct APIs, and only 3–5 of 20 protocols were reconstructed exactly across configurations.Human text, PyMuPDF, pypdf, and Docling recovered all 80 text-layer sequences exactly, compared with 71/80 for MinerU OCR; sequences embedded in figures were not assessed.Source: Figure 2 · p. 5, §3.1 · p. 5
3.2 Baseline molecular-workflow reconstruction
- Across 20 protocols, Task 2 oligo-family F1 ranged from 0.724 to 0.774 and Task 3 molecular-state F1 from 0.495 to 0.723; no system led all metrics.Model-switch differences exceeded harness-switch differences across four metrics, but only molecular-state F1 was significant (∆=0.079, pointwise 95% CI [0.038, 0.124], pBonf=0.016).Source: Figure 3 · p. 6, §3.2 · p. 6
- Repeated GPT-Codex runs had broadly consistent aggregate scores but substantial protocol-level variability, while tested scoring changes preserved leading systems but affected absolute scores and close rankings.The broad benchmark retained single-execution comparisons because repeated evaluations across models and harnesses were computationally costly.Source: §3.2 · pp. 6–7
- Without source bundles, name-only prompting lowered mean F1 across all three systems and four native-task metrics, but none of the 12 paired comparisons remained significant after correction.The controls did not establish training-data memorization or that source bundles are unnecessary.Source: §3.2 · p. 7
- Across 100 outputs, state-assembly and workflow-topology or disposition discrepancies occurred in every output, while missing information occurred in 88 and unsupported additions in 77.Categories were nonexclusive, so their frequencies describe disagreement with the canonical reconstruction rather than independent failure mechanisms.Source: §3.2 · p. 7
3.3 Improving molecular-workflow reconstruction
- In a five-protocol pilot, combined general knowledge and cross-protocol examples raised oligo-family F1 to 0.877, a paired improvement of 0.050, while no augmentation exceeded baseline Task 3 means.The Task 3 paired confidence intervals included zero.Source: §3.3 · p. 7
- On held-out protocols, adding the knowledge base raised Task 2 oligo-family F1 from 0.560 to 0.621 (difference 0.061; 95% CI [-0.024, 0.159]), but Task 3 means remained below baseline.Validation performance was non-monotonic: C30 was selected, while C58 scored below C0 on all four validation metrics; C30 rules alone improved no held-out Task 3 metric.Source: Figure 4 · p. 8, §3.3 · p. 8
- Cumulative learning updated procedural rules and a knowledge base from training protocols, then evaluated frozen checkpoints on the same five validation protocols without learning from their outputs.The trajectory used 58 training, five validation, and 10 held-out test protocols.Source: §3.3 · p. 8
4 Discussion
The discussion emphasizes a gap between identifying library components and reconstructing molecular states and workflow connectivity. It presents model sensitivity and task-dependent knowledge effects as findings from the tested configurations, while recommending controlled, verifiable learning for future work.
4 Discussion
- Agents often identified operations but struggled with intermediate states and connectivity, and benchmark scores measure agreement with canonical references rather than independently validated biological correctness or downstream usability.Repeated GPT-Codex runs also showed protocol-level variability despite similar aggregate performance.Source: §4 · p. 9
- Across tested configurations, model switching produced larger average differences than harness switching, with statistical significance only for molecular-state reconstruction.The discussion identifies intermediate molecular architectures as particularly sensitive to the underlying model.Source: §4 · p. 9
- Additional knowledge and cumulative learning improved oligo-family recovery in limited tests without improving mean molecular-workflow scores, so component recovery did not necessarily translate into workflow integration.The accumulated-knowledge held-out oligo-family point estimate increased with uncertainty spanning zero.Source: §4 · p. 9
- Canonical graph matching may penalize chemically valid intermediates when their molecular-state or workflow representation differs from the reference.The discussion also notes possible complications from selecting and integrating protocol-specific evidence and from learned rules that omit intermediates or alter connectivity.Source: §4 · p. 9
A Related work
- LibStructBench complements structured assay descriptions by testing whether agents can reconstruct them from heterogeneous protocol documents; the discussion proposes evidence-verified provisional workflows and reversible, validated learning.The benchmark uses publicly documented protocols with frozen source bundles as an auditable stand-in for the intended setting of new or unpublished protocols.Source: §4 · p. 9, A Related work · p. 13
B Human-agent ground-truth audit and adjudication
Ground-truth curation combined agent-proposed corrections with human adjudication and deterministic application; the parser-control analysis evaluated sequence recovery across verified strings from five protocols.
B Human-agent ground-truth audit and adjudication
- Claude Code proposed corrections to existing curation, a human curator adjudicated substantive findings, and deterministic code applied only approved decisions.The audit also converted existing curation to the canonical representation and verified it against primary evidence.Source: Figure S1 · p. 14
- A dsciATAC-seq 6-nt segment was relabeled from UMI to TN5_INDEX after the Methods and oligo table identified it as one of 96 Tn5 well barcodes.The correction changed [UMI:6] to [TN5_INDEX:6] in Tasks 1 and 3.Source: B Human-agent ground-truth audit and… · p. 14
- For scDamID, the supplement’s 57-nt P5 primer conflicted with the 58-nt primer required by the final-library structure, so the longer form was retained.PDF re-extraction confirmed that the source omitted the leading A.Source: B Human-agent ground-truth audit and… · p. 14
C PDF-parser-control detail
- The PDF-parser control assessed exact recovery and mean similarity for 80 verified nucleotide strings from five protocols.For each reference, it used the highest normalized Levenshtein similarity among extracted sequences.Source: Figure S2 · p. 14
D Variability across repeated agent executions
Repeated GPT-Codex executions changed aggregate macro-F1 estimates little overall, while individual-protocol scores varied substantially; the analysis was limited to that system.
D Variability across repeated agent executions
- The variability analysis compared one versus three GPT-5.6 Sol Max with Codex executions per protocol across the 20-protocol baseline panel.The three-execution analysis included the original run; Task 2 averaged 19 informative protocols, while each Task 3 metric averaged 20.Source: D Variability across repeated agent… · p. 15, Table S1 · p. 15
- 5/19 oligo-family, 9/20 transition, 8/20 state, and 12/20 typed-edge comparisons had within-protocol F1 ranges above 0.10.The largest observed range was 0.471 for typed-edge F1 in s3-ATAC; three-execution macro-F1 estimates differed from original scores by 0.005–0.024 across metrics.Source: Figure S3 · p. 15, D Variability across repeated agent… · p. 15
- The analysis covered GPT-Codex only, and the main figures retained one selected execution per system and protocol.Its protocol-bootstrap intervals reflect across-protocol uncertainty conditional on those outputs, not execution variability; variability for other systems and interventions was not quantified.Source: D Variability across repeated agent… · p. 15
E Reconstruction without protocol source bundles
The source-bundle-free test withheld protocol documents while retaining generic instructions and tools. Name-only prompting increased sequence similarity in Task 1, but lowered native-agent F1 across Task 2–3 metrics without significant corrected differences.
E Reconstruction without protocol source bundles
- The name-only test withheld source documents and retrieval access while keeping generic instructions, output schemas, and local tools available.It assessed performance without supplied evidence, rather than distinguishing memorization from learned knowledge or reasoning.Source: E Reconstruction without protocol… · p. 16
- Name prompting generally increased Task 1 sequence similarity for all models, but only Claude remained significant after correction (padj = 0.0215).Exact reconstructions fell from 4 to 2 for GPT, 5 to 3 for Claude, and 4 to 3 for Gemini, out of 20 protocols per model.Source: E Reconstruction without protocol… · p. 16, Table S2 · p. 17
- Name prompting yielded lower mean F1 across all native-agent Task 2–3 systems and metrics, but no difference remained significant after correction (all padj > 0.05).Comparisons used one Name and one original Bundle execution per system and protocol; Task 2 excluded ddSEQ.Source: E Reconstruction without protocol… · p. 16, Table S2 · p. 17
F Complementary model-harness performance metrics
Complementary performance metrics are displayed as heatmaps across the same ten protocols and nine model–harness systems used in Figure 3b.
F Complementary model-harness performance metrics
- Heatmaps show complementary metrics across ten protocols and nine model–harness systems, with system and protocol order fixed to the molecular-transition-F1 clustering and color scales spanning 0–1.The fixed ordering permits direct comparison across metrics.Source: Figure S4 · p. 18
G Cost analysis and model-harness effects
Model-switch magnitudes exceeded harness-switch magnitudes across all four metrics, but only molecular-state F1 showed a statistically significant difference. Figure S5 describes estimated API-equivalent costs and performance, but the supplied passages report no cost values.
Cost analysis
- Figure S5 presents per-protocol costs and system means, alongside estimated API-equivalent cost versus mean performance across four F1 metrics.The supplied passages identify the plotted comparisons but do not provide cost values or a cost-performance result.Source: Figure S5 · p. 19, G Cost analysis and model-harness… · p. 19
Model-harness effect analysis
- Across all four metrics, mean model-switch magnitudes exceeded harness-switch magnitudes; only molecular-state F1 had a significant model-minus-harness contrast (mean 0.0793, 95% CI [0.038, 0.124], pBonf. = 0.0156).Intervals are pointwise 95% protocol-bootstrap CIs, conditional on the selected executions.Source: G Cost analysis and model-harness… · p. 19, Figure S6 · p. 19
- Only molecular-state F1 showed a detectable model main effect; no harness main effect or model-harness interaction was detected for any metric.State F1 averaged over the two harnesses was 0.655 for Claude, 0.612 for GPT, and 0.459 for Gemini; nonsignificance does not establish equivalence.Source: G Cost analysis and model-harness… · p. 19
H Baseline output failure modes
The baseline failure-mode materials identify output discrepancies across protocols and systems, while companion tables describe discrepancy categories and the sources and donor protocols used for external-knowledge improvement.
H Baseline output failure modes
- Figure S7 measures the fraction of ten protocols and ten systems with at least one output discrepancy, with values from 0 to 1.The supplied passages do not report the plotted fractions or identify the most common discrepancy.Source: Figure S7 · p. 20, H Baseline output failure modes · p. 20
- Table S3 lists the deterministic output-discrepancy categories used in the baseline error analysis.The supplied passage gives the table's purpose, not its category contents.Source: Table S3 · p. 20
I External knowledge for agent improvement
- Tables S4 and S5 document the general molecular-method primer sources and the protocols selected as cross-protocol memory-pilot donors.The primer summarizes general operations rather than protocol-specific answers; donor protocols were selected for shared molecular features with validation protocols.Source: Table S4 · p. 21, Table S5 · p. 21
J Trajectory-based behavioral changes across autonomous-learning checkpoints
Checkpoint trajectories show how learned rules and examples coincided with changes in reconstructed workflows and scores, including cases where learned representation conventions diverged from the benchmark's canonical graph.
Trajectory-based behavioral changes across autonomous-learning checkpoints
- Behavioral comparisons used selected executions at baseline, C30, C40, and C58 on the same frozen five-protocol validation panel, and do not establish the causal effect of an individual rule or retrieved example.The corresponding table defines learned rules as protocol-neutral instructions and exemplar records as ground-truth-derived cross-protocol examples available for retrieval.Source: J Trajectory-based behavioral changes… · p. 22, Table S6 · p. 22
- Task 3 deterioration was already apparent at C35 for ddSEQ and HyDrop-RNA, while CEL-seq informative Task 2 F1 declined from 0.600 at C30 to 0.450 at C40..Source: J Trajectory-based behavioral changes… · p. 22
- C30-to-C40 ddSEQ transition F1 fell from 1.000 to 0.493, state F1 from 0.830 to 0.449, and typed-edge F1 from 1.000 to 0.364 after the agent split branches that the benchmark connects through a shared fraction-separation transition.A learned rule instructed separate workflows for co-processed but physically distinct molecules; ddSEQ has no required recoverable oligo families and is excluded from the Task 2 macro-average.Source: J Trajectory-based behavioral changes… · p. 22
- C30-to-C40 HyDrop-RNA Task 2 F1 fell from 0.915 to 0.738, state F1 from 0.401 to 0.159, transition F1 from 0.666 to 0.587, and typed-edge F1 from 0.609 to 0.526.The trajectory represented a completed bead construct rather than separate oligos, omitting early bead-construction states and transitions that the benchmark scores; the observed trajectories do not isolate individual-rule effects from execution variability.Source: J Trajectory-based behavioral changes… · pp. 22–23
- At C58, scRRBS state F1 rose from 0.521 at C40 to 0.712, while transition F1 fell from 0.883 to 0.741 and typed-edge F1 from 0.696 to 0.615.The finer event granularity was associated with additional unmatched transitions and edges; chATAC-seq also represented adapter reagents as independent molecular states, coinciding with lower state precision and typed-edge agreement.Source: J Trajectory-based behavioral changes… · p. 23
K Sensitivity analysis
Sensitivity analyses varied similarity weights and matching thresholds separately, finding that broad baseline comparisons were generally robust while absolute scores, close rankings, and some learning-related results remained sensitive. Typed-edge F1 was more threshold-sensitive than transition or state F1, and the analyses used fixed predictions rather than repeated executions.
Sensitivity analysis
- Across 640 scoring configurations, leading native systems remained unchanged for oligo-family, molecular-state, and typed-edge F1, while typed-edge F1 was more threshold-sensitive than transition or state F1.Broad baseline findings were robust to tested assumptions, although absolute scores and close system rankings warrant cautious interpretation.Source: K Sensitivity analysis · p. 23
Sensitivity to similarity weights
- The leading systems for oligo-family and molecular-state F1 remained unchanged across tested weight configurations, while transition rankings varied: Gemini ranked first in 179 of 200 sampled configurations and GPT in the remaining 21.Absolute F1 scores varied with component weighting; the default weights were chosen heuristically rather than estimated from experimental data.Source: K Sensitivity analysis · pp. 23–24, Figure S8 · p. 24
Sensitivity to matching thresholds
- In the autonomous-learning validation experiment, threshold changes produced maximum F1 shifts of 0.055 for transitions and 0.132 for typed edges; changing only the state threshold from 0.25 to 0.40 shifted typed-edge F1 by 0.117 versus 0.025 for state F1.Across the native-system comparison, maximum changes in transition, state, and typed-edge F1 were 0.035, 0.031, and 0.069, respectively.Source: K Sensitivity analysis · p. 24, Table S8 · p. 24, Figure S9 · p. 25
- Typed-edge F1 was more sensitive to matching thresholds than transition or state F1, consistent with connectivity scoring depending on molecular-state alignment and changes in matched states propagating to associated edges.Graph-topology scores should be interpreted alongside molecular-state alignment and the canonical graph representation.Source: K Sensitivity analysis · pp. 24–25
Robustness of knowledge augmentation and cumulative learning
- External knowledge consistently increased oligo-family F1 by 0.022–0.129 across tested scoring configurations, but baseline retained the highest mean scores on all three Task 3 metrics.The final-checkpoint oligo-family difference ranged from −0.021 to 0.015, while all three Task 3 endpoint differences remained negative; held-out knowledge-base supplementation raised oligo-family F1 by 0.049–0.075, but its default-scoring 95% interval included zero ([-0.024, 0.159]).Source: K Sensitivity analysis · pp. 25–26, Figure S10 · p. 25
Limitations
- These sensitivity analyses reused fixed agent predictions, so they assess scoring-assumption robustness rather than execution-to-execution variability and do not establish that default weights are biologically optimal.Model-versus-harness statistical tests were not repeated across all scoring configurations, and canonical graph matching may not accommodate every chemically valid representation.Source: K Sensitivity analysis · pp. 23, 26