PaperlayerView exact revision on arXiv

Source-linked AI summary

Notes2Skills: From Lab Notebooks to Certainty-Aware Scientific Agent Skills

Shi Liu, Jiayao Chen, Chengwei Qin, Yanqing Hu, Jufan Zhang, Linyi Yang

arXiv:2606.11897v1cs.CL

TL;DR

Lab notebooks mix facts, uncertain judgments, and suggestions, but existing scientific-text approaches underexplore them as inputs for AI agents. Notes2Skills preserves certainty through extraction and skill compilation, and is the only tested configuration that avoids both observed failure modes across three wet-lab sessions.

  • Problem

    Lab notebooks intermingle facts, uncertain judgments, and suggestions, while prior scientific-text work underexplores them as inputs to AI agents.

  • Method

    Notes2Skills extracts notebook directives as FACT, JUDGMENT, or SUGGESTION, then compiles source-linked, certainty-labeled MetaSkills for agents.

  • Results

    F1=0.737 on binary directive detection, up from 0.682 for the strongest zero-shot baseline; Notes2Skills alone avoids both observed downstream failures.

  • Takeaways & Limitations

    Preserving author certainty makes lab notebooks verifiable skill sources for AI-for-Science across uncertainty-heavy and FACT-dominated sessions.

  • Takeaways & Limitations

    The evaluation relies on sustained collaboration with senior experimental researchers, annotator training, and repeated adjudication, making it harder to scale.

Abstract

from arXiv · show

Scientific discovery workflows usually contain and rely heavily on lab notes, where researchers record observations, interpret uncertain results, and plan follow-up experiments. Such informative lab notes preserve evolving scientific reasoning and author uncertainty, rather than polished final results exhibited in publications, providing a valuable opportunity for AI to engage in scientific exploration at a more comprehensive and deeper level. However, most prior work on scientific text focuses on papers, protocols, or structured databases, leaving informal laboratory notes underexplored as inputs to AI agents for science. This gap matters because lab notes often intermingle validated observations, tentative judgments, and possible experimental next steps within the same passage. If these signals are conflated, an AI agent may mistake uncertain scientific judgments for confirmed conclusions or executable actions. To this end, we present Notes2Skills, a two-stage framework for turning lab notebooks into verifiable skills for scientific AI agents while preserving the author's certainty. Across seven conditions and three wet-lab sessions, Notes2Skills is the only configuration that neither mistakes uncertain notes for firm instructions nor discards firm ones. We show that certainty preservation is the missing piece between lab notebooks and reliable agent skills, opening a path toward safer AI co-scientist systems.

1 Introduction

Experimental notebooks mix factual observations, uncertain judgments, and suggestions, so treating all procedural text as equally firm can cause unsafe agent behavior. Notes2Skills addresses this by preserving author certainty and reports improved directive detection across three notebook corpora and wet-lab sessions.

  • Motivation: Experimental notebooks intermingle facts, uncertain judgments, and suggestions whose similar surface forms should trigger different downstream behaviors.Examples include a sharp reading drop, uncertainty about a second read, and trying fresh buffer next time.
  • Motivation: Equal treatment produces uncertainty laundering and directive loss by converting tentative judgments into firm decisions or failing to preserve firm directives.Uncertainty laundering may cause agents to act on unresolved interpretations and discard underlying data.
  • Approach: Notes2Skills treats author certainty as a safety boundary for agent-loadable skills generated from single-author experimental notebooks.The framework targets notebooks written close to experimentation, where certainty directly informs decision-making and is entangled with action content.
  • Evaluation: 461 annotated segments span FreeNotes, ONS, and WLP, while downstream skill loading is tested on three real wet-lab sessions.Compiled skills guide file-level data-handling decisions over instrument traces.
  • Results: F1=0.737 for binary directive detection, up from 0.682 under the strongest zero-shot configuration.The result is achieved by the best of six model–prompt configurations across the three corpora.

2 Related Work

Related work spans scientific procedure extraction, compilation of procedural text into executable agent skills, and NLP methods for representing uncertainty and factuality. Notes2Skills builds on these lines while addressing lab notes as a distinct setting involving uncertain scientific reasoning.

  • Scientific Procedure Extraction from Protocols: Scientific procedure extraction has mainly focused on curated, prescriptive documents, progressing from action extraction to protocol planning and flexible-schema materials-science extraction.This line includes Wet Lab Protocols, X-WLP, BioPlanner, and NERRE.
  • Compiling Procedural Text into Agent Skills: SYNTACT and Flow-of-Action compile procedural text into executable skills or workflows by resolving ambiguity before execution.Both systems treat ambiguity in prescriptive text as a defect to eliminate before acting.
  • Uncertainty and Factuality in Text: NLP research has developed methods for detecting hedges and speculation and modeling uncertainty, factuality, assertion status, and modality across scientific and clinical text.Examples include BioScope, the CoNLL-2010 shared task, multi-axis factuality and modality schemas, clinical assertion classification, and CMED.

3 Task Formalization

Notes2Skills formalizes certainty-preserving conversion of notebook text into agent decisions through two stages: directive extraction and deterministic MetaSkill compilation. The framework constrains agent actions by certainty labels, source-linked capsules, runtime evidence, auditability, and conservative human review.

  • 3 Task Formalization: Notes2Skills preserves author certainty from notebook text through agent decisions using a two-stage framework.The framework is shown in Figure 3 and connects notebook statements to downstream agent behavior.
  • 3 Task Formalization: Stage 1, Epistemic Directive Extraction, identifies analysis-relevant notebook statements and labels them FACT, JUDGMENT, or SUGGESTION.Each extracted statement is treated as a directive for downstream processing.
  • 3.2 Stage 2: MetaSkill Compilation: Stage 2 compiles labeled statements into a machine-readable MetaSkill whose actions remain linked to source excerpts and certainty labels.The artifact must satisfy faithfulness, actionability, auditability, and conservative fallback to human review.
  • 3.1 Stage 1: Epistemic Directive Extraction: For each notebook segment, EDE determines whether a directive should be preserved, assigns one of five directive types, and records its certainty label.The formal tuple notation and type-specific attributes are specified in Appendix E.
  • 3.1 Stage 1: Epistemic Directive Extraction: FACT may support strong action when policy and signal evidence agree, JUDGMENT defaults to review-preserving handling, and SUGGESTION remains advisory.Labels are assigned using linguistic cues described in Appendix D.
  • 3.2 Stage 2: MetaSkill Compilation: The compiler is entirely deterministic, inheriting capsule fields from EDE records or fixing them through domain configuration.Stage 2 auditing checks data provenance rather than model quality, while an LLM compiler could collapse uncertainty into an unverified action.
  • 3.2 Stage 2: MetaSkill Compilation: Certainty labels become action commitment levels: FACT permits strong potentially irreversible operations, JUDGMENT permits only review-preserving actions, and SUGGESTION carries no file-level commitment.At runtime, strong action requires agreement between capsule commitment level and file signal evidence.
  • 3.2 Stage 2: MetaSkill Compilation: The MetaSkill stores one capsule per directive and exposes commitment levels to the runtime gate; removing commitment information causes the gate to collapse.The action-only schema leaves the gate with nothing to condition on, whereas the full schema is described in Appendix E.

4 Datasets

Notes2Skills is evaluated on three corpora spanning informal laboratory notes to high-formality protocols. The datasets also support strong annotation agreement for directive detection and ordinal certainty.

  • Dataset scope: Three corpora span a formality spectrum for validating Notes2Skills.The corpora are FreeNotes, ONS, and WLP.
  • Dataset scope: FreeNotes contains 201 segments from experimental notebooks by two senior researchers at two institutions, with Chinese-English code-switching near experimentation time.Three sessions totaling 48 downstream files are used for validation because each file aligns notebook directives, raw instrument records, and an additional evidence source.
  • Dataset scope: ONS contains 155 segments from nine openlabnotebooks.org entries and serves as a semi-formal boundary case between private notes and polished protocols.The entries are licensed CC BY 4.0.
  • Dataset scope: WLP contains 105 prescriptive segments from the Wet Lab Protocols corpus and serves as a high-formality protocol control.Its procedural segments can be expressed under the same schema.
  • Formality and agreement: Formality follows WLP > ONS > FreeNotes on at least two of three proxies, while annotation agreement reaches κ≥0.709 for directive detection and QWK ≥0.732 for ordinal certainty.The proxies are surface regularity, code-switching rate, and author-audience distance; subsamples included 60 FreeNotes, 60 ONS, and 90 WLP segments.

5 Experimental Setup

The experimental setup evaluates directive extraction, certainty-preserving compilation, and downstream skill loading across controlled model, prompting, corpus, and executor conditions. It combines standardized metrics with three FreeNotes sessions and a stress test using predicted Stage 1 outputs.

  • Experimental design: The study runs two experiments and a compilation audit: Stage 1 directive extraction, Stage 2 preservation into MetaSkill capsules, and downstream skill loading on three FreeNotes sessions.The downstream evaluation also stress-tests the full stack by replacing adjudicated Stage 1 outputs with model-predicted ones.
  • Models and prompting: Three LLMs—GPT-4o1, Claude Sonnet 4.52, and Qwen-Max3—are evaluated with zero-shot and few-shot prompting at temperature 0 using document-stratified 80/20 splits.Split details, exemplar counts, and empty-response handling are provided in the appendices.
  • Evaluation metrics: Stage 1 reports macro-F1 for directive type and epistemic status, ordinal certainty agreement using QWKep, and Joint exact-match accuracy across all three predictions.The two macro-F1 metrics use the both-positive subset, while Joint is the fraction of segments whose three predictions exactly match gold.
  • Downstream evaluation: Downstream evaluation covers three FreeNotes sessions with 17, 22, and 9 files, sampling each condition–file pair five times under independent API calls across seven configurations.The configurations include an external raw-prompting baseline and six Notes2Skills ablations; only the loaded skill and executor setting differ.
  • Executor validation: The deterministic executor aligns each LLM proposal with the matched skill capsule and SignalFindings, emitting Authorize, Veto, Substitute, or Abstain outcomes.Strong actions pass only with explicit capsule authorization and supporting signal evidence for their parameters; the function is bit-exact for fixed Layer 2 outputs.

6 Results

Results show that Notes2Skills improves certainty-aware extraction and preserves all directives across corpora, while downstream ablations demonstrate that certainty-preserving schemas are necessary for reliable agent decisions. The proposed MetaSkill + executor configuration is the only tested configuration that avoids both uncertainty laundering and loss of firm actions across three wet-lab sessions.

  • Stage 1 extraction: Few-shot prompting consistently improves macro-F1 on directive_type, while certainty-label performance varies by backbone and remaining errors concentrate at the FACT–JUDGMENT boundary.Claude, GPT-4o, and Qwen-Max gain +0.18, +0.12, and +0.12 respectively; GPT-4o zero-shot reaches QWK 0.946.
  • Stage 2 preservation: Across three corpora, the deterministic compiler emits all 149 fixed EDE directives as inspectable skill capsules while preserving directive keys, certainty labels, schema fields, and source links.The corpus counts are FreeNotes 48, WLP 70, and ONS 31.
  • Downstream ablation: On saturation sessions, the external-LLM baseline has FLAG recall 0% and chance-level κ, but on Step-drop it matches the Action-only skill at κ = +0.80 and FLAG recall 100%.Every saturation FLAG_FOR_REVIEW gold file is routed to KEEP_FULL or TRUNCATE_AT.
  • Executor mechanism: Adding the executor to Action-only collapses Step-drop accuracy from 88.9% to 44.4% and κ from +0.80 to 0.00 because the schema lacks the authorization field required by the Veto rule.The executor therefore degenerates into a blanket downgrade filter and matches Always-FLAG_FOR_REVIEW on Saturation-B and Step-drop.
  • Uncertainty preservation: The Action-only schema reaches κ = +0.80 on Step-drop but fails on saturation, where FLAG recall is 21.4% and 0% and κ is +0.14 and +0.09.Raw notes also achieve only 50.0% on Saturation-A and 0% on Saturation-B, showing that actions alone or unstructured notes do not preserve uncertainty reliably.
  • Stress test: With model-extracted EDE predictions, Stage 1 recall remains high but precision falls in directive-dense regimes, and the proposed configuration degrades modestly on Saturation-A from κ +0.71 to +0.51.Stage 1 recall is 87.5%, 100%, and 92.3% across Saturation-A, Saturation-B, and Step-drop.

7 Conclusion

Notes2Skills preserves the distinction between fact, judgment, and suggestion as notebook content becomes an agent decision. Its pipeline compiles notes into MetaSkill artifacts that retain author certainty through extraction, loading, signal matching, and executor checking.

  • Conclusion: Notes2Skills preserves the author’s distinction between fact, judgment, and suggestion from lab notebooks.The paper identifies this distinction as a triage signal that current pipelines rarely preserve.
  • Conclusion: An experimental note compiles into a MetaSkill artifact and retains certainty through extraction, skill loading, signal matching, and executor checking.Figure 5 depicts this end-to-end process from notebook note to final decision.
  • Conclusion: 149 fixed directives are preserved across FreeNotes, ONS, and WLP.This preservation is attributed to Stage 2.

Limitations

The evaluation benefits from unusually strong experimental context and expert judgment, but reproducing this setup is difficult to scale because it requires sustained collaboration, training, and repeated adjustment.

  • Evaluation context: The notes were written near data collection, raw records were preserved, and expert collaborators judged how textual triage should affect later processing.These conditions make the evaluation more faithful than a standard crowd-sourced text task.
  • Scalability: Building FreeNotes required sustained collaboration with senior experimental researchers across two institutions.The collaboration burden contributes to the difficulty of scaling the evaluation.
  • Scalability: The workflow also required annotator training and repeated adjustment.These requirements add further operational demands beyond the experimental context itself.

Ethics Statement

The ethics statement describes consented, provenance-preserving notebook data use and emphasizes that certainty-aware skills support human review rather than replace human judgment. It also distinguishes LLM-assisted stages from deterministic compilation and notes human review of dataset annotations.

  • Data provenance: FreeNotes was contributed by two senior experimental researchers under a formal inter-institutional data-sharing agreement and with their consent, while ONS entries retain author attribution.The notebooks were authored independently of Notes2Skills, and no third-party personal or sensitive data is present.
  • Human oversight: Epistemic labels and provenance anchors are inputs to review-preserving decision policies, not authorizations to bypass human oversight.Compiled skills are intended for scientific workflows where downstream decisions may affect experimental data inclusion or exclusion.
  • Methodological safeguards: LLMs supported Stage 1 extraction and downstream agent evaluation, whereas Stage 2 preservation artifacts were produced by deterministic compilers from fixed EDE inputs.Any LLM-assisted annotation or drafting during dataset preparation was human-reviewed before inclusion.

A Exp 1: Full Per-Cell Results and Parse Rates … E MetaSkill Capsule Schema

The appendices provide detailed evaluation analyses, document API filtering and corpus-formality measurements, define adjudicated gold construction, and specify a certainty-preserving MetaSkill capsule schema with validity invariants.

  • A Exp 1: Full Per-Cell Results and Parse Rates: The appendix expands pooled evaluation with a complete 18-cell breakdown, parse statistics, per-corpus difficulty, few-shot calibration, and error-pattern analyses.These analyses are presented alongside the pooled metrics reported in Table 4.
  • A Exp 1: Full Per-Cell Results and Parse Rates: Few-shot exemplars recalibrate directive decisions: GPT-4o and Qwen-Max shift from conservative to aggressive FN/FP bias rather than uniformly increasing accuracy.GPT-4o changes from 17/11 to 11/14 FN/FP, while Qwen-Max changes from 18/13 to 9/17.
  • A Exp 1: Full Per-Cell Results and Parse Rates: The evaluation identifies recurring directive-type confusions, led by FLAG_DATA →PROTOCOL_CHANGE with 20 errors and PROTOCOL_CHANGE →CONDITION_CHANGE with 17.Two third-place confusion pairs each occur 13 times.
  • B API Access and Response Filtering: Five of 522 Stage 1 API calls returned empty responses after retries and were excluded from primary scoring, with affected cells logged.The remaining empty responses resulted from provider-side filtering and were retried through an independent endpoint when possible.
  • C Operational Formality Measurements: Corpus formality is measured separately through surface regularity, code-switching rate, and author-audience distance rather than a single collapsed score.Surface regularity is high in WLP, code-switching is high in FreeNotes, and author-audience categories differ across corpora.
  • D Annotation Guideline and Gold Construction: Stage 1 gold construction uses framework and corpus-specific annotation rules, while Exp 3 file-level golds combine notebook directives, Layer 1 findings, and session metadata.Two annotators independently label or cross-check samples, with disagreements adjudicated before evaluation.
  • D Annotation Guideline and Gold Construction: A deterministic compiler generates corpus-specific Stage 2 artifacts, including per-session FreeNotes skills and corpus-level WLP and ONS MetaSkills.The corpus-level outputs contain 70 WLP capsules and 31 ONS capsules.
  • E MetaSkill Capsule Schema: Every directive capsule preserves directive identity, source anchors, directive type, epistemic status, uncertainty markers, scope, cross-directive links, and SHA-anchored provenance.FreeNotes capsules additionally expose default actions, authorization evidence, candidate actions, and executor-specific truncation-boundary information when applicable.

F Downstream Accuracy Confidence Intervals

Table 8 reports Wilson 95% confidence intervals for file-majority accuracy, providing uncertainty checks on the decision-level estimates in Table 6. The intervals reflect evaluated files rather than API repeats and do not extend to aggregate-cell estimates of bAcc, macro F1, or κ.

  • Confidence-interval construction: Intervals are computed over each session’s file-majority outcomes, reflecting the small number of evaluated files rather than five API repeats per file.They serve as uncertainty checks on decision-level point estimates.
  • Confidence-interval construction: Confidence intervals for bAcc, macro F1, and κ require class-specific resampling and are not inferred from aggregate table cells.
  • Confidence-interval construction: Table 8 reports Wilson 95% confidence intervals for file-majority accuracy corresponding to Table 6.Each cell gives successes/files and the interval bounds in percentage points.

G Trivial Always-FLAG Baseline · H FreeNotes Legacy-Action Diagnostic (Exp 2-B) · I Cross-Channel Sanity Check Details

The trivial Always-FLAG predictor achieves misleadingly high raw accuracy on saturated sessions while balanced metrics collapse, whereas the final compiler preserves, downgrades, or rejects legacy actions by type discipline. Cross-channel diagnostics show strong agreement between author judgments, filename markers, and signal density, alongside large pre/post-flush differences.

  • G Trivial Always-FLAG Baseline: Always-FLAG_FOR_REVIEW produces high accuracy on saturation sessions because of class imbalance, while balanced accuracy, macro F1, and κ collapse to chance.Rare strong actions are never recovered, confirming executor-driven FLAG recovery is not a majority-class artifact.
  • H FreeNotes Legacy-Action Diagnostic (Exp 2-B): The FreeNotes diagnostic compares the final MetaSkill compiler with an earlier action-first representation to test preservation as defaults, candidates, or rejection.FreeNotes enables this direct mechanism comparison because it contains both final MetaSkill and earlier action-first artifacts.
  • H FreeNotes Legacy-Action Diagnostic (Exp 2-B): The compiler preserves licensed operational traces, downgrades many strong actions to candidates, and rejects actions violating directive-type discipline.This carries operational memory forward without silently converting every prior strong action into a default commitment.
  • I Cross-Channel Sanity Check Details: Step-drop files encode author judgment as a 1–5 ordinal intensity score derived from verbatim directive excerpts anchored to each file.For example, score 5 indicates observable events suggested as a cross-read anchor, while score 1 indicates uncertainty about whether events exist.
  • I Cross-Channel Sanity Check Details: Filename markers record author-selected labels such as Better, Not Sure, and After Flushing, while signal density measures translocation events per 100 s.These provide independent channels alongside author judgment for the cross-channel sanity check.
  • I Cross-Channel Sanity Check Details: 0.726 Spearman’s ρ(intensity, density) across nine files is significant at p = 0.027, while within-300 mV marker ordering reaches ρ = 1.000.The ordered files have translocation counts of 38, 128, and 1002, a 26-fold spread predicted monotonically by filename markers alone.
  • I Cross-Channel Sanity Check Details: 17.6× and 39.1× higher translocation totals occur in the pre-flush phase than post-flush at 100 mV and 200 mV, respectively.Filename markers are used only for the cross-channel diagnostic.
  • I Cross-Channel Sanity Check Details: The bootstrap 95% confidence interval for Step-drop predicted-EDE is [0.000, 0.556] from 2000 resamples of nine file-majority outcomes.Resampling used replacement and numpy.random.seed(42).

J Sensitivity to 𝜅Weighting · K Prompt Templates

Quadratic-weighted κ preserves the paper’s main sensitivity pattern: Notes2Skills dominates Action-only on saturation sessions, while Action-only retains an advantage on the FACT-dominated Step-drop session. The prompt templates define deterministic Stage 1 extraction, shared six-condition Layer 2 prompting, and conservative execution rules that separate uncertainty from explicit operational directives.

  • J Sensitivity to 𝜅Weighting: The quadratic-weighted κ check uses the same per-file gold labels and majority-vote predictions as Table 6, with weights="quadratic".Table 11 excludes the Action-only skill + executor and stress-test rows from the audited subset.
  • J Sensitivity to 𝜅Weighting: Under quadratic-weighted κ, Notes2Skills dominates Action-only on both saturation sessions: 0.857 vs 0.250 on Saturation-A and 1.000 vs 0.543 on Saturation-B.These margins are wider than under unweighted κ.
  • J Sensitivity to 𝜅Weighting: On the FACT-dominated Step-drop session, Action-only retains its vocabulary-alignment advantage: 0.941 vs 0.500 for the proposed configuration.
  • K.1 Stage 1: EDE prompt: Stage 1 uses a shared core system message plus corpus-specific framing, identical across GPT-4o, Claude Sonnet 4.5, and Qwen-Max under zero-shot and few-shot decoding at temperature 0.Few-shot prompts prepend k=6 exemplars for FreeNotes and k=5 for WLP and ONS, sampled once per corpus from the train fold with seed 42.
  • K.2 Exp 3: Layer 2 agent prompt: The Layer 2 external-LLM baseline uses a standalone prompt, whereas the other six conditions share BASE + SUFFIX plus an optional skill body and executor rules.The user message supplies the full notebook, file identifier, and canonical Layer 1 signal-findings JSON.
  • K.2 Exp 3: Layer 2 agent prompt: The raw-notes Layer 2 condition treats notebook excerpts as evidence rather than executable instructions and matches them cautiously against numerical signal findings.The proposed Notes2Skills condition instead loads a compiled Markdown skill with directive guidance, epistemic interpretation, defaults, and optional candidate actions requiring human review.
  • K.3 Conservative execution rules: The conservative execution rules require exactly one action and prohibit TRUNCATE_AT, RAISE_THRESHOLD, or SKIP_FILE from epistemic uncertainty alone; uncertainty without an operational directive requires FLAG_FOR_REVIEW.Explicit operational directives must provide concrete boundaries, thresholds, localized windows, or non-target-file designations.
Loading 2606.11897v1…