PaperlayerView exact revision on arXiv

Source-linked AI summary

VerifiedLab: Harnessing Knowns for Unknowns

Yu Yao, Jiayi Dong, Yang Yang, Shuiwang Ji, Ju Li

arXiv:2610.37923v1cs.AI

TL;DR

Research agents can extract scientific facts yet still fail to bind each value to the correct experiment, a gap aggregate accuracy can hide. VerifiedLab uses written rules and field-level verification to build records with judged reliability; its benchmark exposes binding failures and its deployed database reports accuracy by field category.

Abstract

from arXiv · show

Across human history, knowledge has undergone several transitions, from orality to writing and then to digital archives. At each transition, humans established new conventions to preserve knowledge for the new era. Nowadays, as research agents are increasingly deployed for laboratory experiments, knowledge is crossing another boundary: from records written for humans to those for machines. In this work, we introduce VerifiedLab, a multi-agent harness framework for this human-to-machine transition of knowledge. VerifiedLab transforms research output from passive artifacts into active systems that accelerate downstream use, adoption and prediction of future trends. In VerifiedLab, a team of AI scientists with distinct roles reasons over records verified field by field: a principal investigator plans, a librarian computes the evidence directly from the records, specialists draft, and a gate and a critic check each claim against the measured reliability of its data before the harness decides whether to release or withhold it. We demonstrate the efficacy of VerifiedLab through extensive experiments on perovskite solar cells: VerifiedLab recovers known physical trends, anticipates devices reported a year later, and lets AI scientists inherit the literature together with a measure of how far it can be trusted. By scaling the inference-time verification compute, VerifiedLab introduces a new pathway for knowledge dissemination and machine harness for AI co-scientists.

Verified Harness: Record, Policy and Check

VerifiedLab turns papers into field-level records whose evidence and reliability can be checked and reused. Its tests show that procedural information remains difficult to recover, written rules improve verification, and reliability-scoped records support trend recovery and prediction of later devices.

Verified Harness: Record, Policy and Check

  • The harness uses a schema for the champion device and versioned annotation and verification Skills to check each field against evidence under explicit rules.Nearly 100 fields cover seven groups; units stay attached to values, layer stacks retain order, and Null means the paper does not say.Source: Verified Harness: Record, Policy and… · p. 2

Procedural Information Hampers Recovery

  • Agents recover isolated facts more reliably than procedural information, especially values that must remain tied to a device, processing step, or condition.Localized scalars are recovered best, while relational fields are often wrong rather than blank; a stronger model narrows but does not close this gap.Source: Procedural Information Hampers Recovery · pp. 2–3

Verified Harness Brings AI Scientists to Expert Agreement

  • Written verification rules improve agreement by accepting equivalent values expressed differently, but cannot correct permissive acceptance of wrong values.The Skill helps most where errors are lexical rejections; ordered layer stacks remain hardest, and two policy defects caused regressions before correction.Source: Verified Harness Brings AI Scientists… · p. 4

Scaling Inference-time Compute Guarantees Reliability

  • Field-wide deployment assigns judged accuracy by category, with local fields easier to recover than synthesis, device-stack, and stability information.Stability fields are hardest; category reliability does not estimate confidence for individual values, and discovery analyses exclude stability.Source: Scaling Inference-time Compute… · p. 4
  • The database’s category-level reliability provides a use boundary, while its coverage ends with literature available through April 2025.The released category scores are not confidence estimates for individual values.Source: Scaling Inference-time Compute… · p. 4

From Knowns to Unknowns

  • Fine-tuned text encoders performed best on the temporally held-out efficiency task, and every trained model beat the training-mean predictor.Models trained on papers up to one year, tuned on the next, and scored on the year after; the full recipe schema improved tree performance over expert-chosen descriptors.Source: From Knowns to Unknowns · p. 5
  • Reliability-scoped records recover established trends and support efficiency prediction for devices reported a year after the training data.The paper’s champion-device records describe reported devices, not every device made; the trend analysis is aggregate and does not establish treatment causality.Source: From Knowns to Unknowns · pp. 4–5
  • The Verified Lab demonstration routes claims through a reliability gate and critic, but is an illustrative design sketch rather than a validated discovery.Its numbers are computed from released records, and the critic removes a claim resting on an unreliable category.Source: From Knowns to Unknowns · pp. 5–6

Discussion

The discussion presents field-level reliability as a way to expose where research agents fail and to carry measured trust alongside literature-derived data. Its trend and forecasting claims are bounded by the reported-device corpus and a single held-out year.

Discussion

  • The benchmark and deployment share an error profile: agents recover isolated facts but fail to bind values to the correct device, step, or condition.Plausible wrong values, rather than blanks, can be hidden by aggregate accuracy and pass into an AI scientist’s plans.Source: Discussion · p. 6
  • The authors argue that written, versioned scientific rules and expert calibration can improve verification without relying on a larger model.They note that rules repair over-strict verifiers, but not lenient ones.Source: Discussion · p. 6
  • Reliability measured per category and attached to records can guide later human or automated analyses about what to trust.The paper argues that measuring reliability once on a benchmark and then setting it aside does not protect later users.Source: Discussion · p. 6
  • Fig. 5 reports that reliable fields rediscover known perovskite trends and predict devices reported a year later.Its panels cover 10,258 champion devices and a forecast of 2025 devices from recipe fields alone.Source: Figure 5 · p. 7
  • The database is judged by language models rather than complete expert annotation, and its schema and category scores may not transfer beyond perovskite solar cells.The authors recommend documenting the corpus and intended use before generalizing.Source: Discussion · p. 7
  • Trend and forecast findings are associations tested on one held-out year, so they suggest hypotheses rather than establish causes.The paper also identifies synthesis and stability as its weakest categories, citing the challenge of following long procedures and reading figures and supplements.Source: Discussion · p. 7
  • The proposed division of labor keeps experts involved in schema and Skill design, with ordered sequences, multi-part concentrations, and stability protocols remaining with experts.The authors say this division is motivated but not yet tested end to end.Source: Discussion · pp. 7–8
  • The Verified Lab is an illustrative design sketch, not an experiment; its dialogue and agent meetings are illustrative.The figure uses released records and a forecaster for its numbers, and its Phase 5 recipes are held-out 2025 devices selected for illustration.Source: Figure 6 · p. 8

Methods

The harness converts a paper package into a fielded record for its best-performing device, using versioned annotation and verification policies. It applies a shared execution boundary to generators and verifiers, then evaluates field decisions against expert references.

Harness design

  • The harness takes a paper package and a 97-field schema and outputs one record for the paper’s best-performing device, with string values or explicit Nulls.Selecting one device gives a comparable unit across heterogeneous reports but introduces a selection effect accounted for in discovery analyses.Source: Harness design · p. 9
  • Versioned Skills separate scientific policy from execution code, specifying champion-device selection, evidence priority, layer alignment, measurement identity, Null use, and field-level verification rules.Schema, tolerance, or decision-rule changes require a new Skill version; model changes preserve the evaluation contract.Source: Harness design · p. 9
  • A shared execution boundary parses packages, validates records, pairs each candidate with its reference and field rule, invokes verification, aggregates metrics, and records provenance.Every generator and verifier traverses this same boundary.Source: Harness design · p. 9
  • Verifiers label each field as TN, FP, FN, TP, or FPmismatch, with false positives counting both FP and FPmismatch.An entirely empty Collection A record scores 36.4% accuracy under the reported metric.Source: Harness design · p. 9

Expert-curated testbed

The expert-curated reference comprises two paper collections annotated against field schemas, with disagreements resolved before automated verifier evaluation.

Expert-curated testbed

  • The reference contains 100 Collection A papers annotated across 97 fields and 50 Collection B papers with 73 overlapping fields, totaling 13,350 field decisions.Experts independently annotated article and supplement against the pinned Skill, then resolved disagreements through discussion and reinspection before verifier evaluation.Source: Expert-curated testbed · p. 9

Generator benchmark

The generator benchmark evaluates records with and without annotation Skills while holding the harness and verifier fixed. The calibrated o3 verifier with the verification Skill serves as the reference verifier for that benchmark and the open-weight study.

Generator benchmark

  • Sixteen proprietary and two open-weight generators each produced records with and without the annotation Skill, evaluated over 9,700 field decisions per generator.The harness, schema, source text, parser, and output validator were held fixed; the o3 verifier with the verification Skill supplied field decisions.Source: Generator benchmark · p. 9

Verifier calibration against experts

  • The o3 configuration with the verification Skill was audited field by field and used as the reference verifier for the generator benchmark and open-weight study.Three proprietary backbones were compared with expert decisions on both collections, with and without the verification Skill.Source: Verifier calibration against experts · p. 9

Open-weight verifier study

The open-weight verifier study evaluates ten checkpoints on field decisions from a fixed sample, comparing verification with and without the Skill.

Open-weight verifier study

  • Ten open-weight checkpoints were evaluated on the same deterministic paper-round-robin sample, with one generator record selected from each of 100 Collection A papers.This yielded 9,700 field decisions per condition.Source: Open-weight verifier study · p. 9
  • 0.9636 missingness-stratum accuracy was identical in both conditions, while overall accuracy covered all 9,700 decisions and semantic accuracy covered 5,198 non-empty comparisons.The verifier decided only the non-empty comparisons; missingness labels were assigned before inference.Source: Open-weight verifier study · p. 9
  • The Skill effect was measured as the paired accuracy difference, with record-level resampling used to estimate percentile intervals.The 100 records were resampled with replacement while retaining all 97 fields in each sampled record.Source: Open-weight verifier study · p. 9
  • The field-wise audit pooled each field across the ten checkpoints.Source: Open-weight verifier study · p. 10

Corpus and deployment

The corpus and deployment pipeline collects experimental perovskite solar-cell papers, verifies extracted records field by field, and releases provenance and reliability information.

Corpus and deployment

  • The corpus contains 10,258 peer-reviewed experimental perovskite solar-cell papers published from 2021 through April 2025, each represented by main text and supporting information.Source: Corpus and deployment · p. 10
  • A gpt-5-mini generator produced records under an annotation Skill, and two independent verifier agents judged each field under a verification Skill.The verifier agents were gpt-5-mini and o3; reported field-category metrics are averaged across the judges.Source: Corpus and deployment · p. 10
  • Released records include source-DOI provenance and category-level reliability scores.Source: Corpus and deployment · p. 10

Reliability-scoped discovery

The discovery analyses follow a reliability rule that excludes stability fields and uses different record categories for rediscovery and prediction.

Reliability-scoped discovery

  • Neither discovery analysis uses stability fields under the reliability rule.Source: Reliability-scoped discovery · p. 10
  • Rediscovery uses high-reliability cell-definition, device-stack, device-fabrication, and J–V-metric categories, while predictive models additionally use material-synthesis fields as inputs.Source: Reliability-scoped discovery · p. 10
  • Fig. 5 distributions include records with parsable values for each plotted field, and its Shockley–Queisser curve uses a 5,778 K blackbody Sun, 1,000 W m−2, and a 300 K cell.Source: Reliability-scoped discovery · p. 10

Qualitative demonstration

The paper presents the Verified Lab as an illustrative design sketch, while its plotted statistics and selected recipe examples have explicitly bounded scopes.

Qualitative demonstration

  • The Verified Lab is a design sketch rather than an experiment, and its agents, meetings, and summaries are illustrative.Source: Qualitative demonstration · p. 10
  • Fig. 6 statistics use parsed PCE, bandgap, and measured-area fields, with large-area devices defined as at least 1 cm2.Dominant transport layers are counted among large-area devices above 20% PCE, and surface treatment is counted when its material field is non-Null.Source: Qualitative demonstration · p. 10
  • The three displayed recipes come from 2025 test-set devices with specified non-empty fields and forecast error below 0.7 PCE points, so they illustrate output format rather than typical accuracy.Typical accuracy is reported in Fig. 5d–h.Source: Qualitative demonstration · p. 10

Temporal prediction benchmark

The benchmark predicts reverse-scan PCE from records split by publication year, using recipe fields while excluding target-proxy information. It compares tabular, text, embedding, and prompted models, with a controlled-subset analysis and multiple evaluation metrics.

Data and split

  • The 8,450 records with publication year and 0 < PCE ≤ 30% were split into 5,422 training records through 2023, 2,170 validation records from 2024, and 858 test records from 2025.Tandem devices and garbled values were removed; the test set was scored once.Source: Temporal prediction benchmark · p. 10

Inputs

  • Models used 71 recipe fields, excluding bibliographic, provenance, J–V, and stability fields to limit target leakage.Forward-scan, stabilized, and certified PCE nearly copy the target, and 610 titles contain a percentage, usually efficiency.Source: Temporal prediction benchmark · p. 10

Tabular features

  • The feature set combined 18 expert descriptors with eight composition-site flags, producing 77 features after recipe-field filtering and encoding.Fields with fewer than 10 training values were dropped; numeric fields required at least 60% of values to contain exactly one number.Source: Temporal prediction benchmark · p. 10

Models

  • The benchmark compared LightGBM, TabPFN, fine-tuned MatSciBERT and T5-base, frozen Llama embeddings with ridge regression, and prompted Llama.Text models used deterministic recipe-field serialization; the prompted model used up to 48 tokens, with an optional five-shot TF-IDF prompt.Source: Temporal prediction benchmark · p. 10

Metrics and controlled-subset analysis

  • Evaluation reported MAE, MAD:MAE, R^2, and ROC AUC for PCE > 20%, with one-sided paired bootstrap tests of test MAE.A complementary analysis modeled 2,445 SnO2 ETL and Spiro-OMeTAD HTL devices using 18 expert descriptors and five-fold cross-validation; feature-importance rankings are hypotheses, not mechanisms.Source: Temporal prediction benchmark · p. 10

Disclosure of Generative AI Usage

LLMs assisted with writing clarity and grammar, experimental, figure, and analysis code, and parts of the appendix; the authors reviewed and validated final scientific content and conclusions.

Disclosure of Generative AI Usage

  • LLMs assisted with writing clarity and grammar, experimental, figure, and analysis code, and drafting parts of the appendix; the authors reviewed and validated all final scientific content and conclusions.The authors also state that final code, results, and figures’ scientific content were produced by them.Source: Disclosure of Generative AI Usage · p. 11

Ethics statement

The database uses peer-reviewed publications and contains no human-subject or personal data.

Ethics statement

  • The database is built from peer-reviewed publications and does not involve human subjects or personal data.Source: Ethics statement · p. 11

Related papers

Loading 2610.37923v1…