PaperlayerView exact revision on arXiv

Source-linked AI summary

DISCERN: Can AI Agents Work Like Scientists and Guide Discovery?

Nan Huang, Mario Tapia-Pacheco, Kun Zhou, Yiming Huang, Kevin José Barrientos Díaz, Tiffany Amariuta, Jingbo Shang

arXiv:2609.33357v2cs.AI

TL;DR

Existing benchmarks often assess analytical task completion or hypothesis generation separately, rather than whether trustworthy evidence supports valid and novel claims. DISCERN evaluates three levels of research work across real datasets and finds uneven capabilities: no model dominates all levels, and current agents do not yet demonstrate reliable autonomous analysis or discovery.

  • Problem

    Existing benchmarks often evaluate analytical task completion or hypothesis generation separately, leaving evidence support for valid and novel claims insufficiently assessed.

    Source: §1 · p. 2

  • Method

    DISCERN is a controlled benchmark of data integrity, analysis verification, and hypothesis generation using real public datasets across 203 tasks, eight life-science tracks, and five scientific areas.

    Source: §1 · p. 2

  • Results

    No model dominates all three levels, similar overall performance can reflect different capability profiles, and 46.8% of submitted hypotheses receive no novelty credit.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Current agents can support bounded research tasks, but reliable scientific judgment across preprocessing, analysis verification, and hypothesis generation remains uneven.

    Source: §6 · pp. 9–10

    Bounded tasks and independently instantiated levels do not establish autonomous end-to-end research, and generalization is constrained by limited domain coverage.

    Source: §6 · p. 9

Abstract

from arXiv · show

Reliable automated research requires agents to vet data, verify analyses, and generate hypotheses grounded in trustworthy evidence, potentially reducing routine scientific workload while allowing scientists to focus on interpretation and discovery. Existing benchmarks often only assess analytical task completion or hypothesis generation separately rather than testing whether reliable evidence supports valid and novel claims. We introduce DISCERN (Data Integrity and Scientific Capability: Evidence, Reasoning, and Novelty), a controlled benchmark on real, publicly available datasets that evaluates three key levels of an automated research workflow. The first two levels test data integrity and analysis verification under confounds and tool traps, while the third tests hypothesis generation and revision under adversarial review, including counterfactual cases in which evidence consistent with real data and documented scientific phenomena conflicts with established expectations, motivating alternative explanations and testable hypotheses. Across 203 tasks, eight life-science tracks, and eight models, DISCERN shows that strong aggregate performance can mask level-specific weaknesses. Agents earn perfect scores in only 60.8% of Level 1, 34.2% of Level 2, and 0.6% of Level 3 evaluations, with penalties attributed to rejection of sound data, failure to carry recognized limitations into conclusions, and wide variation in hypothesis production. Cross-track rankings by token and code use are substantially more stable than rankings by evidence judgment, suggesting greater consistency in computational effort than in evidence-based reasoning. These profiles identify opportunities for supervised scientific assistance, but current agents do not yet demonstrate reliable autonomous analysis or discovery. Code and data: https://huggingface.co/datasets/discern-bench-anon/discern-benchmark

1 INTRODUCTION

The introduction argues that reliable automated research must be evaluated across data integrity, analysis, and scientific reasoning because errors can distort conclusions across stages. DISCERN tests these levels and finds uneven model capabilities, including difficulty carrying recognized limitations into conclusions and producing novel hypotheses.

Evaluation gap

Motivation

  • Errors can propagate across stages: contaminated RNA may be mistaken for a marker and then overinterpreted as a functional driver, potentially misdirecting experiments.The example maps to preprocessing, analysis, and hypothesis generation, respectively.Source: §1 · p. 1, Figure 1 · p. 2

Contribution

  • DISCERN evaluates data integrity, analysis verification, and hypothesis generation in 203 tasks across eight life-science tracks and five scientific areas.Its independently instantiated and scored levels localize weaknesses without treating the tasks as an end-to-end pipeline.Source: §1 · p. 2

Results

  • No model dominates all three levels; similar overall performance can reflect different capability profiles, and 46.8% of submitted hypotheses receive no novelty credit.Models also vary in whether they propose hypotheses and in how they respond to criticism and use computational resources.Source: §1 · p. 2

2 BENCHMARK DESIGN AND SCIENTIFIC VALIDITY

DISCERN spans diverse life-science data and builds distinct challenges for data preprocessing, analysis verification, and hypothesis generation. Counterfactual dossiers are tied to documented phenomena and an agent’s elicited expectations.

Scientific coverage

  • Eight tracks span genetic variation, gene expression, perturbation screens, regulatory measurements, and protein data, with tasks from DNA-level variation through proteins.The tracks cover genome-wide association studies, clinical variant interpretation, bulk and single-cell RNA sequencing, CRISPR screens, enhancer activity, chromatin regulation, and protein stability and structure.Source: §2 · pp. 2–3

L1 and L2 Level-specific challenges

  • L1 pairs real datasets and metadata with controlled integrity defects or matched clean controls; challenges include genome-build mismatches and ambient RNA contamination.Challenges reflect track-specific representations, formats, metadata conventions, and quality-control requirements.Source: §2 · p. 3
  • L2 tests analyses under clean or perturbed conditions, including confounds, tool failures, and statistical pitfalls; a narrow-peak setting fragments broad H3K27me3 domains despite successful execution.The resulting report can look complete while underestimating the total marked region.Source: §2 · p. 3

L3 evidence and counterfactual construction

  • L3 dossiers use reference analyses without passing the evaluated agent’s outputs between levels; counterfactual eligibility requires conflict with the agent’s elicited prior.Injected cases alter a characterized relationship and regenerate consistent supporting statistics, while grounding the plausible behavior in documented phenomena.Source: §2 · p. 3

Task environment and recorded observations

  • Tasks use isolated environments with the same prompt and reporting schema within each analysis family, and a sealed Python sandbox records code, tool calls, reports, and token use.The sandbox has no network access.Source: §2 · p. 3

3 EVALUATION AND PROFILE CONSTRUCTION

DISCERN grades data integrity, analysis verification, and hypothesis quality using shared criteria and human review, then aggregates scores to balance levels and tracks. It also records resource use across the task suite.

Evaluation protocol and score aggregation

  • Track composites use the geometric mean of L1–L3 scores, and the Overall Score averages the eight track composites with equal track weight.This aggregation means strong performance at one level cannot fully compensate for weakness at another.Source: §3 · p. 3

Data-integrity scoring

  • L1 credit progresses from detecting a defect to diagnosing and appropriately handling it; clean controls require accepting sound data and penalize unsupported rejection.Recognizing insufficient evidence as a limitation earns credit rather than an unsupported diagnosis.Source: §3 · p. 4

Analysis-verification scoring

  • L2 averages four dimensions: analytical correctness, issue recognition, methodological response, and calibration of conclusions to analysis reliability and limits.Rubric criteria are tailored to each analysis’s scientific objective and methodological requirements.Source: §3 · p. 4

Hypothesis scoring and adversarial review

  • L3 scores evidence fidelity, scientific reasoning, and novelty; novelty rewards substantive hypotheses with testable experiments that distinguish alternatives.Specific mechanisms and experiments distinguishing them from analytical artifacts earn more novelty credit than merely restating an unexpected association.Source: §3 · pp. 4–5
  • Each L3 response receives three rounds of adversarial review, with critiques and rebuttals informing an LLM judge; justified non-proposals receive partial credit without novelty scoring.The rubric judge, not reviewer votes, determines the score, and authors review and validate each response.Source: §3 · p. 5

Resource efficiency

  • Resource measures cover tokens, API calls, and estimated cost across 203 tasks, excluding grading, verification, and reviewer activity.API-call counts approximate interaction frequency rather than complete analysis–feedback cycles.Source: §3 · p. 5

4 RESULTS AND ANALYSIS

Overall performance varies across models, but no model dominates every capability; performance also shifts across tracks and task types. Their scientific working styles differ in evidence assessment, hypothesis behavior, critique response, and computational effort, with resource-use rankings more stable across tracks than scientific-judgment rankings.

4.1 OVERALL MODEL PERFORMANCE

  • 78.1 is GPT-6 Astra’s highest aggregate DISCERN score, while perfect-score rates are 60.8% for L1, 34.2% for L2, and 0.6% for L3.Astra leads L1 and L3, but Opus leads L2; L3 is the lowest-scoring component in 46.9% of model–track combinations.Source: §4.1 · pp. 5–6

4.2 SCIENTIFIC STRENGTHS VARY ACROSS TASKS

  • GLM-5.3 scores 100 on Protein L1 but 3.5 on CRISPR L2, while analysis-family means also vary widely within domains.In statistical genetics, means range from 77.4 to 32.8; in regulatory genomics, from 76.9 to 20.4.Source: §4.2 · pp. 6–7, Figure 3 · p. 6
  • 29.6% of 425 challenged L2 evaluations with full issue-recognition credit receive only half credit for result calibration.The proportion ranges from 19.4% for Opus to 37.8% for DeepSeek.Source: §4.2 · p. 7

4.3 DISTINCT SCIENTIFIC STYLES EMERGE ACROSS MODELS

  • Only 19 of 64 L1 clean controls are correctly accepted, and responses to evidence conflicting with stated priors range from Kimi hypothesizing in 92.3% of eligible cases to Astra abstaining in all such cases.The benchmark characterizes these observed differences as scientific working styles, not fixed model traits.Source: §4.3 · p. 7, Figure 4 · p. 7
  • 46.8% of 94 submitted hypotheses receive no novelty credit, while Sol and Opus have similar conditional novelty means but submit hypotheses on 18.2% and 81.0% of eligible tasks.Their mean conditional novelty scores are 41.7 for Sol and 41.2 for Opus.Source: Figure 4 · p. 7, §4.3 · p. 8
  • Cross-track rankings by token use and code execution are more consistent (W = 0.89–0.95) than rankings by evidence judgment and hypothesis behavior (W = 0.09–0.57).Greater computational expenditure does not by itself establish better scientific performance.Source: §4.3 · p. 8, Figure 5 · p. 8
  • DeepSeek concedes at least one reviewer point in 41.7% of committed outcomes versus 5.9% for Opus, but concession frequency alone does not establish revision quality.In the DUSP4 case, DeepSeek narrows its claim and proposes experiments to test signal overlap and the effect of reducing repression, receiving a final task score of 95.7.Source: §4.3 · p. 8
  • Fable answers 59.7% of L1/L2 tasks and 9.1% of L3 tasks, with refusals varying across tracks and analysis families for reasons the study does not identify.Its coverage is incomplete, including refusals on routine public-dataset analyses and four tasks all eight comparison models solve perfectly.Source: §4.3 · p. 8

5 RELATED WORK

Prior benchmarks increasingly test executable scientific analysis and multistage inference, while discovery evaluations examine complementary aspects of hypothesis generation. DISCERN distinguishes itself by testing evidence-grounded judgment under controlled problems and adversarial review.

Scientific data analysis benchmarks

  • Scientific-agent benchmarks have progressed from answer generation toward executable analysis, multistage inference, and study-scale workflows measuring research outputs and resource use.DISCERN complements this work by testing judgment under controlled data defects, confounds, and tool traps, including whether agents detect problems and reflect limitations in conclusions.Source: §5 · p. 9

Hypothesis generation and discovery benchmarks

  • Discovery evaluations cover reference-relation recovery, interactive experiments, altered scientific laws, and research assessed through automated review.DISCERN tests whether hypotheses are grounded in supplied evidence, including evidence challenging expectations, and revised under adversarial review.Source: §5 · p. 9
  • DISCERN separately measures hypothesis commitment, evidence fidelity, scientific reasoning, and novelty, distinguishing willingness to propose from proposal quality.Source: §5 · p. 9

6 DISCUSSION, LIMITATIONS, AND CONCLUSION

Scientific-agent capability is compositional and context-dependent: useful components and bounded-task support coexist with uneven judgment across preprocessing, analysis verification, and hypothesis generation. The paper calls for task- and decision-level evaluation while noting that its bounded, independently instantiated tasks do not establish autonomous end-to-end research.

6 DISCUSSION, LIMITATIONS, AND CONCLUSION

  • Strong performance in one workflow component, track, or decision can coexist with weakness elsewhere, so overall scores do not establish uniform scientific reliability.The conclusion describes current agents as supporting bounded research tasks while reliable judgment across preprocessing, analysis verification, and hypothesis generation remains uneven.Source: §6 · pp. 9–10
  • Bounded tasks and independently instantiated levels do not establish autonomous end-to-end research or whether hypotheses support useful follow-up or human–agent integration.Generalization and model comparisons are also constrained by limited domain coverage, possible familiarity with public datasets or published phenomena, task dependence, and residual grading uncertainty.Source: §6 · p. 9

AI USE STATEMENT

  • The authors used AI for L3 scoring and verification, as well as code, analysis implementation, figures, and prose, while retaining responsibility for checking AI-assisted content.Human review and adjudication are also part of the described L3 scoring process.Source: AI USE STATEMENT · p. 11

ETHICS STATEMENT

  • The study uses public data and controlled perturbations that are not scientifically or clinically meaningful; generated hypotheses are neither experimentally confirmed nor established as novel.The authors state that the work should not guide clinical decision making.Source: ETHICS STATEMENT · p. 11

REPRODUCIBILITY STATEMENT

  • The anonymous benchmark release includes prepared task inputs, grading materials, recorded outputs, adjudications, and scripts to regenerate selected tables and figures.Its guide describes running additional models on prepared tasks, including grading and human adjudication.Source: REPRODUCIBILITY STATEMENT · p. 11

A SCIENTIFIC WORKING-STYLE PROFILES AND ROBUSTNESS

The appendix presents scientific working-style profiles and robustness checks, emphasizing that model behavior varies across measures and that aggregation choices expose different aspects of stage imbalance. Its summaries distinguish diagnostic populations and conditional quality from overall performance, and propose review requirements without claiming human replacement.

A.1 FROM A SCIENTIFIC EXPECTATION TO A MEASURED BEHAVIOR

  • Table 2 frames measured behavior as a cue for where oversight may be needed, not evidence of human replacement or improved next experiments.The table maps scientific expectations to benchmark proxies and proposed human checks; its final column is an implication for supervised use, not a measured collaboration outcome.Source: §A.1 · p. 16, Table 2 · p. 16

A.2 DETAILED DIMENSIONAL PROFILES

  • Figure 6 retains individual grading dimensions and diagnostic populations rather than creating five newly averaged scores.Its metrics cover evidence judgment and handling, analytical behavior, and response behavior; all values are 0–100, but rubric scores and rates represent different quantities.Source: §A.2 · p. 16, Figure 6 · p. 17

A.3 POPULATIONS, AGGREGATION, AND STABILITY

  • The profile summarizes L1 from recorded state-level outcomes and L2 using the world-to-family-to-track hierarchy that produces the canonical leaderboard.L3 subaxes cover committed eligible hypotheses, with different denominators by model; conditional novelty therefore is not a full-benchmark ranking, and specified absent or excluded outputs are omitted.Source: §A.3 · pp. 17–18
  • Token and code-cell rankings are more stable across tracks (W = 0.89–0.95; mean split-half correlations 0.96–0.99) than evidence-related rankings, including L3 evidence fidelity (W = 0.27).L1 clean specificity is 0.15; the near-ceiling no-fabrication measure has W = 0.089, constrained by few nonzero events and many ties.Source: §A.3 · p. 18

A.4 AGGREGATION SENSITIVITY AND DIFFICULTY

  • The primary score uses a geometric mean across three stages within each track, then an arithmetic mean across tracks; the mean within-track minimum ranges from 30.6 to 70.1.Geometric Overall ranges from 39.8 to 78.1; neither alternative summary is a newly selected official score.Source: §A.4 · p. 18

A. Overall masks lower-scoring stages

  • Figure 7 compares Overall with the mean within-track minimum stage score, while Figure 8 shows track coverage across score requirements; neither establishes end-to-end success.No threshold is selected as a scientific pass criterion, and separately tested stages mean threshold coverage is not a joint pipeline success probability.Source: Figure 7 · p. 18, Figure 8 · p. 19, A. Overall masks lower-scoring stages · p. 19

A.5 FINE-GRAINED PROFILES

  • The fine-grained profiles retain strengths, weaknesses, commitment denominators, and expenditure, but describe one recorded protocol rather than general model personalities.The profiles do not infer human replacement; defensible supervised-use proposals still require scientific and practical review.Source: §A.5 · p. 19, Figure 9 · p. 19

B SELECTIVE REFUSAL AND THE DERIVED FALLBACK REPLAY

Fable answers 59.7% of assigned L1/L2 tasks and has scored results for only two L3 worlds, so its conditional leaderboard reports coverage but no Overall. Refusal diagnostics do not establish a simple relationship with task difficulty, and the fallback replay is a protocol-derived comparison rather than an observed routed run.

Selective refusal

  • 59.7% of assigned L1/L2 tasks (108/181) are answered by Fable, with scored results for only two of 22 L3 worlds and no conditional Overall.The leaderboard row includes coverage; its L3 results are too limited for an Overall score.Source: B SELECTIVE REFUSAL AND THE DERIVED… · p. 20
  • The empirical difficulty proxy uses the recorded scores of all eight comparison models and excludes Fable’s own outcome.It uses recorded L1 state grades and finalized L2 meta-grades; the authors characterize it as a comparison-model proxy, not intrinsic scientific difficulty.Source: B SELECTIVE REFUSAL AND THE DERIVED… · p. 20
  • Refusal rates across ordered difficulty quintiles are 48.6%, 27.8%, 33.3%, 41.7%, and 50.0%, while the refused-versus-answered means are 0.309 and 0.262 (nominal Mann–Whitney p = 0.473).Related worlds within families are not independent studies, and the test is described as a diagnostic; non-significance does not establish missingness at random or make conditional scores comparable on the full benchmark.Source: B SELECTIVE REFUSAL AND THE DERIVED… · pp. 20–21
  • Fable’s conditional scores cover answered tasks, while deployment-with-refusal-zero is treated as a different estimand.Missing families and tracks are not entered as zero-valued performance observations in the conditional row.Source: B SELECTIVE REFUSAL AND THE DERIVED… · p. 21

Derived fallback replay

  • The Fable-to-Opus replay assigns Opus outputs to tasks not invoked under the refusal-stop protocol; these are protocol-imputed fallback decisions, not observed routed-system runs.Opus is replayed from the beginning of a task, not continued from a Fable state; original Opus scores remain unchanged.Source: B SELECTIVE REFUSAL AND THE DERIVED… · p. 21

C BENCHMARK COMPARISON: SOURCES AND QUALIFICATIONS

This appendix situates DISCERN’s stage-specific evaluations alongside related benchmarks and documents their task inventory and diagnostics, including the L1 integrity registry’s clean controls, boundary cases, and controlled data-integrity challenges.

C BENCHMARK COMPARISON: SOURCES AND QUALIFICATIONS

  • The comparison spans six dimensions but describes inspected benchmark versions, not universal feature absence or evidence of superiority.Benchmarks differ in what graders assess: for example, GeneBench-Pro collects free-text reasoning without grading it, while BiomniBench grades trajectories and accepts valid alternatives. DISCERN’s separate stages do not establish a continuous pipeline, and its novelty credit remains rubric-bounded.Source: C BENCHMARK COMPARISON: SOURCES AND… · pp. 21–22, Table 3 · p. 22

D ADDITIONAL STAGE DIAGNOSTICS

  • 192 scores map the stage-level landscape across eight models, eight tracks, and three stages, while calibration and grading diagnostics qualify stage-specific scores.The ceiling strip does not represent end-to-end success; L3 quality is conditional on commitment, and the three transcript audits claim neither literature-level novelty nor experimental confirmation.Source: Figure 11 · p. 23, D ADDITIONAL STAGE DIAGNOSTICS · p. 23, Figure 12 · p. 24

E FULL TASK AND WORLD INVENTORY

  • Tables 4–6 inventory the eight-track benchmark, its L1 tasks, L2 analysis families, L3 dossiers, and the work grounding each L3 design.Grounding distinguishes real-data substrates, documented native phenomena, and scientific support for injected counterfactuals; the physics extension is separate from the 203-task denominator and molecular-life-science core score.Source: E FULL TASK AND WORLD INVENTORY · p. 26, Table 4 · p. 27, Table 5 · p. 28

E.1 COMPLETE L1 INTEGRITY REGISTRY

  • 79 L1 worlds include 71 non-clean worlds spanning 65 defect classes, with only three classes recurring across tracks.Manifest-defined expected responses distinguish appropriate acceptance from reflexive rejection.Source: §E.1 · p. 29, Table 7 · p. 29
  • Histone tasks test recognition of batch-scale inconsistency, blacklist artifacts, metadata mismatches, and missing input controls, while requiring abstention when antibody efficiency cannot be assessed.The registry calls for per-track normalization, exclusion of blacklist regions, metadata reconciliation, and obtaining the absent input before validating enrichment.Source: §E.1 · p. 32
  • Protein tasks distinguish site-specific defects requiring repair from normal structure conditions, such as absent hydrogens or flexible loops, that should not trigger rejection.Examples include rebuilding a missing mutation-site side chain or resolving a local steric clash, while retaining a structure with flexible loops and adding hydrogens during routine preparation.Source: §E.1 · pp. 32–33
  • Single-cell RNA-seq tasks cover barcode and count QC, ambient-RNA and doublet handling, feature-type separation, contamination filtering, and metadata reconciliation.The registry also cautions against discarding real low-RNA populations such as platelets and neutrophils.Source: §E.1 · p. 33
  • 79 L1 worlds across eight tracks comprise 8 clean controls, 7 no-defect boundary cases, and 64 challenges spanning 58 distinct types.The registry’s method descriptions are abbreviated to their first sentences; accompanying CSV exports contain the full strings.Source: §E.1 · p. 33

E.2 COMPLETE L2 ANALYSIS REGISTRY

The registries document Level 2 checks across 102 tasks and 24 analysis families, pairing shared methods with confounds, tool or statistical pitfalls, clean controls, and expected responses. They also catalog Level 3 cases that test evidence against prior expectations, distinguish native from injected counterfactuals and their citation burdens, and connect design constraints to implementations and targeted failures.

E.2 COMPLETE L2 ANALYSIS REGISTRY

  • The registry covers 102 tasks in 24 analysis families, with shared family methods and world-specific traps or controls documented in Table 8.Missing manifest fields are left explicitly missing rather than reconstructed.Source: Table 8 · p. 34, §E.2 · p. 34
  • CRISPR tasks require reference-normalized guide-level scoring and robust aggregation, while ClinVar tasks require checking annotations against mounted references before classifying variants.The registry warns that naive gene means can dilute real essentials with mostly dead guides and that misleading AF, consequence, ClinVar, or in-silico annotations should not override reference evidence.Source: §E.2 · p. 35
  • ClinVar prioritization and secondary-finding tasks require ranking causal variants while demoting distractors and reporting only pathogenic or likely pathogenic variants within ACMG-SF genes.Distractors include variants that are too common, low-quality artifacts, inheritance or phenotype mismatches, VUS, and variants outside the reportable gene list.Source: §E.2 · pp. 35–36
  • Enhancer tasks distinguish sequence conservation from activity and require family-level ranking, GC adjustment, alignment-quality checks, and safeguards against duplicated or phylogenetically correlated observations.Divergence tasks also call for plausible values, species normalization, deduplication, and consistency across a clade rather than a single sister-species jump.Source: §E.2 · p. 36
  • GWAS tasks cover locus definition and signal diagnostics, including threshold and distance rules, inflation and p-value checks, and scrutiny of suspicious or biologically implausible signals.The registry also flags traps involving underflow, scattered fake loci, artifacts, batch concentration, null signals, and wrong loci.Source: §E.2 · pp. 36–37
  • TWAS tasks require harmonizing GWAS variants to model identifiers and allele orientation, while histone tasks emphasize mark-appropriate quantification, input normalization, and verification against expression.Histone ranking guidance calls for jointly estimating independent feature effects and checking collinearity and coefficient stability.Source: §E.2 · p. 37
  • L2 regression tasks require independent or adjusted effects, checks for heterogeneity and instability, and robust estimates when outliers or nonlinear relationships make Pearson misleading.For promoter marks, the registry calls for Pearson and rank-based estimates, GC/CpG adjustment, and attention to saturation and leverage; for predictor models, it flags shared variance and collinearity.Source: §E.2 · pp. 37–38
  • L2 protein tasks require treating sequence-LM scores as uncalibrated ranks, verifying their sign, and recomputing omitted or mis-joined burial from structure.Burial is identified as the dominant stability determinant, with its association varying between helix and loop; LM scores should not be presented as kcal/mol without calibration.Source: §E.2 · p. 38
  • For mutation classification, L2 tasks require calibrating the LM’s destabilizing bias and flagging near-neutral mutations instead of forcing confident calls.The raw zero-threshold sign overcalls destabilization.Source: §E.2 · p. 38
  • L2 single-cell tasks distinguish real cell types and biology from ambient markers, batch artifacts, doublets, and pseudoreplication.The registry says to preserve the real B-cell depletion despite batch effects, exclude doublets, and aggregate by donor for condition-level differential expression.Source: §E.2 · pp. 38–39

E.3 COMPLETE L3 DISCOVERY REGISTRY

  • Table 9 catalogs 22 L3 dossiers by evidence, prior, expected response, grounding, and scored-model count; native and injected counterfactuals differ in citation burden.Its scored-model counts preserve the original six-model registry, rather than the expanded eight-model results.Source: §E.3 · p. 39, Table 9 · p. 39
  • L3 counterfactuals require following measured evidence over familiar expectations, including PPAN dispensability, TMPRSS15 essentiality, and APOB or TP53 diagnoses over famous-gene anchors.The PPAN case calls for a paralog-buffering mechanism; the injected TMPRSS15 case tests holding measured essentiality despite its non-essential-control prior.Source: §E.3 · pp. 40–41
  • Rank 1/141 (within-family p = 0.010) was a calibrated enhancer candidate, but it did not clear the genome-wide max null.The expected response was to state the candidate and its limits rather than claim a discovery.Source: §E.3 · p. 41
  • Other L3 cases test whether agents verify and retain evidence that contradicts priors, including a GSTM1 direction reversal and silent promoters with both activating and repressive marks.For the chromatin cases, the expected response is to verify the H3K27me3 signal before revising the expected expression direction.Source: §E.3 · pp. 41–42
  • In scRNA counterfactuals, non-classical monocytes had high CD16 (˜2.4549999237060547) and higher CD68 than classical-reference cells, challenging textbook marker priors.The registry notes that canonical markers are context-dependent, not absolute.Source: §E.3 · p. 43
  • 22 L3 dossiers span 8 tracks: 8 real controls, 6 native counterfactuals, and 8 injected cases.Native cases require phenomenon citations for grounding; injected cases require coherence arguments.Source: §E.3 · p. 43
  • Construct-level citations make interventions scientifically plausible but do not establish counterfactual entity-level results as true.For example, the enhancer substrate assayed 348 active tiles from 233 putative enhancers across 11 primates; DISCERN’s different counts refer to its processed subset.Source: §E.3 · pp. 43–44

E.4 DESIGN CONSTRAINTS

  • Table 10 links each benchmark principle to its implementation and the failure prevented, including indiscriminate suspicion and mistaking successful execution for valid analysis.Controls address indiscriminate suspicion, while tool- and procedure-output traps address invalid analysis despite successful execution.Source: §E.4 · p. 44, Table 10 · p. 44

F INITIAL CROSS-DOMAIN PHYSICS EXTENSION

The Physics extension applies DISCERN’s three-level evaluation to calorimetry and nuclear decay as a transfer probe beyond molecular life science, reported separately from the 203-task core. Across the two Physics tracks, stage leaders differ, and aggregate analysis-verification performance is lower than in the life-science tracks.

Comparison with the life-science core

  • The extension applies DISCERN’s L1–L3 contract to calorimetry and nuclear decay, outside the eight-track, 203-task life-science core.These two tracks are reported separately and do not enter the core denominator or score.Source: F INITIAL CROSS-DOMAIN PHYSICS EXTENSION · p. 45

Calorimetry

  • Calorimetry L1 tests integrity defects, L2 tests Cu–heptane equilibrium analyses, and L3 asks whether bulk-Cu heat-capacity priors remain adequate for nanocrystalline specimens.The two L3 counterfactuals increase specimen-level heat capacity and regenerate downstream equilibrium outcomes.Source: F INITIAL CROSS-DOMAIN PHYSICS EXTENSION · p. 45

Nuclear decay

  • Nuclear-decay L1 tests spectrum and acquisition integrity, L2 tests recovery of Co-60 decay rates amid confounds, and L3 uses the frozen 274-day dataset for real and counterfactual worlds.The counterfactuals assume fully ionized Co-60 under stellar conditions and regenerate count trajectories while keeping the observation schedule and control measurements fixed.Source: F INITIAL CROSS-DOMAIN PHYSICS EXTENSION · p. 45

Frozen task inventory

  • Each Physics track includes a clean L1 task, six integrity challenges, a no-defect boundary task, one clean L2 analysis, three challenged analyses, and one real plus two counterfactual L3 worlds.All eight evaluated models passed the prior checks required for inclusion of Physics L3 counterfactuals.Source: F INITIAL CROSS-DOMAIN PHYSICS EXTENSION · pp. 45–46

Physics-extension results

Comparison with the life-science core

  • 71.1 at L1, 55.7 at L2, and 65.0 at L3 were mean Physics scores, versus 74.1, 68.4, and 64.8 across the life-science tracks, respectively.The authors describe aggregate L1 and L3 performance as similar across settings, but Physics L2 performance as lower.Source: F INITIAL CROSS-DOMAIN PHYSICS EXTENSION · p. 46

G COMPLETE L1 AND L2 RESULTS

L1 challenges reveal substantial variation in diagnosis across scientific settings, while L2 results show that recognizing an issue does not guarantee it shapes analysis and conclusions.

G COMPLETE L1 AND L2 RESULTS

  • 56.2% of challenged L1 evaluations achieve full diagnosis and appropriate action; 21.7% identify the wrong cause, and 7.9% diagnose correctly without acting.Figure 15 records behavioral states across all 568 challenged evaluations.Source: G COMPLETE L1 AND L2 RESULTS · p. 46, Figure 15 · p. 47
  • 48.4–51.6% of detectable-defect evaluations in CRISPR, single-cell RNA sequencing, and epigenomic tracks involve wrong diagnoses, versus 4.7–12.5% in the other five tracks.All ten lowest-scoring defect tasks occur in those three settings; Figure 17 covers task-category difficulty across L1 and L2.Source: G COMPLETE L1 AND L2 RESULTS · p. 46, Figure 17 · p. 49
  • 53.6 is the average score on eight challenges requiring agents to withhold a defect claim or recognize insufficient evidence; 37.5% of evaluations overdiagnose a problem.Source: G COMPLETE L1 AND L2 RESULTS · p. 47
  • 29.6% of 425 challenged L2 evaluations with full issue-recognition credit receive exactly half result-calibration credit, which does not uniquely indicate overconfidence.Result calibration averages 64.5 and is the lowest-scoring shared dimension for seven of eight models; Figure 16 presents model scores across all 24 analysis families.Source: G COMPLETE L1 AND L2 RESULTS · p. 47, Figure 16 · p. 48

H.1 COMMITMENT, CONDITIONAL NOVELTY, AND YIELD

L3 outcomes vary in whether agents produce hypotheses and in the novelty they earn; the section also details review responses and high-scoring cases. Novelty yield combines production frequency with conditional novelty, but is a rubric score, not validated usefulness or a discovery rate.

H.1 COMMITMENT, CONDITIONAL NOVELTY, AND YIELD

  • Astra’s two prior-eligibility exclusions leave 20 eligible outcomes, of which one contains a hypothesis and 19 are abstentions.These exclusions do not independently establish a general tendency to avoid stating beliefs.Source: §H.1 · p. 49
  • 5.0% of Astra’s eligible L3 tasks produce a hypothesis, compared with 81.0% for Opus; mean novelty across 94 submitted hypotheses is 30.1.Only six of 172 eligible outcomes receive maximum novelty credit, four in the DUSP4 task.Source: §H.1 · p. 50
  • Sol and Opus have similar conditional novelty scores (41.7 and 41.2) but novelty yields of 7.6 and 33.3, respectively, because their commitment rates differ substantially.Yield combines the frequency of hypothesis production with conditional novelty; it does not measure experimentally validated usefulness or discovery rate.Source: §H.1 · p. 50
  • Agents concede at least one reviewer point in 20.2% of committed outcomes, but a recorded concession does not necessarily mean the final hypothesis is better supported.84.2% of first concessions occur in the first review round; model-level concession rates range from 0% for Sol and Astra to 41.7% for DeepSeek.Source: §H.1 · p. 50

H.2 HIGHEST-SCORING MODEL–WORLD OUTCOMES

  • Two high-scoring non-protein outcomes from the native DUSP4 world propose distinct hypotheses and discriminating experiments after bounding their claims.DeepSeek proposes locus-specific H3K27me3 measurement and an EZH2/PRC2 perturbation time course; Kimi withdraws a uniqueness claim and proposes sequential ChIP.Source: §H.2 · pp. 50–51
  • 29 of 94 committed outcomes receive at least two-thirds of maximum novelty credit, including 25 of 63 counterfactual responses and 4 of 31 real-world responses.These task-weighted descriptive comparisons are not causal effects of world design; the threshold does not establish independently validated experiments.Source: §H.2 · p. 51
  • A median hypothesis-drift score of 0.94 measures textual difference, not semantic narrowing; final-round reviewer judgments cannot establish what caused earlier concessions.The log records the first round with a concession flag, not necessarily abandonment of the hypothesis.Source: §H.2 · p. 51

H.3 ABSTENTION IS NOT ONE BEHAVIOUR

  • Figure 18 separates recorded final L3 states within the commit/abstain split, clarifying that abstention is not one uniform behavior.Its bars include three eligibility exclusions; Opus’s absent pre-execution exclusion is omitted, leaving 21 graded outputs for that model.Source: §H.3 · p. 52, Figure 18 · p. 52

I PROMPTS, TOOLS, AND EXECUTION ENVIRONMENTS

The appendix documents the benchmark’s controlled task environments, a worked analysis trap and outcomes, and operational scoring definitions. It also records budget differences and limits on interpreting some scoring dimensions.

I.1 REPORTING CONTRACT AND BUDGETS

  • Worlds within an analysis family share prompts and reporting schemas, while L1 and L2 run in a sealed sandbox and L3 uses an evidence dossier followed by three adversarial review rounds.The harness records code, tool calls, and token use; restrictions control runtime access, not prior exposure to public source data.Source: §I.1 · p. 52
  • GLM-5.3 and GPT-6 Astra were not strictly budget-matched to earlier evaluations, although both completed all 203 tasks and had verifier files for all 40 run-level graded files.GLM-5.3’s per-call ceiling rose from 8,192 to 32,768 tokens after hidden-reasoning truncation was diagnosed.Source: §I.1 · pp. 52–53

I.2 MODELS

  • The evaluation covers eight models accessed through first-party APIs, with all eight tracks frozen by 2026-08-22.Source: §I.2 · p. 53

I.3 A COMPLETE WORKED TASK

  • A worked Histone-track task asks agents to quantify promoter-level H3K27me3 signal from ChIP-seq alignments and relate it to gene expression using supplied tools and data.The prompt asks agents to verify outputs and report method, correlation, issues, usability, and trajectory.Source: §I.3 · p. 53
  • MACS3’s narrow default returns a plausible 3.4 Mb landscape for broad H3K27me3 domains, while the reference broad-mode result totals 6.4 Mb over 19,352 domains.The rubric requires a broad or domain-aware mode, a total within ±35% of 6.4 Mb, and identification of the narrow-default pitfall for top credit.Source: §I.3 · p. 53
  • GPT-5.6-sol earns a world score of 87.1 after reporting 6.43 Mb across 19,352 domains, while Opus scores 50.0 after reporting 28.21 Mb despite recognizing the tool pitfall.Opus receives zero analytical-correctness credit; human adjudication lowers GPT-5.6-sol’s result-calibration component to 0.5.Source: §I.3 · p. 53

J.1 BEHAVIORAL STATES AND DIMENSION DEFINITIONS

  • L1 distinguishes missed, partial, and complete diagnoses and actions, while its result-calibration summary excludes overdiagnosis, false unusability, excessive caution, and noncompliance.The calibration summary is not a complete measure of uncertainty calibration; fabrication is flagged heuristically, not established by semantic audit.Source: §J.1 · p. 54
  • L2 uses four mapped components, and L3 scientific reasoning averages five axes; L2 correctness and methodological response receive identical scores in 77.7% of evaluations.The four L2 dimensions are not statistically independent; among 425 challenged evaluations with full issue recognition, 126 receive exactly half calibration credit.Source: §J.1 · pp. 54–55

J.2 TWO-PART METHODOLOGY

The appendix documents grading and aggregation corrections, L3 abstention and judge procedures, a TWAS silent-failure case, verification effort, and resource-accounting conventions. It also reports model-specific costs and defines limits on interpreting token efficiency.

J.2 TWO-PART METHODOLOGY

  • Objective quantities are graded deterministically against solver-produced goldens, then independently verifier-checked, with disagreements adjudicated by humans against the rubric.The appendix also records missing verifier rows, frozen L3 cells, and distinct exclusion and prior-inadmissibility cases.Source: §J.2 · p. 55
  • Correcting a missing L2 issue flag changed 70 world scores and 23 of 48 track–model means; L1, L3, and recorded grader evidence were unchanged.The correction reclassified 34 perturbed worlds using existing manifests; no agent reruns or new transcript judgments were required.Source: §J.2 · p. 55

J.3 L3 ABSTENTION SCORING

  • A justified L3 abstention can earn evidence-use and reasoning credit but no novelty credit, and its score ceiling is 83.3/100.Prior-inadmissible outcomes are excluded before aggregation, whether or not a hypothesis was produced.Source: §J.3 · p. 55

J.4 JUDGE-PANEL STABILITY

  • Three-draw median aggregation reproduces recorded judge composites nearly exactly on the paired subset, but does not validate the rubric or rule out systematic judge bias.Small replay residuals reflect human adjudication, which the replay cannot reproduce.Source: §J.4 · pp. 55–56, Figure 19 · p. 56

K.1 THE TWAS SILENT FAILURE, FROM THE RECORDED GRADER STATES

  • On TWAS, some completed pipelines produced plausible tables with zero usable SNP weights, while others processed the full weight set and matched golden z-scores perfectly.The silent-failure state recurred on clean and perturbed TWAS worlds, distinguishing completed software execution from valid analysis.Source: §K.1 · p. 56

K.2 HOW MUCH OF THE EXECUTED WORK IS CHECKING?

  • Verification comprises 2.7–11.4% of executed L1 and L2 code cells across models, based on a keyword classifier of raw code strings.The classifier captures surface form, not code meaning.Source: §K.2 · p. 57, Figure 20 · p. 57

L.1 ACCOUNTING PROTOCOL

  • Bench-cost accounting uses the evaluated agent’s metered input and output tokens, dated first-party list rates, and successful calls; grading and verification are excluded.Dollar values are upper bounds because provider-side cache discounts are not logged.Source: §L.1 · p. 57, Table 16 · p. 58

L.2 LATER-MODEL ACCOUNTING AND SOURCE RECORDS

  • GLM-5.3’s estimated cost was $0.58 per task, while GPT-6 Astra’s reported upper bound was $0.69 per task.The GLM estimate uses 65,455,719 input and 9,759,243 output tokens; Astra’s $139.69 total conservatively applies long-context multipliers to one task’s usage.Source: §L.2 · p. 58

L.4 RECORDED CALLS AND CODE EXECUTIONS

The accounting distinguishes API calls from code executions and reports resource use across task coverage, cost, stage, and difficulty. Its comparisons show that token, dollar, and quality measures must be considered together, while cost alone cannot establish efficient or complete work.

L.4 RECORDED CALLS AND CODE EXECUTIONS

  • Both GLM models execute a median of 20 code cells per L1 task, versus eight for Sol and five for Astra; higher GLM-5.3 token use reflects more tokens per interaction, not more calls.Code executions may end in errors, L3 has a separate structure, and the reported execution medians cover L1/L2 rather than all stages.Source: §L.4 · p. 59, Table 17 · p. 59
  • Hardest-to-easiest difficulty-tertile token-use ratios are 1.51–1.86 for Sol, DeepSeek, and Kimi, versus 1.00–1.16 for GLM-5.2, Opus, and GLM-5.3.These expenditure patterns do not establish that agents recognized difficulty or deliberately allocated effort.Source: §L.4 · p. 59

L.5 COVERAGE

  • The accounting join covers 1,624 assigned model–task outcomes; 1,623 have executed outputs with usage metadata, while one pre-execution exclusion has zero cost.Nineteen canonical Claude outputs use calibrated character-count proxies; other canonical outputs use provider-reported usage.Source: §L.5 · p. 60

L.6 HEADLINE ACCOUNTING

  • Total bench cost is $913.54 over 294.76M tokens, and token and dollar rankings differ, so resource use should be considered alongside quality.GLM-5.3, DeepSeek V4 Pro, and GLM-5.2 consume the most tokens yet remain relatively inexpensive because their first-party rates are lower than frontier-lab rates.Source: §L.6 · p. 60, Table 18 · p. 60
  • Similar Overall Scores can require substantially different token and monetary expenditure, while high measured efficiency can reflect incomplete execution.The paper argues that efficiency should be optimized together with scientific validity, execution completeness, and trustworthy analysis.Source: §L.6 · p. 60

L.7 WHERE THE COST LIVES

  • L3 is the least expensive stage per task for seven models, but not GLM-5.3; eligible abstentions use 4.9–7.4× fewer median tokens than commitments.The abstention contrast compares different tasks and responses, and Astra’s committed median represents one response; cost alone cannot distinguish efficient analysis from limited work.Source: §L.7 · p. 61, Figure 21 · p. 61, Figure 22 · p. 62

L.8 PROVENANCE, LICENSES, AND RELEASE

  • The reported release materials include registries, prompts, solver goldens, graded artifacts, adjudication ledgers, and dated rate cards, but the public-release freeze remains separate.The repository and immutable release identifier were placeholders at the time described.Source: §L.8 · p. 63
Loading 2609.33357v2…