PaperlayerView exact revision on arXiv

Source-linked AI summary

Sci$\Pi$vot: Benchmarking Principal-Investigator Judgment at Pivotal Moments of Scientific AutoResearch Agents

Youran Sun, Xingyu Ren, Chugang Yi, Kejia Zhang, Di Wu, Yingkang Cao, Allen Zhou, Jianping Ye, Suying Liu, Jiaxuan Guo, Jianda Du, Weicheng Ye, Kecheng Liu, Hefan Zhang, Haixin Wang, Jie Xiao, Eric Yang, Haizhao Yang

arXiv:2610.21082v1cs.AI

TL;DR

Autonomous scientific research requires judgment about which plans merit resources, but those decisions depend on project context. SciΠvot evaluates that judgment by asking models to choose between original and human-corrected plans using pre-intervention records. With that context, four of five models prefer the original plan on the selected cases.

  • Problem

    Scientific AutoResearch agents can keep experiments running, but choosing experiments that advance the research question remains difficult and context-dependent.

    Source: §1 · p. 2, §2.2 · p. 4

  • Method

    SciΠvot evaluates 248 cases by asking models to choose between original LLM plans and human-corrected alternatives using project records from before intervention.

    Source: §1 · p. 3, §3.1 · p. 6, §3.2 · p. 6

  • Results

    At medium reasoning effort with the project record, four of five models prefer the original LLM plan over the human-corrected alternative on the selected cases.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The case pairs provide preference supervision for scientific corrections and a testbed for NLP training and evaluation methods.

    Source: §1 · p. 3

    SciΠvot covers theoretical and computational work using CPUs and GPUs, not wet-lab execution or workflows using specialized commercial scientific software.

    Source: Limitations · p. 10

Abstract

from arXiv · show

Large language model (LLM) agents can keep scientific experiments running for months with little human supervision, but reliable autonomous research also requires scientific judgment. Evaluating scientific judgment is difficult because research decisions are open-ended and depend on project context. We construct Sci$\Pi$vot, a 248-case benchmark of scientific judgment based on human corrections. The benchmark draws on records from six months of research in which senior researchers ran LLM agents on 26 projects across six scientific fields and repeatedly intervened to correct the agents' research judgments. In each case, an agent chooses between the original LLM plan and the human-corrected alternative, using only the project record available before the correction as context. We compare LLM and human performance and examine the effects of reasoning effort, access to the project record, and reconsideration. At medium reasoning effort with the project record, models systematically prefer the original LLM plan over the human-corrected alternative. Four of five frontier models perform below chance on the selected cases. The highest accuracy in the project condition across the tested models and reasoning effort settings is 63.5%, compared with 73.4% for fresh researchers. At medium effort, every model achieves lower accuracy with the project record than without it, and reconsideration does not consistently improve overall accuracy. Even with a human-corrected alternative available, frontier models struggle with the selected research decisions, while Sci$\Pi$vot supports systematic evaluation of scientific judgment for LLM agents.

1 Introduction

Autonomous research depends not only on running experiments, but on judging which experiments can advance the scientific question. SciΠvot makes that judgment measurable by asking models to choose between original and human-corrected plans using the pre-intervention project record.

1 Introduction

  • Reliable autonomous research requires diagnosing what an experiment can establish and revising plans as evidence arrives.Senior researchers repeatedly intervened because plausible plans could consume weeks without resolving the research question.Source: §1 · p. 2
  • SciΠvot pairs original LLM plans with human-corrected alternatives reconstructed from completed histories across 248 cases and 26 projects.Models choose between the plans using the record available before intervention.Source: §1 · p. 3
  • At medium effort with the project record, four of five models prefer the original LLM plan over the human-corrected alternative on the selected cases.Further deliberation can undo a correct choice and return to the same error, which the paper calls re-trapping.Source: §1 · p. 3
  • SciΠvot’s case pairs provide preference supervision for scientific corrections and test how agents read project records, compare plans, and justify choices.The paper presents these connections as a testbed for NLP training and evaluation methods.Source: §1 · p. 3

2 Scientific Judgment as a Bottleneck

The paper frames scientific judgment as a bottleneck for autonomous research: agents need to choose experiments that answer the research question, not merely keep experiments running. Human corrections illustrate recurring ways plans can fail to do so.

2.1 What We Require of AutoResearch

  • Frequent human intervention limits how many projects one scientist can supervise, making better judgment important to autonomy and scale.The program’s requirements also include parallel projects across disciplines and original, publication-worthy research.Source: §2.1 · p. 3

2.2 What Scientific Quality Depends On

  • Scientific quality depends on model capability, agentic systems, and human judgment, while accumulated rules can be difficult to follow and consume tokens.Existing checks did not reliably replace researchers’ diagnosis of mistaken assumptions or choice of the next test.Source: §2.2 · pp. 3–4

2.3 Two Examples of Human Scientific Judgment

  • In the Transformer example, a zero score prompted a larger-model retest, but a later audit found the scorer had skipped answer extraction.The researcher instead retained the small model and asked for suitable tasks.Source: §2.3 · p. 4, Figure 2 · p. 4
  • In the cloth example, the researcher challenged fixing four parameters at zero and called for fitting all six.The agent had proposed refining a two-parameter search after sampled fits reached the boundary.Source: §2.3 · p. 4, Figure 2 · p. 4
  • The 248 cases cover judgments including focus, planning, adapting, anomaly awareness, evidence sufficiency, and repetition; categories can overlap.Focus includes experiments that run correctly yet leave the research question unanswered.Source: §2.3 · p. 5, Table 2

3 From Human Corrections to Evaluation

SciΠvot reconstructs decisions from completed project histories, then evaluates whether models select the human-corrected plan using only evidence available at the decision point. Its cases capture recurring scientific judgment failures and complement benchmarks focused on other research activities.

3.1 What Should the Evaluation Preserve?

  • The evaluation preserves the uncertainty of the original decision by showing evaluators the earlier project record and two plans, without later outcomes.The full history links interventions to subsequent investigation and evidence of their scientific value.Source: §3.1 · p. 6

3.2 Recovering a Decision from Its History

  • Case writers reconstruct the original plan and human-corrected plan from project history, then score whether evaluators select the reference plan.Evaluators receive the project record at the commit before intervention, not the later history.Source: §3.2 · p. 6

3.3 Recurring Failure Modes

  • The taxonomy groups recurring failures such as experiments that run correctly yet leave the research question unanswered; cases can have secondary judgment labels.Historical-error coding assigns 243 cases a primary label, with up to two secondary labels.Source: §3.3 · p. 6
  • Benchmark construction requires a recoverable project state and documented human correction, and fields are grouped by each project’s primary objective.Biological research using neural models is classified as a biological application.Source: §3.3 · p. 6

3.4 Relation to Existing Benchmarks

  • SciΠvot evaluates planning and decision-making from experimental records and senior researchers’ corrections, unlike benchmarks centered on replication, workflows, or defined objectives.Its cases come from real frontier scientific research.Source: §3.4 · pp. 6–7, Table 3 · p. 7

4 Can Models Recognize Human Corrections?

The evaluation tests whether models can choose human-corrected plans using project records, and finds that models often favor the rejected original plan. Added context and further deliberation do not reliably improve their judgments.

4.1 Evaluation Setup

  • The main comparison uses medium effort, presents each case in both option orders, and averages correctness across orders with equal weight for each case.The project condition provides the two plans and project record; the options-only condition presents the same plans without the record.Source: §4.1 · p. 7

4.2 Models Often Reject the Human-Corrected Plan

  • At medium effort with the project record, four of five models select the rejected plan more often than the reference plan; Astra scores 59.5% and the five-model average is 39.2%.Astra’s highest observed project score is 63.5% at xhigh, versus 73.4% for the human baseline.Source: §4.2 · p. 8
  • All five models choose the rejected plan in both orders on 49 cases, while all choose the reference plan on 16; this pattern reflects the original plan’s on-policy status and the correction’s off-policy status.The passage identifies systematic preference for the original plan as the structural signature of this asymmetry.Source: §4.2 · p. 8

4.3 Different Judgments Reveal Different Gaps

  • The cited comparison reports 56.1% on planning cases and 54.9% on focus cases, and illustrates that experimental scale should match investigation needs rather than consistently favor larger or smaller experiments.In case 21, models retained a 200-item-per-cell batch in seven of ten pilot responses; in case 13, they retained three short gated runs in six of ten responses despite a need for a full-length training run.Source: §4.3 · p. 8

4.4 Lower Accuracy with the Project Record

  • At medium effort, mean accuracy was 39.2% with the project record versus 55.8% without it; paired 95% intervals excluded zero for four models, while Astra’s included zero.The comparison evaluates the complete project-condition protocol, including retrieval, interpretation, and the client-specific reading interface.Source: §4.4 · p. 9

4.5 More Thinking Can Lead Back to the Same Mistake

  • Higher effort has inconsistent effects: Sol falls from 36.1% at low effort to 29.2% at max, while Terra rises to 52.0% after one review then falls to 42.7% after the second.Across five models, the first review repairs 422 errors and introduces 302; the second repairs 264 and introduces 305. In the Transformer case, Opus returns to a larger-model plan using a premise contradicted by frozen code.Source: §4.5 · p. 9

5 From Diagnosis to Better AutoResearch Agents

SciΠvot turns senior researchers’ corrections into tests of scientific judgment for AutoResearch agents. The paper proposes preference-based training as a possible route toward more autonomous research, while leaving its effectiveness on new projects as a future test.

5 From Diagnosis to Better AutoResearch Agents

  • SciΠvot converts human corrections from sustained research into tests of PI Judgment, including adapting to experimental results and keeping experiments focused on the research question.Even with project records and further deliberation, models often return to decisions senior researchers corrected.Source: §5 · p. 10
  • The case pairs could provide preference supervision for research actions, and a preference scorer trained on them could supply rewards for agentic reinforcement learning.Whether such training improves decisions in new projects and reduces human intervention remains a future test.Source: §5 · p. 10
  • Learning shared judgments could support more autonomous scientific AutoResearch agents across disciplines, including research aimed at improving AI systems.Source: §5 · p. 10

Limitations

SciΠvot evaluates recognition of corrective decisions from senior-researcher-supervised projects, not autonomous research generation or execution. Its selected cases do not estimate how frequently such failures occur across scientific research.

Limitations

  • Coverage is limited to theoretical and CPU/GPU computational research, including simulation and analysis of existing data; wet-lab and specialized commercial software workflows are excluded.Other settings would be needed to test transfer across different tools and experimental procedures.Source: Limitations · p. 10
  • The benchmark measures recognition of selected corrective decisions; it does not estimate how often such failures occur across scientific research.It also does not evaluate agents’ generation of plans or execution of research.Source: Limitations · p. 10
  • Effort and reconsideration experiments use available interface controls, not equal token budgets, controlled compute scaling, or verifier-guided search.The evaluated clients also expose different reading tools.Source: Limitations · p. 10

Ethics Statement

The paper describes privacy protections for released project materials and reports consent and anonymized identifiers for fresh researchers. It also specifies which benchmark materials are publicly available.

Ethics Statement

  • Public materials should retain evidence needed to understand each decision while removing personal identifiers, account information, and unrelated private discussion.Fresh researchers were informed about the study, gave consent, and are reported under anonymous identifiers.Source: Ethics Statement · p. 10
  • The authors used language models for drafting, literature organization, and analysis code, while retaining responsibility for checking claims, references, and results..Source: Ethics Statement · p. 10
  • The released materials include all 248 cases, exact option wording, reference answers, prompts, and aggregate results for twelve conditions.Individual model response logs are not included; frozen project records are too large to upload and will be hosted in the repository.Source: Ethics Statement · p. 10

A Model Results and Statistical Details

The appendix documents the project-reading clients, scoring and uncertainty procedures, and project and effort coverage underlying the model results.

A.1 Project-Reading Clients

  • Opus and Grok use reading and search tools through a shared Claude client, while GPT models use Codex shell commands; the options-only condition removes the project record.Because tools differ by client, the condition comparison includes retrieval, interpretation, and client-specific tool interfaces.Source: §A.1 · p. 13

A.2 Scoring and Uncertainty

  • All 29,760 retained runs completed without transport errors; eight unparsed responses score zero, while both medium-effort primary conditions are fully parsed.The scoring retains the last record for each specified run combination, replacing failed or superseded attempts rather than counting retries as additional observations.Source: §A.2 · p. 13
  • Uncertainty is estimated using case-bootstrap intervals and paired project-minus-options differences, with project-cluster intervals also reported because cases share projects.Option orders are repeated measurements, and the same sampled cases or projects are used in both conditions when estimating differences.Source: §A.2 · p. 13

A.3 Project Coverage and Effort Settings

  • The benchmark uses the same 248 cases and two option orders per case; requested effort labels are interface settings, not equal token budgets or compute.Project sizes range from one to 43 cases, so averaging project means differs from averaging cases.Source: §A.3 · p. 13

A.4 Option Order and Reconsideration

The appendix describes how option order and reconsideration are evaluated, reports a writer-sensitivity pilot, and introduces illustrative scientific decisions.

A.4 Option Order and Reconsideration

  • Option orders are equally weighted rather than selecting the better order, and observed A/B differences may reflect position preference or an interaction with reasoning.The reconsideration experiment follows project-condition answers through two medium-effort continuation stages and tracks changes in selected plans.Source: §A.4 · p. 13

A.5 Writer Sensitivity

  • The writer pilot compares three writers on 13 shared cases with matching frozen commits and option hashes across writers.Seven other shared cases are excluded because their frozen commits differ across writers.Source: §A.5 · p. 14
  • In the writer-sensitivity pilot, the released-set writer never leads on its own cases; Opus scores 34.6% on Grok-written project-record cases versus Grok’s 23.1%.Writer identity changes absolute case difficulty, but the released-set writer has no advantage over peers.Source: §A.5 · p. 14

B Two Illustrative Scientific Decisions

  • The illustrative examples show human interventions changing what projects can learn next, with Table 13 reporting five models’ reference-plan choices in both option orders.The reference and rejected plans reproduce the benchmark task wording.Source: B Two Illustrative Scientific Decisions · p. 14

B.1 Checking Evaluation Before Changing Model Size

The examples and coding guidance show how project context distinguishes sound scientific judgment from plans that change model size, retain restrictive assumptions, or act on weak evidence. The section also describes the benchmark’s annotation boundaries, judgment-category profiles, illustrative cases, and human-baseline protocol.

B.1 Checking Evaluation Before Changing Model Size

  • 70 of 480 code items were correct after regrading, but the same four difficulty cells remained unfilled, exposing the scoring fault without completing the asset.The reference plan kept the 0.8B model, used official grading functions, and required all 12 cells before matched-depth runs; the rejected plan tested 2B first.Source: §B.1 · pp. 14–15

B.2 Testing a Restrictive Physical Model

  • The cloth project’s ten sampled minima all lay at one search-domain corner, prompting a six-parameter fit alongside the original two-parameter slice rather than treating four zeroed parameters as fixed.The reference vector included published-fit values and simulator defaults; the plan used the defaults as the search center and compared both fits on the same objective.Source: §B.2 · pp. 15–16

C.1 Coding the Historical Decision

  • The codebook organizes decisions around research goals, evidence interpretation, and experiment or resource choices, with categories including planning, adaptation, and anomaly awareness.Planning concerns whether experiment scale or sequence resolves the next question; adaptation concerns using available results to revise a hypothesis or plan; anomaly awareness concerns diagnosing suspect results before interpreting them.Source: §C.1 · p. 16
  • The remaining categories distinguish unsupported conclusions, work that adds no new information, and experiments that answer a different question from the project’s target.A justified negative conclusion or budget-driven stop is not an evidence-sufficiency failure; planned independent replication is not repetition.Source: §C.1 · p. 16

C.2 Boundaries and Review Procedure

  • All 248 cases have judgment labels, while 243 receive a primary historical-error label; error labels follow the smallest decision change needed to correct a plan.Annotators separate project records, agent interpretations, and original human messages, and do not use post-intervention outcomes as evidence available at the decision point.Source: §C.2 · pp. 16–17

C.3 Profiles by Judgment Category

  • Figure 3(b) reports case-weighted accuracy by judgment category, and its scores describe this benchmark’s cases rather than general abilities independent of the set.Categories may overlap, and repetition has only four cases; limiting analysis to the 243 cases with primary error labels changes model scores by at most 1.6 percentage points.Source: §C.3 · p. 17

C.4 Additional Illustrations

  • The examples show that appropriate experiment scale can point in opposite directions, while cloth and simulator cases illustrate checking whether measurements answer the intended question.The batch-API example starts with 20 items per cell before 200-item tables; the reinforcement-learning example instead requires one seed to run the full 8,000 iterations, both as local project requirements.Source: §C.4 · pp. 17–18

D Human Baseline Protocol

  • Thirteen fresh PhD students and postdocs learned the source projects from the same records, then chose between plans with an assistant available but no role in the final choice.The baseline is intended to measure what researchers can recover about the senior researcher’s correction from the project record.Source: D Human Baseline Protocol · p. 18

D.1 Coverage and Comparison

Fresh researchers answered all benchmark cases using the project records, providing a human comparison for the model evaluations.

D.1 Coverage and Comparison

  • 73.4% accuracy (182 of 248 cases) was achieved by fresh researchers, who recorded responses for all 248 cases.The same accuracy measure is used within each scientific field or judgment category in the radar plots.Source: §D.1 · p. 18

E Evaluation Prompts and Available Materials

The evaluation varies access to project records, requested reasoning effort, reconsideration, and option order. The appendices document the prompts, available materials, and conventions for interpreting supplementary tables.

E.1 Twelve Evaluation Conditions

  • Ten initial conditions combine project-record and options-only settings with five requested effort levels; two further conditions continue the medium-effort project session through reconsideration.Each reconsideration retains the prior conversation and provides no correctness feedback; every condition is evaluated in both option orders.Source: §E.1 · p. 18

E.2 Prompts and Available Materials

  • The project prompt directs models to read project files and choose between two plans, while the options-only prompt presents the plans without project access.Network access and access outside the project directory are prohibited in the project condition.Source: §E.2 · pp. 18–19
  • The prompt materials specify the two option placeholders, a choice rationale and response format, and a reconsideration prompt that permits keeping or changing the prior answer.The archive includes prompts, source materials, and the complete 248-case set with anonymized identifiers, exact options, and reference answers.Source: §E.2 · pp. 18–19
  • Tables 7 and 8 describe benchmark projects, with project questions framed as scientific objectives rather than established findings.Area labels describe the current sample, and secondary methodological links are not counted separately.Source: Table 7 · p. 20, Table 8 · p. 21
  • Table 9 reports accuracy at every requested effort setting as Project / Options only, counting unparsed completed responses as incorrect.Source: Table 9 · p. 21
  • Tables 10 and 11 document paired accuracy-change intervals and accuracy by option order, respectively.Table 10 averages both option orders within cases and uses paired bootstrap intervals; Table 11 reports 248 responses per order.Source: Table 10 · p. 21, Table 11 · p. 22
  • Tables 12–14 report a controlled writer pilot, medium-effort reference-plan choices across option orders, and choices for four selected examples.The example cases were selected for decision structure before model scores were inspected; repeated responses are not independent cases.Source: Table 12 · p. 22, Table 13 · p. 23, Table 14 · p. 24

Related papers

Loading 2610.21082v1…