PaperlayerView exact revision on arXiv

Source-linked AI summary

Are We Measuring Scientific Intelligence? Rethinking the Evaluation of AI Scientists

Kate Zhang, Yuante Li

arXiv:2610.04915v1cs.AI

TL;DR

A correct answer on a data-driven benchmark does not establish that an AI agent used the supplied data, and a single-run score cannot distinguish evidence-based answers from recalled ones. The paper tests this with verified data versions that preserve, withdraw, or reverse evidence. On single-cell questions, Claude agents had 95% accuracy but 41% evidence-grounded accuracy, and benchmark scores and LLM judges could still reward answers that ignored changed evidence.

  • Problem

    A correct answer scored against a fixed key may come from prior knowledge or option elimination rather than the supplied data, and a single-run score cannot distinguish these routes.

    Source: §1 · p. 2

  • Method

    The study edits each item’s data to preserve, withdraw, or reverse evidence, verifies each edit with a pre-registered statistic, and runs the same agent on every version to assess agents and evaluators.

    Source: §1 · p. 2, §4 · p. 4

  • Results

    Claude agents had 95% accuracy but 41% evidence-grounded accuracy, while benchmark scores and LLM judges could reward answers that ignored changed evidence.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    Evaluating data-driven scientific reasoning requires testing whether answers follow evidence changes and whether scores reward appropriate responses.

    Source: §7 · p. 9

    The study covers 18 BAIS-SD questions and four GeneBench-Pro problems, with gene-name anonymisation tested on ten tasks; responsiveness to evidence is necessary but not sufficient for scientific quality.

    Source: A Discussion · p. 14

Abstract

from arXiv · show

AI agents can now carry out data-driven scientific analyses end to end, and benchmarks assess them by giving an agent a question and a dataset and scoring its final answer against a fixed key. These benchmarks assume that a correct answer was derived from the supplied data, a property we call evidence grounding. However, an agent can also reach the key from prior knowledge or by ruling out the other options, and a score based on a single run cannot tell these cases apart. We show how to test this assumption and find that it often fails. For each question, we build versions of its data files in which the evidence for the answer is left intact, withdrawn or reversed, check each edit with a pre-registered reference statistic, and run the same agent on every version. We then measure evidence-grounded accuracy, which credits a correct answer only if the agent also responds when the evidence is withdrawn and follows it when it is reversed. We evaluate three agent scaffolds and five models on 18 single-cell questions from BAISBench and four synthetic problems from GeneBench-Pro. On the single-cell questions, the Claude agents are 95% accurate and answer 83% correctly without any data, but their evidence-grounded accuracy is only 41%. Hiding gene names raises the share of runs that follow reversed evidence from 58% to 93% on ten gene tasks, suggesting that prior knowledge competes with the supplied data. The benchmark score and LLM judges can also reward answers that ignore the changed evidence. Measuring scientific intelligence rather than recall therefore requires checking whether answers follow the evidence and whether scores reward them for it.

1 Introduction

Scientific benchmarks use answer scores to compare AI agents and make claims about scientific capability, but a correct answer alone does not show that it came from the supplied data. This section proposes verified data counterfactuals to test whether agents and evaluators respond appropriately when evidence changes.

1 Introduction

  • Benchmarks for AI scientists pair real or synthetic data with verifiable answers and are used to compare systems, track progress, and argue that agents can do science.The paper situates this growth within end-to-end scientific agents that plan analyses, run code, and interpret results.Source: §1 · p. 1
  • The study tests evidence grounding, the requirement that a correct answer be derived from the data supplied in that run.A single-run score cannot distinguish data-derived answers from answers based on prior knowledge or elimination.Source: §1 · pp. 1–2
  • Existing controls are aggregate or one-off and audit agents rather than whether scores reward responses to new evidence.Examples include no-data baselines, shuffled data, label swaps, and process judges.Source: §1 · p. 2
  • The proposed protocol creates preserved, withdrawn, and reversed versions of each item’s data, verifies edits with a pre-registered statistic, and runs the same agent on each version.It tests both whether agents follow evidence and whether benchmark scores and LLM judges reward that response.Source: §1 · p. 2
  • Claude agents score 95% accuracy but only 41% evidence-grounded accuracy, while the benchmark and LLM judges can reward answers that ignore changed evidence.Without data, Claude agents answer 83% correctly; hiding gene names raises flip-following from 58% to 93% on ten gene tasks.Source: §1 · pp. 2–3
  • The paper formalises evidence grounding through invariance and responsiveness, measures it with evidence-grounded accuracy, and separates agent grounding from evaluator validity.Its contributions also include a verified counterfactual protocol, gene-name anonymisation, and distinct mitigations for evaluator blind spots.Source: §1 · pp. 2–3

2 Related work

Prior work studies benchmark validity, process evaluation, and counterfactual tests, including in scientific benchmarks. The paper identifies a remaining gap: these approaches do not establish whether a particular credited answer follows verified evidence changes or whether the score rewards that response.

Benchmarks for AI scientists and scientific discovery

  • Scientific benchmarks have expanded across knowledge, question answering, data-driven analysis, code execution, replication, and computational biology tasks.Stronger models may reach reference answers through recall, option elimination, or exploiting the grader.Source: §2 · p. 3

Validity of benchmark scores

  • Process evaluation grades intermediate steps, but a judge sees one run and may rely on an agent account that does not reflect what produced its answer.Prior work reports that written reasoning need not reflect the cause of an answer and that LLM judges can be misled by completion claims.Source: §2 · p. 3

Counterfactual tests

  • Scientific benchmarks’ existing data tests show average performance dependence, but not whether a particular credited answer follows evidence or whether its score rewards responding to verified changes.The proposed distinction is to edit each item’s raw data, verify edits with a reference statistic, and test both agent and benchmark score.Source: §2 · p. 3

3 Preliminaries: a taxonomy of current evaluation

A data-driven benchmark item combines a task, supplied data, and a reference answer; an agent uses tools and observations to produce its answer. Existing evaluation families differ in what they compare and whether they test individual evidence responses or evaluator behavior.

3 Preliminaries: a taxonomy of current evaluation

  • An agent acts on a task and data through tool calls and observations before producing a final answer.The data reach the answer through observations, while model parameters carry pretraining; a correct answer alone therefore does not reveal the data’s contribution.Source: §3 · p. 3
  • Only verified counterfactual evaluation tests single answers against a known correct response and also tests the evaluator.Single-run scores, aggregate ablations, and unverified perturbations do not establish whether a particular correct answer tracks supplied evidence.Source: Table 1 · p. 4, §3 · p. 4
  • Current evaluation includes outcome scoring, process scoring, input ablation, and ad hoc perturbation, each comparing different aspects of an answer or run.Outcome and process scoring assess one run; input ablation compares aggregate accuracy, while ad hoc perturbation edits one item without verifying the appropriate response.Source: §3 · p. 4

4 Method: evidence grounding and verified counterfactuals

The method uses verified data counterfactuals to test whether agents follow evidence and whether evaluators reward that behavior. Paired runs distinguish evidence-grounded answers from answers that happen to match the original key.

4.1 Problem formulation

  • Evidence-grounded answers remain unchanged under null edits, react to withdrawn evidence, and follow the data-supported answer after a flip.The criterion concerns the answer’s relationship to evidence, not the analysis path, model, or scaffold.Source: §4.1 · p. 4
  • A single run cannot establish grounding: two agents can produce the same trajectory and answer on original data yet respond differently when the evidence changes.An evaluator seeing only the original run must score them identically, despite their different responses to an intervention.Source: §4.1 · p. 4

4.2 Constructing verified counterfactuals

  • Counterfactual versions preserve, withdraw, or reverse evidence, with each edit verified against a pre-registered statistic across data files and raw counts.Edits target quantities read by the statistic; a canonical writer also helps prevent file metadata from revealing changes.Source: §4.2 · p. 5

4.3 Scoring agents and evaluators

  • Evidence-grounded accuracy credits a correct answer only if it responds to withdrawal and follows the flip; paired runs also distinguish derived answers from other routes.Only the derived route is evidence-grounded; elimination can preserve the key after withdrawal, while recalled answers can persist through a flip.Source: §4.3 · pp. 5–6
  • Counterfactually valid evaluators remain stable under null edits, stop crediting the original key when unsupported, and favor answers that follow a verified flip.The protocol measures null sensitivity, the withdrawal blind spot, and flip AUC on the same paired runs used to assess agents.Source: §4.3 · p. 6

5 Experimental setup

The experiments cover data-driven single-cell questions and synthetic genetics problems, using multiple agent scaffolds and models. Evaluators are tested on paired runs to assess whether their scores reflect evidence-following behavior.

5 Experimental setup

  • The study uses 18 BAIS-SD questions from 13 studies and four GeneBench-Pro synthetic problems; the BAIS-SD items are data-decidable under a reference statistic.Four otherwise eligible BAIS-SD questions were excluded because their original data did not support the benchmark key.Source: §5 · p. 6
  • Three scaffolds and five models are evaluated, with three replicates for Claude Code and Codex CLI and one Biomni replicate.Runs use fresh sandboxes, receive the same prompt across data versions, and have no web or retrieval access; a no-data condition supplies only the question and background.Source: §5 · p. 6
  • The evaluator audit compares the official score, LLM process judges, and agents’ own reports on the same paired runs.An audit of 2,944 runs and 16,091 tool calls flagged no access outside the sandbox or to benchmark answer material.Source: §5 · p. 6
  • The results are organized around evidence tracking, evidence-grounded accuracy, reasons for failures, and whether evaluators detect or can repair those failures.Source: §5 · p. 6

6 Results and analysis

Across the tests, many agents keep the keyed answer when evidence is withdrawn or reversed, and most correct answers do not meet both evidence-response conditions. Anonymisation increases flip-following, while benchmark scores and LLM judges often fail to reward it.

6.1 RQ1: Do agents track the evidence?

  • 98% answer agreement under null edits matches rerun agreement, but after withdrawal Claude agents retain 77% of benchmark credit; only 35% flag uncertainty, and 84% of those still give the key.Trajectories indicate that agents often reach the keyed answer by elimination or memory; the benchmark cannot distinguish those answers from analysis.Source: §6.1 · p. 7
  • 61% of runs follow reversed evidence, while 30% keep the key and still receive credit; ordinary-looking cell-proportion reversals are followed only slightly more often, at 67%.The sticky original answer persists across scaffolds and without the two tasks that allow a second valid analysis.Source: §6.1 · pp. 7–8

6.2 RQ2: How much of the accuracy is evidence-grounded?

  • 95% accuracy falls to 41% EGA for Claude agents; every agent loses more than a third of its accuracy, and EGA ranks Sonnet and GPT-5.6-Sol lowest but Haiku highest.The strict variant, which also requires flagging missing evidence, reduces all agents to 17–28% and leaves the ordering inverted.Source: Figure 4 · p. 8, §6.2 · p. 8

6.3 RQ3: Why do agents not follow the evidence?

  • 83% of Claude answers are correct without data; hiding gene names raises flip-following from 58% to 93% on ten gene tasks, while data-backed accuracy barely changes.Anonymisation also removes semantic cues, so the result supports prior competition as an explanation without establishing it as the only one.Source: §6.3 · p. 8
  • On GeneBench-Pro, answers transform with flipped ground truth about as often as repeated runs agree on unchanged data, but accuracy remains low.With little task-specific prior to fall back on, responsiveness appears limited mainly by analytical precision.Source: §6.3 · p. 8

6.4 RQ4: Do evaluators notice, and can they be repaired?

  • Flip AUC is 0.11 for the benchmark score and 0.12 for an LLM judge, indicating that both favor answers ignoring flips; targeted repairs address separate failures but none covers all.Grounding prompts lift judges’ flip AUC above 0.7, while an insufficient-data option cuts withdrawal credit without changing intact-data accuracy.Source: §6.4 · p. 9, Figure 6 · p. 9

7 Conclusion

High accuracy on data-driven benchmarks does not establish that agents used the supplied data: Claude agents averaged 95% accuracy but 41% evidence-grounded accuracy. The paper recommends testing whether both answers and evaluators respond to verified evidence changes, while noting that responsiveness alone does not establish scientific quality.

7 Conclusion

  • 95% accuracy but 41% evidence-grounded accuracy for Claude agents shows that benchmark correctness can coexist with answers that do not track evidence changes.The conclusion recommends reporting evidence-grounded accuracy alongside conventional accuracy and testing whether scores respond to verified changes in evidence.Source: §7 · p. 9

A Discussion

  • EGA measures responses to interventions, not reasoning within a run; withdrawal counts either a changed answer or, under the strict variant, an explicit statement that the data cannot decide.The routes are behavioral patterns consistent with recall or elimination, not direct observations of those processes.Source: A Discussion · p. 14
  • Exact-match scoring against the original key penalizes answers that follow flipped evidence, so it measures key agreement rather than evidence-following.Under the standard prompt, LLM judges also followed the key; only a grounding prompt changed this behavior.Source: A Discussion · p. 14
  • The authors recommend verified withdrawal and flip versions, reporting both EGA variants and no-data priors, offering an “insufficient data” option, and validating process judges on counterfactual pairs.They also recommend crediting transform consistency where ground truth transforms exactly and passing every file version through one canonical writer.Source: A Discussion · p. 14
  • The study covers 18 BAIS-SD questions and four GeneBench-Pro problems, and the authors state that evidence responsiveness is necessary but not sufficient for scientific quality.They identify compute and budget constraints and call for larger task sets, more replicates, open-weight models, and human validation of LLM judges.Source: A Discussion · p. 14

B A worked example of the definitions

The worked example follows one endothelial-marker question through verified null, withdrawal, and flip edits. Its 16/16 accuracy contrasts with 3/16 evidence-grounded accuracy, while exact-match scoring rewards answers that ignore changed evidence.

B A worked example of the definitions

  • Deleting PECAM1 made the reference statistic undefined, while swapping its values with PAX7 changed the data-supported answer from C to D.The null edit preserved C, and each intervention was checked by recomputing the statistic on the edited files and raw counts.Source: B A worked example of the definitions · p. 14
  • Opus 5.5 noticed that PECAM1 was absent after withdrawal but still answered C, illustrating that noticing an edit need not change the final answer.It justified C as “the only option left.”Source: B A worked example of the definitions · p. 14
  • 16/16 runs answered the item correctly, but only 3/16 were evidence-grounded, because the remaining runs followed elimination or recall routes.Haiku followed the flip in all three replicates; two GPT-5.6-Terra runs followed the flip without flagging withdrawal, and eleven runs kept the original answer under both edits.Source: B A worked example of the definitions · pp. 14–15
  • Exact-match scoring gave full credit after withdrawal and to all eleven runs retaining C after the flip, while the five runs following the evidence to D received none.The evaluator’s flip AUC was 0; every evidence-following run scored below every run that did not.Source: B A worked example of the definitions · p. 15

C Benchmarks and tasks

The study tests data-grounding on selected BAIS-SD and GeneBench-Pro tasks using verified evidence-preserving, withdrawal, and flip versions. Agents and evaluators are assessed on paired runs, including how their scores respond to changed evidence.

C Benchmarks and tasks

  • The study uses 18 of 193 BAIS-SD questions and four GeneBench-Pro problems selected because their data edits have exactly known effects on the ground truth.BAIS-SD questions come from published single-cell studies; the selected GeneBench-Pro problems are among ten public problems.Source: C Benchmarks and tasks · p. 15
  • The BAIS-SD tasks span five evidence families, while the GeneBench-Pro edits transform ground truth through swaps, relabelling, recoding, or locus changes.All tasks use null edits that permute cells and genes and replace identifiers; every version is checked against the reference statistic on staged files and raw counts.Source: C Benchmarks and tasks · p. 15, Table 4 · p. 16, Table 5 · p. 16
  • Four BAIS-SD candidates were excluded because the original data did not support the benchmark key.The cases involved tied expression, an absent gene, a stronger alternative marker, or a more down-regulated alternative gene.Source: C Benchmarks and tasks · p. 15
  • Without two tasks admitting a second defensible analysis, flips were followed in 65% of runs and withdrawal credit was 76%, versus 61% and 77% across all tasks.This comparison indicates that those two alternative analysis paths do not account for the overall pattern.Source: C Benchmarks and tasks · p. 15

D Agent protocol

  • Agents received the same prompt across data versions in fresh sandboxes with task files only, no web or retrieval access, and a 40-minute limit.Runs returned an answer, confidence, data-support judgment, and short evidence statement.Source: D Agent protocol · p. 16

E Evaluators

  • The process judge scored answer matching, analysis quality, evidence support, and overall performance from a condensed trajectory, while a second judge showed high agreement on overall scores.The cross-family judge’s overall-score agreement with Opus was 0.91 Spearman and 0.89 Krippendorff’s α on n = 306.Source: E Evaluators · p. 17
  • The benchmark score changed 0.02 under null edits, 0.67 under flips, and 0.22 under withdrawal; the standard judge changed 0.42 under flips and 0.25 under withdrawal.The judge’s analysis-support sub-score tracked withdrawal more than flips (0.49 versus 0.26); the GeneBench-Pro grader changed only 0.21 under flips versus 0.08 under null edits.Source: E Evaluators · p. 17

F Additional analyses

Additional analyses show that responsiveness varies across evidence families and tasks, while confidence and scoring can fail to distinguish evidence-following from key retention. Gene-name anonymisation increases flip-following, though prior knowledge and elimination remain possible.

F Additional analyses

  • Marker-gene questions had the highest no-data accuracy; agents flagged deleted markers in only 24% of runs, and 69% of withdrawal credit survived.Withdrawal was flagged most often for abundance questions (53%), while differential-expression questions followed flips least.Source: F Additional analyses · p. 17, Table 7 · p. 18
  • Withdrawal preserved full credit in every run on six of 18 tasks, while two tasks never followed the flip.Among three tasks rarely answerable without data, two followed every flip; the third admitted an alternative analysis path.Source: F Additional analyses · p. 18, Figure 8 · p. 18
  • After a flip, confidence remained 77 and 59% of runs reported full data support, including 13% of runs that retained the key.Runs that followed the flip reported full support in 84% of cases; after withdrawal, mean confidence fell from 90 to 69 and only 16% reported full support.Source: F Additional analyses · p. 18, Figure 9 · p. 19
  • The withdrawal blind spot and partial flip response held for every model, while median wall time rose from 30 seconds on intact data to 49 after withdrawal and 38 after a flip.After withdrawal, extra analysis usually confirmed that a gene or group was missing, yet most runs still gave the key.Source: F Additional analyses · p. 19, Table 8 · p. 19, Table 9 · p. 19
  • Transform-consistent GeneBench-Pro answers occurred on every problem, yet the strict grader often failed to pass them; overall transform consistency was 53%, near the 47% agreement between repeated unchanged-data runs.The authors suggest run-to-run variability contributes to inconsistent responses; GPT-5.6-Sol showed 50% transform consistency against 50% replicate agreement, and Biomni had transform-consistent flips on two problems despite no flipped answers passing.Source: F Additional analyses · p. 19
  • Flip-following was 62% among runs with a correct no-data answer and 57% among the others; removing study background lowered no-data accuracy only from 83% to 78%.The authors characterize anonymisation, rather than this run-level comparison, as the direct test of prior knowledge.Source: F Additional analyses · p. 20

G Gene-name anonymisation

  • Across ten anonymised gene tasks, the reference statistic still supported the same option on every task; excluding the background-answerable task, flip-following rose from 54% to 93% for Claude and 48% to 89% for GPT.Evidence-grounded accuracy also rose from 35% to 65% for Claude and from 13% to 50% for GPT.Source: G Gene-name anonymisation · p. 20
  • After withdrawal of a queried gene, agents sometimes selected the absent option by elimination after checking the three remaining genes and finding none fit.Their explanations explicitly described this route, accounting for some withdrawal credit that persisted under anonymisation.Source: G Gene-name anonymisation · p. 20

H Repairs

The repairs test abstention and grounding-focused judging on the same tasks, models, and protocol. Abstention responds to withdrawn evidence but rarely to flipped evidence, while the grounding-focused judge better distinguishes answers that follow flips.

Evaluation setup

  • 486 further Claude Code runs test an abstention option across 18 tasks, three models, three replicates, and three data versions.The grounding-focused judge regrades existing runs rather than requiring new ones.Source: H Repairs · p. 20

Named and anonymised gene tasks

  • For Claude models, task-bootstrap intervals exclude zero for no-data accuracy, flip-following, withdrawal and flip credit, withdrawal flagging, and EGA, but not original-data accuracy.The table reports 95% task-bootstrap intervals for differences between named and anonymised gene tasks.Source: Table 11 · p. 21

Abstention option

  • 57% of withdrawal runs abstain, but only 6% of flip runs do; key credit is 31% after withdrawal and 21% after a flip.Agents never abstain on intact data, where accuracy is 97% in these runs.Source: H Repairs · p. 21, Table 12 · p. 21

Grounding-focused judge

  • 0.82 flip AUC under the grounding-focused judge, versus 0.11 for the benchmark score across all Claude runs.On the common run set, the respective flip AUCs are 0.83 and 0.11; withdrawal sensitivity rises from 0.25 to 0.44 while null sensitivity remains 0.06.Source: H Repairs · p. 21

I Case studies

Case studies show agents can earn full benchmark credit by relying on prior knowledge or rejecting evidence that contradicts expected answers. Conversely, an agent that tracks a flipped measurement can receive zero when its estimate misses the grader’s tolerance.

Answer from memory, rewarded

  • After TMPRSS2 is deleted, Sonnet identifies its absence and calls its answer a low-confidence extrapolation from prior knowledge, yet receives full benchmark credit.With an abstention option on the same data, the model abstains; the trajectory already contained the information needed to abstain.Source: I Case studies · p. 21

Cross-scaffold memory use

  • After deep-caries cells are removed, Opus, GPT-5.6-Sol, and Sonnet answer VCAM1 from known function and each receive full benchmark credit.The standard judge gives each 7 of 10, while the grounding judge gives 3 of 10.Source: I Case studies · p. 21, Table 13 · p. 22

Elimination and overruling the data

  • After a flip, Opus selects HES1 despite barely expressed evidence, while Sonnet rejects RHO’s cone expression as mis-annotated; both receive full credit.The paper notes that questioning surprising measurements can be good practice, but says it is applied selectively to contradictory evidence and rewarded by the benchmark.Source: I Case studies · pp. 21–22

Following the evidence, scored zero

  • On flipped Hi-C data, Opus’s loop-strength estimates exchange to the third decimal and its delta changes sign with ground truth, yet it scores zero on both versions.The estimates fall outside the grader’s tolerance, so the score matches that of an agent that ignored the swap.Source: I Case studies · p. 22
Loading 2610.04915v1…