PaperlayerView exact revision on arXiv

Source-linked AI summary

Agentic AI Scientists Are Not Built For Autonomous Scientific Discovery

Harshit Bisht, Vinay Kumar, Kevin Maik Jablonka, Mausam, N. M. Anoop Krishnan

arXiv:2605.08956v1cs.AI

TL;DR

Agentic AI systems are increasingly presented as capable of autonomous scientific discovery, but the paper argues that their training and deployment leave key challenges unresolved. It examines these challenges and tests hypothesis generation across frontier models, finding high inter-model similarity in open-ended outputs.

  • Problem

    The paper asks whether agentic systems can conduct natural science autonomously, as growing system claims and benchmark performance suggest, given the potential costs of mismatch for evaluation and scientific effort.

    Source: §1 · p. 2

  • Method

    The paper examines challenges in current training and deployment and tests hypothesis convergence through interpretive and open-ended tasks with frontier models from independent providers.

    Source: §5 · p. 5, Empirical evidence: the hypothesis… · p. 5

  • Results

    Inter-model similarities remained high in open-ended hypothesis generation, leading the authors to conclude that querying multiple AI systems effectively samples from a single model.

    Source: Empirical evidence: the hypothesis… · p. 5

  • Takeaways & Limitations

    The authors argue that addressing the autonomy gap requires changes to training and deployment design, not just greater scale and scaffolding.

    Source: §7 · p. 7, §9 · p. 10

    The relevant comparison is collective science with corrective mechanisms; given model convergence, querying additional AI systems is unlikely to provide the same corrective pressure.

    Source: §7 · p. 8

Abstract

from arXiv · show

A growing body of work pursues AI scientists capable of end-to-end autonomous scientific discovery. This position paper argues that although they already function as co-scientists, agentic AI scientists are not built for autonomous scientific discovery. We identify the following challenges in building and deploying autonomous AI scientists: (1) Problem selection is influenced by the McNamara fallacy; (2) Agents are built on large language models (LLMs) whose training corpora omit tacit procedural and failure knowledge of laboratory practice; (3) Preference optimisation during post-training compresses output diversity toward consensus; and (4) Most scientific benchmarks measure single-turn prediction accuracy and lack feedback from physical experiments back to the computational model. These challenges are not just questions of scale and scaffolding; they require revisiting fundamental design choices. To build truly autonomous AI scientists, we recommend the use of scientific simulations as verifiers for training, the design of persistent world models that represent the shifting objectives governing real investigations, the establishment of a centralized preregistration repository for all AI-generated hypotheses, and application driven by scientific need rather than tool affordance.

1 Introduction

Scientific reasoning closes the loop from conjecture and prediction to physical verification. Although AI contributes to scientific work, the paper argues that current agentic systems are not built for autonomous discovery and identifies challenges in their training and deployment.

1 Introduction

  • Science is characterized by testing conjectures against nature through experiment or experience, with disagreement showing the guess is wrong.The passage presents physical verification as what distinguishes scientific reasoning from other structured reasoning.Source: §1 · p. 1
  • AI systems have accelerated experimental-data processing, advanced automated tool-use and laboratories, uncovered structure in biological and chemical spaces, and shortened the path from hypothesis to computational candidate.Examples span drug discovery and materials design.Source: §1 · pp. 1–2
  • Agentic systems increasingly claim to automate problem selection, hypothesis development, experimentation, and communication without meaningful human involvement.Such claims matter because, if correct, they would transform research organization; if not, the mismatch carries costs for evaluation and scientific effort.Source: §1 · p. 2
  • The paper argues that current agentic systems are not built for autonomous science because of shortcomings in training and deployment, while remaining useful as co-scientists that extend human judgment.It identifies challenges in problem selection, training-corpus knowledge, output diversity, and scientific benchmarks.Source: §1 · p. 2

2 Challenges in autonomous discovery

AI systems have advanced scientific tasks, but humans still set the problem, supply the data, and define success criteria. The paper uses solid-state battery electrolytes to illustrate the complexity facing autonomous discovery.

2 Challenges in autonomous discovery

  • For cited examples of AI-generated candidates and reports, human researchers still specified the problem, supplied the dataset, and defined success criteria.The authors argue this division of labor reflects design choices, not merely a temporary limitation fixable by scaling systems.Source: §2 · p. 2
  • Solid-state electrolyte discovery lacks a ground-truth objective because ion-transport mechanisms remain incompletely understood, and progress spans quantum calculations, molecular dynamics, and device-scale models.Laboratory realization adds synthesizability, processability, and manufacturing challenges.Source: §2 · pp. 2–3
  • Table 1 contrasts desired and current behavior across challenge themes, using solid-state electrolyte discovery as its running case study.The themes are examined in Sections 3–6.Source: Table 1 · p. 3

3 Problem selection: McNamara fallacy in AI-driven science

Scientific problem selection requires judgment about significance, tractability, need, resources, and human curiosity. The paper argues that AI-driven optimization can favor measurable problems and tool-led agendas over scientific need.

3 Problem selection: McNamara fallacy in AI-driven science

  • Problem selection requires judgment integrating theoretical significance, practical tractability, community need, resource constraints, and the human desire to understand the universe.The authors distinguish this judgment from problem solving because it resists formal specification.Source: §3 · p. 3
  • Across 41.3 million papers, AI-augmented scientists published 3.02 times more papers and received 4.84 times more citations, while AI adoption shrank studied topic volume by 4.63% and scientist engagement by 22%.Hao et al. [2026] report that AI accelerates established fields while hard, data-sparse problems go untouched.Source: §3 · p. 3
  • AI's reliance on numerical optimization signals and computationally tractable data can steer research toward problems that are easy to quantify, a form of the McNamara fallacy.The fallacy describes measuring what is easy, disregarding what cannot be quantified, and treating the unmeasured as unimportant or nonexistent [Yankelovich, 1972, O’Mahony, 2017].Source: §3 · p. 3
  • In technology-push research, tools define the agenda; in demand-pull research, scientific needs guide tool development, while AI deployment currently seems almost entirely technology-push.MatterGen can generate materials for specified property constraints, but scientists—not the system—choose which battery bottleneck to target and what can be validated.Source: §3 · pp. 3–4

4 Data bias: gaps in the training corpus

Published scientific literature filters knowledge toward conclusions, positive results, and prevailing theories. The paper argues that missing procedural and failure knowledge, alongside consensus-shaped training data, constrains what models can learn about discovery.

4 Data bias: gaps in the training corpus

  • Scientific publishing filters for conclusions over process, positive over negative results, and ideas consistent with prevailing theory, so models inherit the filtered record.The authors argue that these omissions cannot be recovered by training on more of the same data.Source: §4 · p. 4
  • Laboratory tacit knowledge includes reliable synthesis conditions, supplier-dependent reagent behavior, undocumented protocol adjustments, and measurement artifacts that published accounts may omit.Polanyi and Sen [1966] describe this knowledge as irreducibly non-propositional; experienced researchers may recognize problematic results without fully articulating why.Source: §4 · p. 4
  • Publication bias removes negative results, and models trained on final papers lack abandoned research paths and may fail to detect anomalies that should prompt abandoning a hypothesis.A model trained on unpublished failed syntheses outperformed models trained on published literature alone for predicting vanadium selenite synthesis conditions [Raccuglia et al., 2016]; simulated-discovery LLMs also failed to detect anomalies [Ding and Li, 2025].Source: §4 · p. 4
  • Training on a fixed publication corpus makes ideas that directly contradict field consensus less likely, although cross-domain synthesis remains possible.The paper points to Shechtman's five-fold diffraction finding, initially contrary to accepted crystallographic theory, as an example of a discovery at odds with consensus [Shechtman, 2013].Source: §4 · p. 4

5 Diversity: preference optimisation compresses the hypothesis space

The paper argues that preference optimisation narrows the hypothesis space toward published consensus, and reports that frontier models from independent providers remain semantically similar even when asked for novel hypotheses.

5 Diversity: preference optimisation compresses the hypothesis space

  • RLHF and DPO shift outputs toward annotator-familiar responses, making consensus-shaped preferences a poor objective for scientific hypothesis generation.The annotators' preferences reflect the same published consensus encoded in pretraining.Source: §5 · p. 5
  • Distillation can pass a teacher's systematic tendencies to students, while successive training on AI-generated text can further contract output diversity.The passage describes this as compounding across generations as AI-assisted literature enters future training corpora.Source: §5 · p. 5
  • For solid-state electrolytes, the authors expect these biases to concentrate generated candidates in familiar LLZO and NASICON families rather than hypotheses that challenge the status quo.Both the literature represented in model priors and annotator familiarity are described as reinforcing this concentration.Source: §5 · p. 5

Empirical evidence: the hypothesis hivemind

  • Inter-provider output similarity stayed high even when models were prompted for novel hypotheses, making multiple queried systems effectively one source of hypothesis directions.Figure 1 compares average cosine similarities for interpretive hypothesis recovery and open-ended generation; the authors characterize the effective epistemic sample size as close to one.Source: Empirical evidence: the hypothesis… · p. 5, Figure 1 · p. 6

6 Evaluation, inference, and physical validation

The section argues that scientific capability must be judged against physical reality, while current benchmarks and agentic systems miss important features of investigations. It describes gaps in benchmark validity, counterfactual reasoning, shifting objectives, and feedback from experiments to computational models.

6.1 What benchmarks measure

  • Benchmark scores may not reflect the intended capability: chest X-ray VQA models scored highly without image input, and leakage corrections collapsed complex-model advantages over classical baselines.The cited survey covered 294 papers across 17 fields and corrected for eight common leakage types.Source: §6.1 · p. 6
  • Most benchmarks reward single-turn prediction, not maintaining an investigative thread as evidence changes questions, candidate lists, strategies, and property weights.Across more than 25,000 agent runs in eight domains, evidence was ignored in 68% of traces and refutation-driven belief revision occurred in 26%.Source: §6.1 · p. 6

6.2 Reasoning across an investigation

  • Current reasoning models with agentic harnesses do not exhibit the trained counterfactual habit of asking what evidence would look like if a hypothesis were wrong.The passage distinguishes generating text from a prompted counterfactual premise from cultivating counterfactual interrogation as a default.Source: §6.2 · pp. 6–7
  • Scientific investigations have non-stationary objectives, so a fixed scalar reward may discard anomalous results instead of treating them as information for revising the reward.The passage identifies reward specification as difficult because objectives change in response to information gathered during an investigation.Source: §6.2 · p. 7
  • In the solid-state electrolyte example, lower experimental performance than predicted could reveal simulation assumptions or omitted physical mechanisms, but a fixed simulation-agreement reward might discard the candidate.The passage lists interatomic potentials, boundary conditions, grain boundaries, secondary phases, and dendrite penetration as considerations.Source: §6.2 · p. 7

6.3 From computation to physical reality

  • Of 2.2 million GNoME predicted structures, 736 had been independently experimentally realized at publication; evaluation also considered novelty, synthesizability, and practical utility [Cheetham and Seshadri, 2024].The authors use the example to highlight a broader pattern of generative materials models omitting feasibility constraints.Source: §6.3 · p. 7
  • Closing the gap between computational predictions and physical experiments requires relational knowledge across discovery steps and a feedback loop that current agentic systems lack.The passage connects synthesis feasibility to tacit laboratory knowledge and measurement interpretation to expertise in distinguishing bulk transport, grain-boundary resistance, and electrode artefacts.Source: §6.3 · p. 7

7 Alternative views

The authors argue that more scale and scaffolding will not resolve the upstream limitations they identify. They favor the co-scientist model because key judgments remain human-supplied and meaningful human-AI collaboration outperformed full automation in a controlled study.

7 Alternative views

  • More context, retrieval, and tool-use capability will not resolve failures rooted in measurable-problem bias, missing tacit and failure knowledge, or consensus-oriented preference optimisation.The authors argue these gaps stem from publishing norms and training objectives, not merely limited scale or scaffolding.Source: §7 · pp. 7–8
  • AlphaFold and GNoME relied on human researchers to specify objectives, define validation criteria, and interpret results, so their successes do not establish autonomy.The cited controlled study found the highest-quality scientific outputs came from meaningful human-AI collaboration, with quality declining under full automation [Bianchi et al., 2026].Source: §7 · p. 8
  • The comparison is between AI and science's collective correction mechanisms, since individual scientists are fallible and querying similar AI systems may not provide equivalent corrective pressure.The authors name peer review, replication, adversarial collaboration, and reputational accountability as mechanisms developed across the scientific enterprise.Source: §7 · p. 8

8 Toward autonomous AI scientists

The paper proposes changes to how AI scientists are judged, trained, and supported, arguing that autonomous discovery needs more than better tools or larger models. Its recommendations include public records of scientific judgment, preregistration, richer training data, diversity-preserving post-training, simulation-based verification, and persistent world models.

8 Toward autonomous AI scientists

  • Public reviews of papers and proposals can provide training data on scientific judgment, including assessments of ideas on merit and impact without experiment results.The authors note that consent, privacy, and copyright concerns must be addressed before using such data.Source: §8 · p. 8
  • A structured, machine-readable repository should register AI-generated hypotheses before simulation or automated laboratory tests, preserving failed attempts as training data and supporting diversity and anti-redundancy signals.The authors argue that automated registration through an API makes circumvention a deliberate act; they acknowledge challenges in achieving consensus and compliance around a centralized schema.Source: §8 · pp. 8–9
  • Training data should include annotated video of laboratory procedures because published literature omits tacit practice and unpublished dead ends.Generating such records is expensive and difficult to standardize.Source: §8 · p. 9
  • Training failure knowledge also requires publishing incentives that support negative results and deposition of failed experiments.The authors call for community norms that treat documented failure as a scientific contribution rather than a career liability, noting that the AI community cannot generate this infrastructure alone.Source: §8 · p. 9
  • Post-training can preserve diverse hypotheses by matching the full distribution of annotator responses rather than optimizing toward consensus [Chakraborty et al., 2024, Yao et al., 2024].The authors suggest combining these techniques with a large number of expert annotators.Source: §8 · p. 9
  • Scientific simulators can act as training oracles that provide tacit and failure knowledge, and simulator queries can be diversified indefinitely without a data ceiling.Synthetic simulator data has improved performance on International Physics Olympiad problems with zero-shot transfer to real benchmarks [Prabhudesai et al., 2026].Source: §8 · p. 9
  • Autonomous investigations need persistent, mutable representations of epistemic state that carry hypotheses and observations across steps and sessions.One proposed implementation is a Bayesian network updated with experimental results, with new hypothesis nodes added in response to anomalies; the model should support uncertainty propagation and experiment selection by expected information gain.Source: §8 · p. 9

9 Conclusion

The conclusion argues that autonomous scientific discovery faces fundamental challenges that scale and scaffolding alone will not resolve. It summarizes those challenges and proposes simulators, persistent world models, and preregistration repositories as possible ways forward.

9 Conclusion

  • The barriers to autonomous discovery span problem selection, missing tacit and failure knowledge, compressed hypothesis diversity, and benchmarks that omit multi-step reasoning and physical feedback.The authors argue these challenges will not be closed by scale and scaffolding alone.Source: §9 · pp. 9–10
  • In the solid-state battery electrolyte example, target choice relies on human judgment, hypotheses may cluster around LLZO and NASICON, anomalous findings do not adapt the research process, and translating candidates to devices remains difficult.The authors use this example to ground their discussion of autonomous scientific discovery.Source: §9 · p. 10
  • Potential responses include simulator-based training verification, persistent world models that carry epistemic state, and preregistration repositories.Source: §9 · p. 10

A Experiment: hypothesis hivemind

The hypothesis hivemind experiment tested whether models from different providers converge on both hypothesis recovery and open-ended hypothesis generation, using outputs from papers in the 2025 NeurIPS AI4Mat track.

A Experiment: hypothesis hivemind

  • The experiment used 50 publications from the 2025 NeurIPS AI4Mat track for two tasks: recovering underlying hypotheses and proposing novel extensions.Task 1 used experiment summaries; Task 2 used full paper text.Source: A Experiment: hypothesis hivemind · p. 16
  • For each task, researchers drew 10 independent samples per model and compared cosine similarities between intra-family and inter-family output embeddings.Outputs were embedded with text-embedding-3-small; the prompts appear in Box A.Source: A Experiment: hypothesis hivemind · p. 16
  • Models converged semantically on both hypothesis recovery and novel-hypothesis generation, while inter-model similarities were not significantly lower than intra-model similarities.Figure 3’s same-paper versus different-paper output distributions were used to check that this reflected semantic similarity rather than embedding-model degeneracy.Source: Figure 2 · p. 16, A Experiment: hypothesis hivemind · p. 16, Figure 3 · p. 17
  • Task 1 asked models to infer the testable underlying hypothesis from experimental methods, while Task 2 asked for a specific, testable, falsifiable hypothesis extending a paper’s findings.Task 1 served as a convergence baseline; Task 2 was expected to yield more diverse outputs.Source: A Experiment: hypothesis hivemind · pp. 16–17

A.1 Dataset

The dataset section lists papers selected from accepted works at the 2025 NeurIPS AI4Mat workshop.

A.1 Dataset

  • The dataset consists of selected papers from the accepted works at the 2025 NeurIPS AI4Mat workshop.Examples in the list include work on polymer surface adhesion, multimodal molecular representation, and converging representations of matter.Source: §A.1 · p. 18
  • Other listed titles address preference learning from physics-based feedback, pharmacophore-guided molecular design, and cross-modal material-property prediction.These titles appear among the papers considered for the experiment.Source: §A.1 · p. 18
  • The list also includes work on machine-learning interatomic potentials, dynamic materials-discovery benchmarks, and closed-loop reactor-yield optimization.These are examples of the selected publication titles.Source: §A.1 · pp. 18–19
Loading 2605.08956v1…