Why LLMs Aren't Scientists Yet: Lessons from Four Autonomous Research Attempts

• Six agents divide the workflow into idea generation, hypothesis generation, experiment planning and evaluation, revision, and paper outlining.Claude Code then developed the outline into a full paper.Source: §1 · pp. 2–3
• Paper writing proceeded from a detailed outline to sequential section drafting and two human–Claude Code editing rounds, because human quality control remained necessary.The edits addressed continuity and flow, then tempered overoptimistic claims; initial drafts could be mechanically written or lack narrative nuance.Source: §1 · p. 3
• Each research idea used a dedicated repository to store artifacts from the initial idea through code, outputs, and the submission template.Source: §1 · p. 4
• Agents received the repository state in their prompts, leaving them to decide which context files to inspect rather than heavily engineering context.The authors also viewed that file-selection decision as a relevant information-retrieval skill for researchers.Source: §1 · p. 4
• Idea review used four zero-shot prompts before hypothesis generation and experimentation, with web search available only to the custom evaluation prompt.Source: §1 · p. 4

• The idea-generation process explored World Models, Multi-Agent Reinforcement Learning, and AI Safety and Alignment, then narrowed to four ideas for experimentation.The authors began from top-tier venue papers and consulted parent-paper authors on candidate quality and feasibility.Source: §2 · pp. 4–5
• Three of the four candidate ideas were pruned during early execution; only AI Alignment and Safety idea AS-1 was completed.Table 1 summarizes the candidates’ final status and primary failure modes.Source: §2 · p. 5, Table 1 · p. 5
• Reviewers recognized the paper’s empirical contribution and negative results while noting limited analysis and comparisons; the decision was majorly borderline accept.One of three AI reviewers scored it 6 or strong accept.Source: §2 · p. 6

3 Observed Failure Modes and Mitigation

• Six recurring failure modes emerged across research attempts, revealing systematic limitations in current LLM capabilities for autonomous research.The subsections pair examples of these modes with mitigation strategies.Source: §3 · p. 6

3.1 Bias on Training Data

• Models overrode explicit instructions with memorized defaults across infrastructure setup, library selection, and dataset handling.Even after changes to avoid early anchoring and provide specific execution-stage instructions, the execution model regressed to training-data versions when errors arose.Source: §3.1 · pp. 7–8

3.2 Implementation Drift

• When technical complexity or execution barriers arose, models simplified implementations and abandoned core innovations rather than addressing root causes.Timeouts in long-running training loops and errors in multi-step tasks could trigger simpler or sample-mode implementations, causing downstream failures; the authors used hypothesis portfolios and separate code verification before execution as mitigations.Source: §3.2 · pp. 8–9

3.3 Memory and Context Issues

• As context artifacts accumulated, models lost track of prior decisions and configurations, producing redundant work and inconsistent experiments.This affected hyperparameter management, code references, and paper outlining; session logs helped with context management, but the growing number of generated files created a further management problem.Source: §3.3 · pp. 9–10

3.4 Overexcitement and Eureka Instinct

• Models reported success despite clear failures and overstated research contributions, particularly during outlining, revision, and output evaluation.Evaluation based on report files rather than raw experimental logs contributed to ignoring degeneracies; paper writing also inflated novelty and scope.Source: §3.4 · pp. 10–11

3.5 Lack of Sufficient Domain Intelligence

• Models lacked tacit domain knowledge for predicting outputs, identifying failure modes, selecting suitable baselines, and handling subdomain-specific processes.Plans also missed conceptual hurdles in complex architectures, while a baseline 95% below established benchmarks made comparative analysis scientifically meaningless.Source: §3.5 · pp. 11–12

3.6 Lack of Scientific Taste

• Models missed fundamental flaws in experimental design and statistical methodology, including a one-seed WM-1 run and computationally infeasible plans.Other examples included WM-2's offline-training plan despite Dreamer's need for online learning, and degenerate AS-1 output that required manual intervention to flag.Source: §3.6 · pp. 12–13

4 Design Takeaways for AI Scientist Systems

• The authors identify four design principles for robust autonomous research systems from their analysis of failure modes.The principles address challenges across research domains and implementation attempts.Source: §4 · p. 13
• The principles address key challenges across research domains.Source: §4 · p. 13
• The principles address challenges observed across implementation attempts.Source: §4 · p. 13

1. Start Abstract, Ground Later

• Introduce domain expertise and technical details gradually, making prompts and generations more specific as research progresses.Source: §1 · p. 13
• Keep ideation high-level to avoid premature implementation anchoring, then add specificity during implementation.The authors connect this sequencing to preserving novelty and avoiding slightly reworded reproductions of existing research.Source: §1 · p. 13
• Avoid specifying datasets, metrics, and field-specific practices during planning, since early detail can steer models toward older training-data methods.Such details remain important for execution, when current solutions must be identified or reasoned about.Source: §1 · p. 13

2. Verify Everything

• Verify every relevant stage, from ideas and hypotheses through code and results, to help prevent conceptual or implementation errors from cascading.Source: §2 · p. 13
• Verification should assess both process versus outcome and correctness versus scientific contribution.The latter includes reproducibility across planning, ideation, and output review.Source: §2 · p. 13
• Evaluate experimental results from raw data rather than LLM interpretations, which may read signal into errors or overstate weak results.Source: §2 · p. 13
• If LLM evaluators are used, direct them to inspect raw logs, statistical metrics, and original outputs rather than summary reports.Source: §2 · p. 14
• Verification can support human oversight by exposing long-running tasks and helping remove visualizations that are not central to the paper's story.The authors used plan-stage test instructions to improve observability during execution.Source: §2 · p. 14

3. Plan For Failure and Recovery

• Scientific discovery is long-duration work prone to error accumulation, so many researcher micro-decisions must be prespecified for autonomous LLMs.Source: §3 · p. 14
• Multi-turn task design, external verification, critique, and reflection help plans include details that can reduce later drift or errors.The authors contrast this with zero-shot generation.Source: §3 · p. 14
• Modular coding tasks and separating code generation from execution prevent error cascading and allow verification hooks.The authors also cite Jupyter notebooks’ cell-level limits on errors.Source: §3 · p. 14
• System-wide execution instructions call for checkpoints, tests at multiple scales, detailed metric logging, and safeguards against context and memory errors.Source: §3 · p. 14
• Separating visualization generation from paper integration enabled human review and removal of unhelpful charts before manuscript inclusion.Source: §3 · p. 14

4. Log Everything

• Comprehensive logging of LLM responses, agent-created files, and experiment metrics supports long-duration execution and later human or LLM review.Source: §4 · p. 15
• The system used write tools and session logs for context artifacts, logged domain-specific primary and secondary metrics during planning, and included process artifacts in paper writing.The authors say this logging enabled their two-fold evaluation framework.Source: §4 · p. 15
• Figure 7 presents a comprehensive prompt template for maintaining detailed research execution records across agent sessions.Source: §4 · p. 15, Figure 7 · p. 15

5 Discussion and Broader Implications

• Human intervention remained necessary at idea review, paper writing, and experiment meta-prompting, and domain experts will likely continue to nudge, course-correct, and verify.The authors note that Sakana’s approach required researchers to select 3 ideas from 40 [1], while Google’s co-scientist requires scientist-specified goals [4].Source: §5 · p. 15
• OpenAI’s hiring of academics and examples of GPT-5 research collaborators support a focus on human-LLM collaboration; Gowers notes LLMs are not yet likely to originate the main idea for a difficult problem [8].Source: §5 · pp. 15–16
• A reported human-LLM collaboration compressed a six-month workflow into six hours, a “factor of 1000” acceleration [8].The authors connect such collaboration to kick-starting the scientific-method data loop.Source: §5 · p. 16
• Scientific-workflow training data lacks expert literature-review trajectories and records of failed attempts, including why obvious strategies do not work [8].Published citation lists show which papers made the cut, but not which references to ignore.Source: §5 · p. 16
• Current long-horizon reliability falls short of science’s weeks-to-months context demands; as of November 2025, GPT-5.1-Codex-Max had a roughly 2-hour-40-minute horizon at 50% success.The authors anticipate smaller modules producing data for later specialization.Source: §5 · p. 16
• ScientistBench’s 22 papers and 28 mostly highly guided tasks illustrate limited evaluation data, while memorized training data raises plagiarism concerns without robust evaluation [12] [15].Source: §5 · pp. 16–17
• Specialized agent systems include domain-tool agents matching elite IPHO 2025 gold medalists and multi-agent virtual labs for accelerated discovery [16].Source: §5 · p. 17
• The authors envision decomposing discovery into specialized capabilities learnable with clear rubrics and reward functions, alongside open science that shares results and enabling materials.They describe larger labs collaborating with domain specialists and smaller labs developing benchmarks and domain-specific models.Source: §5 · p. 17

6 Limitations and Future Work

• The study covers computational ML only, with four research ideas across three subdomains and one implementation run per idea, limiting generalizability and statistical conclusions.It excludes physical sciences requiring experimental infrastructure and lacks controlled architectural ablations, making it difficult to isolate changes behind the single success and three failures.Source: §6 · p. 17
• Qualitative observation rather than systematic measurement limits estimates of failure modes’ importance and frequency across research contexts.Source: §6 · p. 17
• Releasing prompts and selected outputs without the complete system architecture limits reproducibility and others’ ability to build on the approach.Source: §6 · p. 17
• Planned work includes controlled tests, hundreds of ideas across subdomains, open-source architectures, collaborative workflow-data collection, and quantitative research-quality metrics.The authors also plan to optimize human-LLM collaboration patterns.Source: §6 · p. 18

A Deep Dives: Research Ideas and Anecdotes

• Each subsection combines the Idea Generation Agent’s complete specification, feedback from original-paper authors, and implementation anecdotes.The anecdotes document failure modes and system responses during autonomous execution.Source: A Deep Dives: Research Ideas and… · p. 21
• Block quotes reproduce agent outputs or excerpts from execution logs as examples of system behavior.Source: A Deep Dives: Research Ideas and… · p. 21
• The authors thank seed-paper authors for feedback and accept responsibility for remaining errors or misinterpretations.Source: A Deep Dives: Research Ideas and… · p. 21

A.1 MARL-1: Zero-shot Coordination in Multi-Agent RL

• MARL-1’s single-file MVP skeleton prioritized clarity over Hanabi performance, while its implementation simplified the planned work under execution pressure.The plan used a minimum-viable hypothesis and one complete Python file; hard computation was guarded behind a fast flag.Source: §A.1 · p. 22
• The MARL-1 implementation repeatedly reverted to the outdated DeepMind Hanabi environment despite instructions to use an alternative.The authors identify this as training-data bias: memorized patterns overrode current information.Source: §A.1 · p. 23

A.2 WM-1: Differentiable Planning in Stochastic World Models

• The WM-1 planner shifted from differentiable tree search to actor-critic under execution constraints, and evaluation flagged the departure from the proposed method.The evaluator also cautioned that “Stochastic-DTS” misnamed the implemented algorithm.Source: §A.2 · p. 25
• 0.0 catastrophe rates for both S-DTS and its baseline left WM-1’s hypothesis untestable on FrozenLake; the experiment also used only one seed.The evaluation report judged the test insufficient to assess the hypothesis meaningfully.Source: §A.2 · pp. 25–26

A.3 WM-2: Replacing Pixel Reconstruction with Perceptual Loss in World Models

• WM-2 assumed static offline data, although Dreamer requires online learning; this undermined action–reward relationships and the baseline’s validity.The seed-paper authors also questioned whether replacing pixel reconstruction would generalize beyond domains such as MineRL.Source: §A.3 · pp. 26–27
• 95% below established benchmarks: WM-2’s baseline made comparison scientifically invalid, after implementation errors obscured the experiment’s signal.A .detach() call severed perceptual-loss gradients to the decoder; dummy rewards also left the critic without reward supervision.Source: §A.3 · pp. 27–28

A.4 AS-1: Semantic Entropy as Signal for Jailbreak Prompts

• After semantic entropy failed to detect jailbreaks effectively, the Revision Agent redirected AS-1 toward demonstrating and investigating that failure.The initial paper-writing process emphasized positives despite degenerate outputs and statistical insignificance, requiring human oversight for transparent reporting.Source: §A.4 · pp. 29–30
• 85–98% false negative rates: semantic entropy failed across two model families and benchmarks, with templated refusals accounting for 73–97% of false negatives.The authors report that stronger alignment can make outputs more predictable, confounding this diversity-based detector.Source: §A.4 · p. 30