Evaluating Sakana's AI Scientist: Bold Claims, Mixed Results, and a Promising Future?

• The AI Scientist aims to automate the research lifecycle from idea generation through experimentation, manuscript writing, and review.Source: §1 · pp. 2–3
• A comprehensive, independent assessment based on direct experimentation had not yet been conducted.Source: §1 · p. 3
• The study evaluates capabilities, limitations, and trajectory across idea generation, experimentation, manuscript production, and review.Source: §1 · p. 3
• The AI Scientist stands out from tools focused on individual tasks by promising open-source, end-to-end automation of scientific inquiry.Source: §1 · p. 3
• The evaluation finds that the system can automate significant research activities but does not yet fulfill its promises because of methodological, experimental, and literature-retrieval problems.Source: §1 · p. 3
• The authors characterize these limitations as technical hurdles rather than fundamental barriers, while noting risks to the reliability and generalizability of AI-generated knowledge.Source: §1 · p. 3

• The section surveys the AI Scientist’s features and architecture alongside case studies, critiques, and evaluation evidence.Source: §2 · p. 4

• Setup and usability results should be treated as lower bounds because the operator was a computer-science bachelor’s student, whereas typical users may have greater expertise.Source: §2.1 · p. 4
• The researchers installed the AI Scientist on two computers, finding setup relatively straightforward and requiring approximately five hours including minor troubleshooting.Source: §2.1 · p. 4
• Experiments used FunkSVD on a CPU rather than the deep-learning GPU pipeline for which the system was optimized.Source: §2.1 · p. 4
• The study used gpt-4o-2024-05-13 as the foundation model, although the system also supported Anthropic, DeepSeek Coder V2, and Llama 3.1 models.Source: §2.1 · p. 4

• The AI Scientist requires a user-defined template, which significantly limits its autonomy before experiments begin.Source: §2.2 · p. 4
• The experimental pipeline uses experiment.py for data and training and plot.py for visualization, with FunkSVD trained and evaluated on an 80/10/10 MovieLens-100k split using RMSE.Source: §2.2 · p. 5
• Users cannot directly provide arbitrary Python code because existing code must be adapted to the system’s required pipeline format.Source: §2.2 · p. 5
• The researchers supplied two seed ideas—adaptive learning rates for SGD and E-Fold Cross-Validation—even though both were acknowledged as not entirely novel.Source: §2.2 · p. 5
• The supplied interestingness, feasibility, and novelty values had no apparent impact on the AI Scientist’s processing.Source: §2.2 · p. 5

• The generated idea set included energy-efficient matrix-factorization and stochastic-gradient-descent variants, including pruning, quantization, early stopping, micro-batching, sparse-aware, and cluster-aware approaches.Source: §2.3 · p. 6
• The system queried Semantic Scholar for up to 10 results per query and iterated up to 10 times before assigning a novelty label.Source: §2.3 · p. 6
• All 10 generated ideas and both seed ideas were classified as novel, including established techniques such as micro-batching, adaptive learning rates, hybrid matrix factorization, and e-fold cross validation.Source: §2.3 · p. 6
• Its novelty judgments likely relied on keyword matching rather than deeper synthesis of prior research.Source: §2.3 · p. 6
• Many proposed ideas applied existing techniques such as pruning to matrix factorization, offering incremental rather than groundbreaking improvements.Source: §2.3 · p. 6

• The system depended on Aider to iterate on experiment.py and remained dependent on the provided framework and predefined templates.Source: §2.4 · p. 6
• 529 characters (+8%) were added in the first iteration, followed by 118, 83, 66, and 21 characters in later iterations.Source: §2.4 · p. 6, Table 1 · p. 7
• Five of twelve ideas failed to execute because of unresolved coding errors, often cycling through flawed code versions.Source: §2.4 · p. 6
• In the e-fold experiment, the system fixed e at 2 instead of testing {2, 3, 4, 5} and failed to rerun the k-fold baseline, making e-fold appear superior.Source: §2.4 · p. 6
• The AI Scientist could not critically assess its own results, so methodological flaws and logical inconsistencies could mislead researchers without human oversight.Source: §2.4 · p. 7

• The seven manuscripts ranged from six to eight pages, with six of seven containing tables or charts, but their quality would likely fail reputable peer review.Source: §2.5 · pp. 7–8
• The manuscripts had a median of five references, only five (14.7%) of 34 references dated from 2020 or later, and frequently cited foundational works instead of relevant literature.Source: §2.5 · p. 7
• Four manuscripts (57%) had missing or misplaced figures, incomplete sections such as “Conclusions Here,” duplicates, or repeated figures.Source: §2.5 · p. 7
• The results sections lacked confidence intervals and p-values and sometimes presented logically inconsistent claims, such as better RMSE alongside only a one-second training-time reduction and increased memory use.Source: §2.5 · pp. 7–8
• Four of seven manuscripts (57%) contained incorrect or hallucinated numerical results, including discrepancies in hyperparameters and performance metrics.Source: §2.5 · p. 7
• The output resembled a student rushing to complete an assignment: formally structured but careless in execution.Source: §2.5 · p. 8

• For all seven AI-generated manuscripts, the reviewer recommended rejection, commonly criticizing reliance on a single small MovieLens 100k dataset.Source: §2.6 · p. 8
• The reviewer identified some concerns about theoretical justification and bias but overlooked redundant text, formatting errors, missing sections, and flawed experimental results.Source: §2.6 · p. 8
• The reviews were structured and plausible enough that editors or chairs might not immediately recognize them as AI-generated, though they were less rigorous than strong human reviews.Source: §2.6 · p. 8
• On ten human-written manuscripts, the reviewer rejected 9 out of 10, including four accepted by humans, and accepted one rejected by OpenReview.Source: §2.6 · p. 8
• Its reviews lacked depth, often missing broader contributions and argumentation, and it struggled with context-dependent judgment despite following predefined templates.Source: §2.6 · pp. 8–9
• The evaluation raises whether AI peer review should mimic human reviewers or provide a systematic alternative, since human judgment alone may be an insufficient benchmark.Source: §2.6 · p. 9

Costs and Effort

• In the main experiments, the researchers generated 10 ideas and ran experiments for 7 of 12 total ideas.Source: §2.7 · p. 9
• $42 produced seven manuscripts, averaging $6 per manuscript, while the authors consider Sakana’s approximately $15 estimate realistic.The experiments were less complex than typical recommender-systems research or Sakana’s experiments.Source: §2.7 · p. 9
• Approximately 5 hours were required for setup and another 15 hours to implement the experimental template.Reimplementing similar pipelines may take less time with experience, whereas more complex experiments would increase the required time.Source: §2.7 · p. 9
• About 10 additional hours were spent brainstorming seed ideas and reviewing outputs.Source: §2.7 · p. 9
• $6 to $15 and 3.5 hours of human effort produced a research paper, compared with 20 to 40 hours for an undergraduate and 10 to 20 hours for an experienced researcher.The authors estimate the AI Scientist operates approximately 3 to 11 times faster than a human researcher at comparable work.Source: §2.7 · p. 9

Limitations

• The study used one dataset, MovieLens-100k, and one domain, Green Recommender Systems.Source: §2.8 · p. 9
• Fixed seed ideas and a single template further constrained the evaluation design.Source: §2.8 · p. 9
• These constraints supported a controlled assessment but may affect generalizability and reproducibility.The authors nevertheless consider the findings meaningful.Source: §2.8 · p. 9
• The reported cost analysis excludes electricity and other costs for the laptop or university cluster.Source: §2.8 · p. 9

3 Future Work and Recommendations

• The recommendations target researchers, conference organizers, journal editors, and institutions, primarily within Information Retrieval but also as broader academic best practices.They emphasize practical steps for integrating AI while preserving scientific standards.Source: §3 · p. 10

3.1 Researchers should familiarize themselves with research-assistive AI tools

• Researchers should actively use, monitor, and help shape research-assistive AI tools so their development aligns with rigorous scientific principles rather than industry interests.Current tools such as ChatGPT and GitHub Copilot already improve writing, coding, and brainstorming, while future systems may generate ideas, experiments, and manuscripts with minimal oversight.Source: §3.1 · p. 10

3.2 Updated Guidelines for Authors and Researchers on the Use of AI

• AI-generated research requires revised publication guidelines because existing policies address manuscript assistance but not autonomous research or complete manuscript generation.The paper recommends community policies governing AI-generated submissions, with editors, ethics committees, reviewers, and conference stakeholders maintaining academic integrity and quality control.Source: §3.2 · pp. 10–11

3.3 Addressing the Risks of AI-Generated Assignments and Homeworks

• AI-generated assignments threaten academic integrity because coherent outputs may evade detection, requiring universities to redesign assessments and clarify acceptable AI use.Recommended measures include iterative or personal work, oral and in-class assessments, project-based demonstrations, intermediate drafts, video explanations, interactive discussions, AI literacy, and instructor training.Source: §3.3 · p. 11

3.4 Benchmarking AI-Driven Research Agents in Information Retrieval

• Structured benchmarks should evaluate AI research agents across peer review, idea generation, experimentation, manuscript quality, literature review, and dataset selection.Suggested criteria include expert agreement, acceptance and citation impact, reproducibility, methodological flaws, empirical accuracy, hallucinated citations, retrieval relevance, citation analysis, and bias detection.Source: §3.4 · p. 11

3.5 Release of AI Research Logs for Transparency

• AI-generated studies should release detailed execution logs and versioned artifacts so researchers can trace decisions, identify errors, compare approaches, and improve reproducibility.Logs should include prompts, code modifications, abandoned runs, configurations, decision rationales, and containerized computational environments.Source: §3.5 · p. 12

3.6 Pilot Projects and Competitions

• Pilot projects can extend existing initiatives on generative AI in peer review and manuscript submission, while targeted competitions can assess AI-generated research.The paper presents competition-based evaluation as a promising way to investigate AI’s impact on scientific research.Source: §3.6 · p. 12
• Blind peer-reviewed competitions comparing AI-generated papers with human-authored research could provide fair assessments, although current tools remain limited and viability may emerge within years.Early community planning is recommended to establish evaluation frameworks and ethical considerations beforehand.Source: §3.6 · p. 12

3.7 AI-Generated Content Tagging: Research Attribution Markup Language (RAML)

• RAML would standardize disclosure of AI-generated content in academic manuscripts, tracking contribution levels, model metadata, drafts, change logs, and prompts.The proposal calls for adoption by publishers, preprint servers, journals, conferences, and DOI systems to support transparency, accountability, and reproducibility.Source: §3.7 · pp. 12–13

3.8 Research Process Markup Language (RPML)

• RPML would record experiment setups, code and dataset versions, containers, hyperparameters, and citation contexts to address undocumented modifications and improve reproducibility.The proposed JSON- or XML-based schema is intended to provide full traceability across the research workflow.Source: §3.8 · p. 13

3.9 Standardized Dataset Annotation and Integration

• Standardized dataset annotations should provide domain applicability, quality scores, versioning, benchmark results, and use cases so the AI Scientist can select and integrate suitable datasets automatically.The proposal targets providers including Hugging Face, Kaggle, and the UCI Machine Learning Repository.Source: §3.9 · p. 13

3.10 Code Repositories Integration

• Direct integrations with PapersWithCode, GitHub, and Code Ocean would let the AI Scientist retrieve, run, and compare current baseline implementations and performance metrics.The recommendation addresses its limited access to recent state-of-the-art code and the resulting difficulty benchmarking innovations against best practices.Source: §3.10 · p. 13

3.11 Participatory Strategic Retreat

• A multi-day workshop or strategic retreat involving leading IR experts and young researchers could turn initial discussion into white papers and coordinated research efforts.The authors present their ideas as an initial stimulus for discussion, with Dagstuhl or NII Shonan-style meetings proposed as a suitable setting.Source: §3.11 · p. 14

Overall assessment

• The AI Scientist is a significant step toward automating scientific research, but its current capabilities remain far from its ambitious promises.Source: §4 · p. 14
• The system can generate research ideas but only sometimes conducts experiments successfully, and many ideas are not truly novel.It heavily depends on user input and struggles with methodological soundness.Source: §4 · p. 14
• In its present form, the AI Scientist is at best an advanced research assistant requiring substantial supervision rather than an independent scientific agent.Source: §4 · p. 14

Future use

• The paper argues that AI could contribute to scientific critique through adversarial feedback loops, while users should critically reassess assumptions and alternative perspectives.Source: §4 · pp. 14–15

Peer review

• AI-generated reviews often provide surface-level, well-formatted critiques while missing deeper methodological flaws, increasing the importance of second-level reviewers.Such reviewers must look beyond surface analyses to judge whether work holds promise.Source: §4 · p. 14

Replication and validation

• Replication and validation are promising applications because AI can support proof of work, verifiable procedures, metadata creation, and provenance for scientific claims.The proposed mechanisms include open-weight and open-data models, structured deposits, and version-controlled repositories.Source: §4 · p. 14

Human and ethical effects

• AI-generated research may improve productivity, but one cited study found that 82% of scientists reported lower job satisfaction after AI-assisted workflows.Source: §4 · pp. 14–15

Publishing risks

• Mass AI-generated submissions could overwhelm venues with seemingly novel, low-value papers, motivating authorship verification, disclosure policies, and detection methods.The risk is amplified because modern AI-generated papers can be difficult to distinguish from human writing.Source: §4 · p. 15