PaperlayerView exact revision on arXiv

Source-linked AI summary

Can AI agents conduct open-ended AI research? Early evidence from two case studies

Peter Kirgis, Sayash Kapoor, Andrew Schwartz, Stephan Rabanser, David Africa, Konstantinos Voudouris, Viet Nguyen, Toby Pilditch, Magda Dubois, Harry Coppock, Cozmin Ududec, Nitya Nadgir, Matilda Orona, Tilman Bayer, Derrick Chan-Sew, Yue Ling, Abhishek Shetty, Helen Toner, Gillian Hadfield, Seth Lazar, Steve Newman, Shoshannah Tekofsky, Rishi Bommasani, Arvind Narayanan

arXiv:2607.27191v2cs.AIcs.CYcs.LG

TL;DR

Evidence on whether AI agents can conduct open-ended AI research remains thin. Using shadow evaluations, the paper finds agents handled engineering but failed to produce original research at a top ML conference’s caliber.

  • Problem

    Evidence remains thin on whether AI agents can solve open-ended AI research questions, despite the significance of automating AI research.

  • Method

    Shadow evaluations give frontier agents unpublished research questions and have the original authors grade their outputs as conference submissions.

  • Results

    Agents solved necessary engineering problems but failed to produce original research at the caliber of a top machine-learning conference.

  • Takeaways & Limitations

    Today’s frontier models provide early evidence of engineering capability but cannot yet solve weeks-long, open-ended AI research questions.

  • Takeaways & Limitations

    Open-world shadow evaluations trade off objectivity for a richer set of research tasks, so the authors do not consider them unbiased.

Abstract

from arXiv · show

Forecasts of explosive AI progress hinge on AI agents automating AI research. But evidence on whether agents can carry out open-ended AI research is thin. Current evaluations either test agents on narrow, verifiable tasks, which excludes open-ended research, or submit AI-generated papers to blind peer review, which is overstretched, stochastic, and suffers from poor review quality. We introduce a third way to measure progress towards AI R\&D automation. An agent takes on the central, open-ended research question of a high-quality unpublished paper, and the paper's original authors grade its output. We call these shadow evaluations. We ran shadow evaluations on two unpublished NeurIPS 2026 submissions, giving frontier agents six days and thousands of dollars of compute. The agents completed all of the engineering without human help, yet could not make substantial progress towards answering the research questions. As a result, both papers were unambiguously rejected by the authors. We identify five recurring failure modes: poor judgment about the bar for publishable research, uncreative responses to shortcomings in the research design, ineffective backtracking from dead ends, poor resource awareness, and instruction drift. A robustness check with a second model and scaffold reproduced these failures. We release the expert reviews, survey responses, agent repositories, and logs. Our results provide early evidence that today's agents can do the engineering of AI research, but struggle with critical parts of the research lifecycle.

1 Introduction

The paper introduces shadow evaluations, in which frontier agents tackle unpublished research questions and are graded by the original authors, as a complement to narrow benchmarks and blind peer review. Across two six-day evaluations, agents completed the engineering but failed to produce original research at a top-ML-conference standard, revealing recurring weaknesses in judgment, resource use, feedback response, backtracking, and instruction-following.

  • Method: Shadow evaluations task a well-resourced frontier agent with answering the central question of an unpublished paper, then have the original authors grade its output.The design uses unpublished work to prevent agents from looking up the authors’ findings and leverages the authors’ months-long investment in the question.
  • Main finding: The agents completed the engineering necessary for research but failed to produce original work at the caliber of a top ML conference.Both papers were unambiguously rejected by their original authors.
  • Failure modes: The agents lacked judgment about publishability, falsified promising hypotheses on small or synthetic datasets, and treated underpowered negative results as substantive findings.They also engaged only shallowly with the literature.
  • Failure modes: Less than 50% of the API budget was spent in both runs, while agents rushed initial exploration, abandoned ambitious targets within ten hours, and failed to substantially change course.These behaviors indicate poor resource awareness and ineffective backtracking from unpromising approaches.
  • Failure modes: The agents repeatedly identified research-design problems without creatively addressing them and ignored explicit instructions about exploration, review frequency, and paper length.Their final papers therefore failed the technical requirements for AI-conference submission.
  • Robustness and limitations: A replication using GPT-5.6 Sol with Codex and its native scaffold produced similar results, while the authors acknowledge non-blinded review as a potential bias.The replication increased confidence that the findings were not simply artifacts of the original scaffold.

2 Shadow evaluations: A new method for measuring progress towards automating AI research

Shadow evaluations measure open-ended AI research by having frontier agents tackle unpublished papers’ central questions and having the original authors assess the outputs in depth. The method addresses weaknesses of verifier-scored benchmarks and blind review, but requires expert time and judgment and was therefore applied to only two papers.

  • Limitations of existing methods: Unlike blind review, shadow evaluations avoid relying on overstretched or poorly matched reviewers and reduce the ability to report only successful submissions from many inexpensive attempts.Conference growth has weakened reviewer expertise matching, while blind-review acceptance rates can conceal failed generated papers.
  • Method: Shadow evaluations give a well-resourced frontier agent an unpublished paper’s central research question, then have the original authors grade its output as a conference submission.The agent solves the same question as the authors without access to the final paper.
  • Advantages: The design combines open-ended tasks with expert grading while avoiding contamination and enabling authors who spent months on the question to judge progress in detail.The target questions come from actual conference submissions and are not available on the web or in the agent’s training data.
  • Limitations: The method can introduce reviewer bias and inherits open-world evaluation limits because expert grading requires specialists and days of review, restricting the study to two papers.Potential biases arise because authors know the work is AI-generated and have already answered the question using a specific method.
  • Robustness and transparency: The researchers documented human interventions, repeated one experiment with a different model and scaffold, and released reviews, survey responses, repositories, and logs for transparency.These measures were intended to mitigate concerns about judgment and reviewer bias.

3 The task and the research setup

The study tests whether well-resourced frontier agents can conduct novel, open-ended AI research using unpublished, uncontaminated questions and expert author grading. It evaluates two questions—LLM persona controllability and distribution-shift detection for tabular foundation models—under a controlled agent setup with documented interventions and prior predictions.

  • Research questions: The evaluations use high-quality AI research questions that were unpublished during the experiments, preventing agents from memorizing answers or finding them online.This design targets real, uncontaminated research questions rather than tasks recoverable from training data or the internet.
  • Research questions: The two research questions concern the structure and controllability of LLM personas and distribution-shift detection for tabular foundation models.The experiments refer to these as the Personas paper and the TabPFN paper, respectively.
  • Research questions: Authors formulated the questions without hinting at promising paths, helped set sufficient resource budgets, and graded the completed papers as top-tier conference reviewers.Their subject expertise allowed them to judge how effectively the agents answered the questions, although they were not blind reviewers.
  • Agent setup: Both experiments ran Claude Opus 4.8 with extra-high reasoning on OpenClaw, giving agents a Linux AWS virtual machine, tools, subagents, GPU jobs, budgets, and remaining-time monitoring.Agents could delegate work and maintain a running research log; the visible passage also describes a PDF-only review subagent.
  • Agent setup: The agents required three human interventions: a scaffold bug fix, a 24-hour deadline extension, and a rewrite request to make inscrutable submissions accessible.The extension was granted to elicit upper-bound performance after the agents submitted drafts self-rated as a “Weak Reject.”
  • Prior predictions: Twelve collaborators supplied pre-experiment predictions, but they had low confidence, substantial variation, and predictions that often differed from the observed results.The survey covered collaborators working on AI research, evaluation, and AI policy.

4 Results

Both agents completed substantial research engineering but failed to answer the central research questions, producing papers that authors unambiguously rejected. Their failures reflected weak research judgment, premature exploration choices, and ineffective use of review feedback.

  • Expert evaluation: Both papers were rejected: Personas received 2 (“Reject”) and TabPFN received 1 (“Strong Reject”), citing weak motivation, no novel contribution, and impenetrable prose.Reviewers also described the experiments and methodological choices as bizarre, difficult to understand, and seemingly shaped by post hoc decisions.
  • Expert evaluation: The agents made a major reasoning error by generalizing from a few failed tests to broad conclusions about unavailable signals.One reviewer characterized this as “proof by example,” while reviewers also found the writing too dense and hedged to distinguish important findings from noise.
  • Research execution: The agents completed large literature reviews, debugged GPU environments, and ran hundreds of experiments, but produced only minor findings without substantively answering the research questions.One potentially interesting Personas finding was that narrow finetuning on misaligned style did not cause broad misgeneralisation.
  • Exploration failures: The Personas agent ended exploration after 5 hours despite budgeting 42 hours, while the TabPFN agent committed to its headline finding 40 hours early.Both agents quickly rejected reasonable hypotheses using small datasets or underpowered methods, ultimately settling on weaker approaches.
  • Review and self-correction: Across 15 revision rounds, self-review never returned acceptance, yet agents narrowed claims and added caveats instead of reconsidering fundamentally unsound data or methods.They also overweighted a lenient reviewer’s early acceptance recommendations despite more critical external and self-reviews.

5 Log analysis reveals five failure modes

The agents completed the engineering required to run AI research without human intervention but failed to produce top-conference-caliber research. Log analysis identified five recurring failures: poor judgment, uncreative responses to negative feedback, ineffective project-level backtracking, poor resource awareness, and instruction drift.

  • Judgment about publishable research: The agents lacked judgment about publishable research, prematurely converging on directions and using underpowered or narrowly selected evidence.Their internal reviews mostly rated papers “Weak Reject,” despite expert reviewers unambiguously rejecting them, suggesting poor calibration of research quality.
  • Creative problem solving: When experiments and reviews exposed weak results, the agents addressed minor critiques or narrowed hypotheses instead of creatively redesigning experiments or reframing the research question.The setup successfully surfaced central critiques, but responses produced thorough negative findings rather than new ideas.
  • Backtracking from dead ends: Despite repeated negative self-reviews, the agents never abandoned their overall approaches and restarted, relying instead on local reruns and robustness checks.In the TabPFN run, six approaches were falsified within fourteen hours, yet the agent did not revise its solution approach during the remaining 110 hours.
  • Resource awareness: The agents showed poor resource awareness: both ended with less than half of their API budgets spent, and one declared completion seven hours before the deadline.They could monitor token, compute, and time budgets but did not recognize that substantial work remained.
  • Instruction drift: Instruction drift increased over the runs: one agent used the strongest available review tool, both exceeded paper-length limits, and both ignored earlier exploration-time rules.The authors attribute this partly to context loss during compaction and the need to actively manage information over multi-day projects.

6 A robustness experiment with Codex and GPT-5.6 Sol Ultra reproduced these failure modes

A robustness experiment using Codex with GPT-5.6 Sol at ultra reasoning reproduced many of the original failure modes, including underpowered experimentation, lack of novelty, formatting problems, and poor budget management. It also preserved positive behaviors such as ethical reporting and improved dataset selection.

  • Experimental setup: The robustness scaffold used ultra reasoning, nearly identical instructions, a persistent verifier goal, and a recurring evaluation loop.The setup was designed to address concerns about scaffold overhang from broken tools, missing instructions, or missing verifiers.
  • Failure modes: The Codex experiment reproduced failures in experimental power, novelty, and paper presentation.The agent returned misformatted figures and no appendices.
  • Failure modes: GPT-5.6 exhausted the $3,000 token budget in just over two days, leaving nearly 100 hours of allotted time.It spent most of its budget iterating over hypotheses before scaling a candidate solution, contributing to underpowered experiments.
  • Positive findings: The agent maintained scientific ethics by registering hypotheses and reporting a negative finding rather than fabricating a positive result.The AI self-review process also continued to appropriately return rejects on the generated drafts.
  • Positive findings: Unlike the OpenClaw experiments, the agent found and used a real-world distribution-shifted dataset instead of relying almost exclusively on synthetic shifts.This was identified as one significant improvement over the first experiments.

7 Limitations

The evaluation’s conclusions are limited by scaffold reliability, incomplete model coverage, and the short experimental horizon. The authors argue these constraints likely did not materially alter the central findings, while cautioning that the results may not settle how agents could accelerate AI R&D.

  • Scaffold and model coverage: Two-thirds of respondents said a failed run might be explained by scaffold limitations, although the authors expect vendor-provided scaffolds to be more reliable.The authors also note that capable open-source scaffolds appear within the margin of error on long-horizon tasks.
  • Scaffold and model coverage: OpenClaw’s session crashes caused by conflicts with Anthropic’s cryptographic thinking-block signatures required resets that discarded accumulated context.The resets occurred 14 times in TabPFN and five times in Personas, but the runs showed similar final-paper quality and failure modes despite differing bug frequency.
  • Time and compute: The six-day horizon was shorter than the original authors’ timelines, but agents underused resources and reviewers mainly criticized research choices, data judgment, and reasoning rather than experiment quantity.The compute budgets were based on authors’ estimates for answering one selected research question, which agents still failed to advance.
  • Scaffold and model coverage: The strongest Anthropic model could not be tested because Fable 5’s frontier AI R&D abilities were deliberately limited.The authors are seeking access to Fable/Mythos 5 for future experiments.
  • Interpretation and scope: The findings may not resolve whether agents can accelerate AI R&D because full open-ended-task automation might be unnecessary or the measured skills might not lie on the critical path.The authors nevertheless regard the distinction between verifiable-task capabilities and open-ended research as relevant to that broader question.

8 Potential biases

The open-world evaluation permits researcher discretion, so prior beliefs and interpretive biases may shape both the study and its conclusions. The authors argue that unbiased evaluation is impossible, but transparency, epistemic diversity, and released raw materials can make interpretations more scrutinizable.

  • Sources of bias: Researcher discretion over questions, study design, and execution means prior beliefs and biases could affect the evaluation’s results.The authors specifically note that the core team’s positions may influence the evaluation.
  • Interpretive disagreement: Interpreting the agents’ failures as deficits in creativity and judgment is contested, with coauthors instead proposing reasoning, logic, or epistemic lock-in failures.The selected studies also reflect the researchers’ understanding of what constitutes empirical results.
  • Objectivity tradeoff: Open-world evaluations cannot be unbiased: clearer benchmarks improve objectivity but narrow task specifications, whereas open-world designs enable richer research questions.The authors characterize this as a tradeoff between objectivity and evaluation-task richness.
  • Mitigations: Epistemically diverse evaluation teams, disclosure of biases, and explicit disagreement can make open-world evaluations more persuasive.The authors report that coauthors with different priors helped surface disagreements about interpreting the results.
  • Unknown biases: Unknown biases may arise during discretionary log analysis, so the authors release expert reviews, survey responses, agent repositories, and run logs for independent checking.They acknowledge that failures fitting the researchers’ expectations may have been identified more readily than contrary failures.

Funding … 1. Summary

The paper acknowledges funding and API support, frames two open research questions on persona control and deployment-time shift detection, and summarizes TabPFN’s silent degradation under distribution shift.

  • Funding: Coefficient Giving and Schmidt Sciences funded the project, while OpenAI provided API credits for evaluating GPT-5.6 Sol.
  • A.1 Persona Cartography: The Persona Cartography question asks whether LLM personas can be decomposed, measured, and controlled as structured trait-space positions using weight-space interventions.The motivation is to replace brittle prompting or costly retraining with independently controllable persona components.
  • A.2 TabPFN: The TabPFN question asks for a theoretically justified, empirically validated deployment-time detector that calibrates alarms when deployment accuracy materially trails in-distribution accuracy.The detector receives a PFN, labeled in-distribution reference batch, and unlabeled deployment batch, with access to gradients and intermediate activations but not pretraining data.
  • A.2 TabPFN: TabPFN and related prior-fitted networks perform tabular prediction in-context using labeled support and unlabeled query batches, without task-specific gradient updates.They are transformer-based foundation models that have surpassed gradient-boosted trees on standard benchmarks and are entering clinical, financial, and industrial pipelines.
  • B Reviews from paper authors: The paper authors drafted the review documents at the CRUX team’s request.
  • 1. Summary: The TabPFN summary investigates white-box mechanisms for detecting harmful query-distribution shifts while remaining robust to harmless shifts.It highlights two findings: concept-shift detection cannot be detected, whereas covariate-shift detection concerns changes in p(x).

2. Strengths and Weaknesses · 3–6. Criterion Ratings · 7. Questions

The review identifies one significant empirical strength but substantial concerns about quality, clarity, originality, and overlap with prior work. It rates the paper poorly overall and asks for clarification of the detector, broader real-shift experiments, and validation of the D3M implementation.

  • 2. Strengths and Weaknesses: The proposed method’s main strength is beating D3M and Detectron on synthetic datasets and shifts, but AUROC falls from 0.76 on synthetic data to 0.60 on real shifts.The review also notes that comparisons with D3M and Detectron on real shifts are absent.
  • 2. Strengths and Weaknesses: The paper’s claim that no usable PFN-internal signals exist is criticized as an unsupported “proof by example” based on only a few unsuccessful signals.The reviewer says this invalidates a major contribution highlighted in the Introduction.
  • 2. Strengths and Weaknesses: Clarity is weakened by irrelevant engineering details, confusing notation such as acc_R versus acc(D), and explanations that obscure the method’s central ideas.The relabeling y → (y + 1) mod C is specifically described as irrelevant to understanding the method.
  • 2. Strengths and Weaknesses: The review considers the PFN whitebox signal selection the only clearly original contribution, while the proposed model-agnostic test appears to reproduce or partially reuse prior methods.It identifies overlap with Guillory et al. and Pouget et al.’s suitability filter using confidence.
  • 3–6. Criterion Ratings: The criterion ratings are Quality 1, Clarity 2, and Significance 2, reflecting the unsupported generalization, excessive engineering detail, and limited research value described above.The reviewer nevertheless commends the paper’s honesty and extensive documentation of limitations.
  • 7. Questions: The reviewer asks which detector produced Table 3 and how it differs from Pouget et al.’s confidence-based suitability filter, noting an apparent contradiction between Sections 3.2 and Appendix B.The review suggests Contribution 3 should be restated as a baseline finding because the detector may be dominated by the prior method it borrows from.
  • 7. Questions: The reviewer requests a broader experimental grid using real datasets with covariate shift, including additional experiments on Folktables and ACS or shifts induced by sampling.The reviewer says this request would not change the score and is mainly for completeness.
  • 7. Questions: The reviewer asks why D3M was re-implemented and requests comparison between the approximation and the publicly available reference implementation.The requested comparison is intended to establish implementation validity.

8. Limitations … Strengths

The review finds the limitations adequately addressed and expresses maximal confidence, while assigning a Strong Reject score. It summarizes the paper’s persona-steering geometry and identifies its significance, originality, and potential usefulness as strengths, but provides no separate evidence for the Personas review section.

  • 8. Limitations: The authors do an excellent job stating every limitation and clearly discussing assumptions in Sections 5.2–5.5.The review notes that some limitations may be overstated, but still judges them adequately addressed.
  • 9. Overall Score: The submission receives a score of 1 — Strong Reject, indicating fundamental problems, serious errors, or unaddressed ethical issues.The selected rubric describes Strong Reject as appropriate when fundamental problems are present.
  • 10. Confidence: The reviewer rates confidence 5 — Absolutely certain after checking the mathematics, details, and related work carefully.This is the highest confidence category in the provided rubric.
  • 1. Summary: The paper studies weight-difference directions as a geometric basis for controlling personas and compares them with activation steering.Experiments examine trait composition, shared structure, transfer to emergent misalignment, and reward hacking on Qwen 2.5.
  • 2. Strengths and Weaknesses: The review frames strengths and weaknesses across quality, clarity, significance, and originality, requiring specific evidence and named prior work for novelty criticisms.The instructions emphasize citing sections, equations, tables, or figures rather than making vague claims.
  • Strengths: The geometric question is useful and potentially could inform improved fine-tuning methods or steering of existing weight differences.This assessment depends on finding sensible geometric structure in persona-steering directions.
  • Strengths: The paper reports a significant finding: models can express an emergent-misalignment-like style without taking misaligned actions or proposing strongly misaligned answers.The significance comes from contradicting expectations about emergent misalignment, rather than from general interest alone.

Weaknesses … 10. Confidence

The review identifies major concerns about motivation, methodology, clarity, and evaluation, assigns a reject score of 2, and reports confidence level 4. The reviewer requests stronger evidence that weight space offers practical benefits beyond capacity-matched activation baselines.

  • Weaknesses: The paper’s practical or scientific motivation is unclear because its headline result concedes no advantage over capacity-matched activation space.The remaining claims about structured geometry and identifiable directions do not establish a clear payoff, and the reviewer notes that additional controls are needed.
  • Weaknesses: The writing is dense, heavily hedged, terminologically inconsistent, and difficult to interpret because it lacks informative figures.The main figure is only a diagram, and understanding it requires cross-referencing the appendix.
  • Weaknesses: The choice of traits, measures, and datasets appears post-hoc, with hand-picked traits, weak lexical proxies, incompletely reported LLM judges, and small, insufficiently justified corpora.Expanding from five to ten traits does not preserve the reported results, while lexical scoring may be circular and corpus design may explain failures to elicit reward hacking and EM.
  • 3–6. Criterion Ratings: The reviewer rates Quality 2, Clarity 1, Significance 2, and Originality 3, citing bizarre methodological choices, poor presentation, limited justification, and a novel research approach.The ratings characterize the work as potentially interesting for personalization but insufficiently justified over comparable methods.
  • 7. Questions: The reviewer asks what weight-space representations enable beyond capacity-matched activation baselines and whether geometry results survive alternative traits, corpus designs, and proxy-alignment tests.Suggested tests include randomly sampled or adversarially chosen traits and experiments assessing how much measured control is driven by lexical alignment with the fine-tuning objective.
  • 8. Limitations: The limitations and potential negative societal impacts are marked as adequately addressed.The checklist records “Yes” for adequate coverage and provides no missing items or corrective suggestions.
  • 9. Overall Score: The overall recommendation is 2 — Reject, reflecting notable technical flaws, weak evaluation, poor reproducibility, or inadequately handled ethical concerns.The review does not select the stronger or borderline acceptance categories.
  • 10. Confidence: The reviewer reports confidence level 4: confident but not certain, with a small chance of misunderstanding or missing unfamiliar related work.This confidence level is below “Absolutely certain.”

C Pre-experiment expectations survey

Before the experiments, twelve coauthors forecast how frontier AI agents would perform, including their criticality, shortcuts, scaffold dependence, and likely trajectory failures. Expectations were notably pessimistic about shortcuts and operational failures.

  • Survey design: Twelve coauthors were surveyed about frontier AI agents’ capabilities as of June 2026, after receiving full scaffold details but not the research questions.Respondents worked on AI research, evaluation, or AI policy and answered from that specified perspective.
  • Expected behavior: 3 respondents expected a more-critical final self-review than human reviewers, 3 expected dependence, and 6 expected a less-critical review.The agent was asked to create a blind review using only the paper PDF and reviewer guidelines.
  • Expected behavior: 10 of 12 respondents expected the agent to take shortcuts that a skilled researcher would not, while 2 said it depended.Examples included trivializing the research question, running underpowered experiments, or offloading key reasoning or analysis to humans.
  • Expected failures: 8 respondents said scaffold improvements might resolve failures to meet a moderate success bar, compared with 3 yes and 1 no.The question concerned whether the deficiency could have been resolved with a stronger scaffold for current models.
  • Expected failures: 9 respondents expected unresolvable-error loops, while 7 each expected p-hacking or selective reporting and obvious misformatting in at least one trajectory.Other anticipated events included fabricated or hallucinated results (6), misreported or dishonest experiments (4), and catastrophic actions (2).

D Comprehensive survey of prior autonomous-AI R&D experiments · E CRUX 2 OpenClaw agent scaffold diagram

Prior autonomous-AI R&D experiments span benchmarked engineering tasks, agent or scaffold improvement, and end-to-end research workflows, using verifiers, objective metrics, LLM judges, or human review. The CRUX 2 OpenClaw scaffold coordinates a core agent, subagents, tool calls, and long-running GPU experiments during research runs.

  • D Comprehensive survey of prior autonomous-AI R&D experiments: PostTrainBench, MLGym, and RE-Bench evaluate research-engineering performance with verifiers under fixed time budgets across question answering, iterative research, and seven machine-learning problems.These environments compare agents with human experts or task-specific baselines using verifier-scored outcomes.
  • D Comprehensive survey of prior autonomous-AI R&D experiments: MLRC-Bench, AIRS-Bench, ResearchGym, and MLS-Bench broaden evaluation to competitions, full-lifecycle tasks, new methods for recent papers, and methodological improvements.Their evaluators use objective competition metrics, held-out task-specific metrics, code-based tests, or expert human baselines with narrow edit scopes.
  • D Comprehensive survey of prior autonomous-AI R&D experiments: STOP, Meta Agent Search, AIDE2, Darwin Gödel Machine, and Red Queen Gödel Machine improve agents, scaffolds, or evaluators through program revision, archives, branching, benchmark tests, and co-evolution.These systems generally select or retain candidates using benchmark, utility, or agent-as-judge signals while leaving some search components unchanged.
  • D Comprehensive survey of prior autonomous-AI R&D experiments: ASI-Arch, AlphaEvolve, Automated Weak-to-Strong Researcher, Autoresearch, and NanoGPT Speedrun use iterative or evolutionary search over hypotheses, algorithms, code, training configurations, or supervision.Their evaluations rely on fixed empirical, training-loss, efficiency, training-speed, or model-quality criteria, often across many automatically evaluated trials.
  • D Comprehensive survey of prior autonomous-AI R&D experiments: Reported results include agents exceeding a human baseline on training speed and model quality, while one generated workshop submission exceeded its acceptance threshold before withdrawal.These outcomes were verifier-scored or obtained through double-blind peer review rather than direct evaluation by the original paper authors.
  • D Comprehensive survey of prior autonomous-AI R&D experiments: End-to-end research systems generate experiments and papers using templates, guided tasks, or agent tree search, with outputs assessed by LLM rubrics, blind peer review, or human review.Examples include AI Scientist, AI Scientist-v2, PaperBench, Zochi, AI-Researcher/Scientist-Bench, CodeScientist, and Agent Laboratory, with varying degrees of human involvement.
  • E CRUX 2 OpenClaw agent scaffold diagram: OpenClaw coordinates the core agent, subagents, tool calls, and long-running GPU experiments during a research run.Figure 5 presents this coordination as the CRUX 2 AI Research scaffold.
Loading 2607.27191v2…