PaperlayerView exact revision on arXiv

Source-linked AI summary

Agent Laboratory: Using LLM Agents as Research Assistants

Samuel Schmidgall, Yusheng Su, Ze Wang, Ximeng Sun, Jialian Wu, Xiaodong Yu, Jiang Liu, Michael Moor, Zicheng Liu, Emad Barsoum

arXiv:2501.04227v2cs.HCcs.AIcs.CLcs.LG

TL;DR

Limited research resources leave many high-quality scientific ideas unexplored. Agent Laboratory uses autonomous language agents to turn human research ideas into reports and code repositories, with optional human feedback. Human-guided co-pilot mode scored higher than autonomous mode across most metrics, while different backends led usefulness and experimental quality.

  • Problem

    Researchers have limited resources for exploring scientific ideas, leaving many high-quality ideas unexplored.

    Source: §1 · p. 2

  • Method

    Agent Laboratory uses autonomous language agents to turn human research ideas into research reports and code repositories, allowing user-selected levels of human feedback.

    Source: §1 · p. 2

  • Results

    Human-guided co-pilot mode scored higher than autonomous mode across most metrics; o1-preview was rated most useful, while o1-mini achieved the highest experimental quality.

    Source: §6 · p. 21

  • Takeaways & Limitations

    Human-guided research agents may help researchers spend less time on repetitive tasks and more on creative, conceptual work.

    Source: §6 · p. 22

    LLM-based self-evaluation is less reliable than human review for assessing research report quality.

    Source: §5.1 · p. 19

Abstract

from arXiv · show

Historically, scientific discovery has been a lengthy and costly process, demanding substantial time and resources from initial conception to final results. To accelerate scientific discovery, reduce research costs, and improve research quality, we introduce Agent Laboratory, an autonomous LLM-based framework capable of completing the entire research process. This framework accepts a human-provided research idea and progresses through three stages--literature review, experimentation, and report writing to produce comprehensive research outputs, including a code repository and a research report, while enabling users to provide feedback and guidance at each stage. We deploy Agent Laboratory with various state-of-the-art LLMs and invite multiple researchers to assess its quality by participating in a survey, providing human feedback to guide the research process, and then evaluate the final paper. We found that: (1) Agent Laboratory driven by o1-preview generates the best research outcomes; (2) The generated machine learning code is able to achieve state-of-the-art performance compared to existing methods; (3) Human involvement, providing feedback at each stage, significantly improves the overall quality of research; (4) Agent Laboratory significantly reduces research expenses, achieving an 84% decrease compared to previous autonomous research methods. We hope Agent Laboratory enables researchers to allocate more effort toward creative ideation rather than low-level coding and writing, ultimately accelerating scientific discovery.

1. Introduction

Agent Laboratory turns human-provided machine-learning research ideas into reports and code repositories, with adjustable human feedback and compute budgets. Its evaluations report backend-dependent research quality, benefits and trade-offs from co-pilot use, and reduced research costs.

1. Introduction

  • Agent Laboratory converts human research ideas into reports and code repositories through autonomous language agents, with user-controlled feedback and flexible compute budgets.The open-source framework accommodates available CPU, GPU, memory, and inference budgets. It addresses constraints that leave high-quality ideas unexplored, aiming to free researchers for creative ideation and experiment design.Source: §1 · pp. 2–3
  • o1-preview was rated most useful and performed best in human NeurIPS-style evaluations, while o1-mini achieved the highest experimental quality scores.o1-preview particularly excelled in clarity and soundness; gpt-4o lagged across experimental quality, report quality, and usefulness.Source: §1 · p. 2
  • 6.1/10 automated versus 3.8/10 human overall scores show that automated evaluations significantly overestimated research quality.Similar discrepancies appeared in clarity and contribution, motivating human feedback to complement automated assessment.Source: §1 · p. 2
  • Co-pilot mode scored higher overall than autonomous mode, but experimental quality and usefulness trade-offs revealed difficulties aligning outputs with researcher intent.Evaluations covered custom and preselected topics. Human users rated utility and usability highly, and most participants decided to continue using the framework.Source: §1 · pp. 2–3
  • mle-solver achieves state-of-the-art performance on a subset of MLE-Bench challenges, earning more medals than MLAB, OpenHands, and AIDE.The reported results include higher consistency and scores than other solvers, with gold and silver medals.Source: §1 · p. 3
  • $2.33 USD per paper with a gpt-4o backend illustrates Agent Laboratory’s greatly reduced automated research costs compared with other works.The paper also provides inference-time statistics and cost breakdowns by research phase for different backends.Source: §1 · p. 3

2. Background & Related Work

The background connects autoregressive LLMs and tool-using agents to automated machine learning and scientific research. Mixed findings on ideation and limitations in autonomous research support combining human guidance with LLM-based workflows.

Large language models

  • The framework’s research agents use autoregressive transformer LLMs trained to predict token sequences and generate text completions through sampling.Transformer architectures (Vaswani (2017)) capture long-range dependencies, while extensive pretraining and scaling enable language tasks such as translation, summarization, and reasoning Brown (2020).Source: §2 · p. 3

LLM Agents

  • LLM agents extend language models through structured frameworks, combining prompting strategies, iterative refinement, self-improvement, and external tools for complex task execution.Applications span software engineering, cybersecurity, medical diagnosis, autonomous robotics, web tasks, and game playing.Source: §2 · p. 3

Automated machine learning

  • 75, 74, and 6 Kaggle challenges underpin MLE-Bench (Chan et al. (2024)), DS-bench (Jing et al. (2024)), and MLAgentBench (Huang et al. (2024)), respectively.These benchmarks assess data preparation, model development, and submission; existing solvers automate feature implementation, bug fixing, and code refactoring. Agent K (Grosnit et al. (2024)) demonstrates human-level Kaggle challenge solving from a challenge URL.Source: §2 · p. 4

AI in Scientific Discovery

  • Earlier AI approaches support discovery across mathematics, materials science, chemistry, algorithms, and computational biology as tools rather than autonomous research agents.The section distinguishes these disciplinary applications from agents that perform research autonomously.Source: §2 · p. 4

LLMs for research related tasks

  • LLMs support code generation, software development, discovery-oriented coding, research question-answering, ideation, paper reviewing, literature search, and experiment-outcome prediction.These capabilities cover individual research-related tasks rather than necessarily constituting an autonomous research workflow.Source: §2 · p. 4
  • Mixed findings on LLM creativity and human–AI idea novelty motivate combining human-guided ideation with LLM-based research workflows.Si et al. (2024) report greater novelty than humans, while Chakrabarty et al. (2024) report reduced creativity and Anderson et al. (2024); Zhou et al. (2024) report greater homogeneity. Human–AI collaboration studies also find mixed novelty outcomes (Ashkinaze et al. (2024); Liu et al. (2024); Padmakumar & He (2024)).Source: §2 · p. 4

LLMs for autonomous research

  • The AI Scientist (Lu et al. (2024a)) automates end-to-end research, but feasibility limitations suggest a complementary role for LLMs (Si et al. (2024)).ChemCrow (M. Bran et al. (2024)) and Coscientist (Boiko et al. (2023)) automate chemistry ideation and experimentation, while ResearchAgent (Baek et al. (2024)) generates ideas, designs experiments, and iteratively incorporates reviewer-agent feedback. Swanson et al. (2024) combine scientist agents with high-level human feedback to produce nanobody binders targeting recent SARS-CoV-2 variants.Source: §2 · pp. 4–5

3. Agent Laboratory

Agent Laboratory organizes research into literature review, experimentation, and report writing, using specialized agents to collect papers, conduct experiments, and generate reports. It supports autonomous execution or human checkpoints after each subtask.

3.1. Literature Review

  • The PhD agent iteratively queries arXiv, examines abstracts or full texts, and curates relevant papers until the specified review size is reached.Each summary query retrieves the top 20 relevant abstracts; selected summaries or full texts are added to the review.Source: §3.1 · pp. 5–6

3.2. Experimentation

  • PhD and Postdoc agents plan experiments and interpret results, while ML and software engineers prepare and validate the data-processing code.Planning specifies models, datasets, and experimental steps; the Postdoc submits the agreed plan and interpretation. The ML Engineer can search HuggingFace datasets, and preparation code undergoes iterative Python checks before submission.Source: §3.2 · pp. 6, 8
  • mle-solver generates experimental code from scratch, iteratively edits or replaces programs, and ranks successful programs using an LLM reward model.The reward model scores alignment between the research plan, code, and observed output from 0 to 1. Failed programs receive up to three repair attempts before a new replacement.Source: §3.2 · pp. 6–7
  • Self-reflection targets compilation failures or higher scores, while top-program sampling and parallel modifications maintain diversity among high-scoring programs.Each step randomly samples a retained program, generates N simultaneous modifications, and uses the best modification to replace the pool's lowest-scoring program.Source: §3.2 · p. 7

3.3. Report Writing

  • paper-solver helps PhD and Professor agents summarize completed research through structured report generation, optional literature searches, and compilation-checked LaTeX edits.The scaffold contains eight standard academic sections, and optional arXiv searches supplement the original literature review. The tool produces a human-readable research summary rather than replacing the entire academic paper-writing process.Source: §3.3 · pp. 8–9
  • 65% accuracy versus 66% for humans and F1 score 0.57 versus 0.49 after calibration were reported for the reviewer from Lu et al. (2024b).Agent Laboratory adapts this NeurIPS-style review system, evaluated on 500 ICLR 2022 papers, for iterative paper scoring. Three reviewer agents guide the PhD agent's decision to finalize or revisit planning, experimentation, or interpretation.Source: §3.3 · pp. 9–10
  • 7 is the Overall score, with an Accept decision, in the o1-mini Word Order Sensitivity example review.It praises experimental design and presentation but flags limited novelty, mitigation coverage, and discussion of limitations and societal impacts. It also notes RACE-only evaluation and residual option-order sensitivity.Source: §3.3 · pp. 9–10

3.3.1. Autonomous versus Co-Pilot Mode:

  • Co-pilot mode adds human checkpoints after every subtask, whereas autonomous mode proceeds sequentially without human involvement beyond the initial research idea.At each checkpoint, the human can approve progression or request another attempt with high-level improvement notes.Source: §3.3.1 · p. 10

4. Results

Agent Laboratory’s mle-solver earned more medals than the compared solvers on the evaluated MLE-Bench subset, while human involvement improved paper ratings without reaching accepted-paper averages. Results also show backend-dependent research quality, inflated automated reviews, and trade-offs between cost, speed, and reliability.

4.1. Evaluation of quality by language model

  • 4.4/5 usefulness and 3.4/5 report quality made o1-preview the autonomous leader on those measures, while o1-mini led experimental quality at 3.2/5.gpt-4o averaged 2.6/5 experimental quality, 3.0/5 report quality, and 4.0/5 usefulness; o1-mini averaged 3.2/5 report quality and 4.3/5 usefulness, while o1-preview’s experimental quality was 2.9/5. Topic effects were substantial: word order led report quality and usefulness, cognitive bias led experimental quality, and image-noise experimental quality ranged from gpt-4o’s 1.5/5 to o1-mini’s 4.0/5.Source: Figure 5 · p. 11, §4.1 · pp. 11–12

4.1.1. Human reviewer scores by language model

  • 3.8/10 from human reviewers versus 6.1/10 from automated reviewers exposed substantial overestimation of generated papers’ quality.Human overall ratings were 3.5/10 for gpt-4o, 3.8/10 for o1-mini, and 4.0/10 for o1-preview, below the reported accepted-NeurIPS average of 5.9. o1-mini led technical quality at 2.3/4 and o1-preview led soundness at 2.2/4, but contribution averaged 2.1/4; the discrepancies support reporting human evaluations alongside automated reviews.Source: §4.1.1 · pp. 12, 14, Figure 6 · p. 13

4.2. Evaluation of co-pilot quality

  • Co-pilot evaluation incorporated researcher feedback after each subtask and compared custom with preselected topics, using o1-mini except for literature review.Each researcher generated two papers, assessed the tool and their papers, and had the papers evaluated by external researchers for comparison with autonomous operation.Source: §4.2 · p. 14

4.2.2. Evaluation of co-pilot generated papers

  • +0.58 overall points separated externally evaluated co-pilot papers from autonomous papers, but co-pilot scores averaged only 4.38/10.Co-pilot self-evaluations averaged 4.13/10 versus autonomous papers’ 3.8/10; external evaluations were higher overall but lower for clarity, and favored preselected topics whereas self-evaluations favored custom topics. Against autonomous papers, external reviewers found gains in quality (+0.75) and soundness (+0.48), but little improvement in significance or contribution; co-pilot papers remained below the accepted NeurIPS 2024 average of 5.85.Source: Figure 7 · p. 15, §4.2.2 · p. 16

4.2.1. Quality as a tool

  • 3.5/5 utility, 3.75/5 continuation, 3.63/5 satisfaction, and 4.0/5 usability characterized researchers’ overall co-pilot tool ratings.Researchers rated generated outputs at 2.38/5 experimental quality, 3.13/5 report quality, and 3.75/5 usefulness—all below corresponding autonomous o1-mini ratings—with feedback attributing dissatisfaction to difficulty realizing their exact project vision. Custom topics received higher utility, continuation, and satisfaction ratings, while preselected topics were easier to use; requested improvements included interface and intermediate-result inspection, more figures, and better literature review.Source: §4.2.1 · pp. 14–15, Figure 7 · p. 15

4.3. Runtime statistics

  • $2.33 per workflow and 1165.4 seconds made gpt-4o the cheapest and fastest backend, ahead of o1-mini and o1-preview.o1-mini required $7.51 and 3616.8 seconds, while o1-preview required $13.10 and 6201.3 seconds; the previous gpt-4o workflow of Lu et al. (2024b) cost approximately $15. Report writing was the most expensive phase across backends, costing o1-preview $9.58 despite comparable task success rates.Source: Figure 8 · p. 17, §4.3 · pp. 17–18
  • 95.7% average subtask success made o1-preview the most reliable backend, followed by gpt-4o at 94.3% and o1-mini at 92.8%.Literature review was inexpensive but failure-prone: o1-mini completed it in 56.8 seconds for $0.16, versus gpt-4o’s 92.9 seconds for $0.12. gpt-4o led plan-formulation speed and cost at 23.3 seconds and $0.03; data-preparation success was 100% for gpt-4o, 90% for o1-preview, and 80% for o1-mini.Source: Figure 8 · p. 17, §4.3 · p. 18

4.4. Evaluating mle-solver on MLE-Bench

  • 4 medals for mle-solver exceeded OpenHands’ 2, AIDE’s 2, and MLAB’s 0, with valid solutions submitted for every evaluated challenge within two hours.Evaluation covered ten low-complexity text/tabular MLE-Bench challenges, using dataset descriptions, distilled notebook knowledge, an 80/20 training/development split, and final Kaggle test-set evaluation. mle-solver exceeded median human performance on six challenges versus AIDE’s five, OpenHands’ two, and MLAB’s zero; comparator averages excluded invalid submissions.Source: Figure 9 · p. 18, §4.4 · pp. 18–19

5. Limitations

Agent Laboratory faces limitations in self-evaluation, workflow flexibility, factual reliability, and runtime robustness. Its autonomous research capabilities also raise risks involving scientific quality, bias, and misuse that require transparency and safeguards.

5.1. Workflow limitations

  • 53.3% versus 56.1% agreement for LLM versus human reviewers highlights self-evaluation limitations, potentially reflecting superficial patterns rather than robust evaluation criteria.Despite higher reviewer scores, Agent Laboratory reports were qualitatively less satisfying and had poorer figures than The AI Scientist (Lu et al. (2024b)). Complete-paper scoring also mismatches reports intended to help humans understand and scale experiments rather than replace human writing; unreliable subjective evaluation affects both mle-solver and paper-solver.Source: §5.1 · p. 19
  • Two-figure limits, fixed paper structures, and missing autonomous repository management constrain Agent Laboratory’s workflow.Proposed improvements include passing all generated figure paths to paper-solver and allowing flexible repository-level file modification and execution. Currently, files are supplied at each necessary step and saved according to the producing phase.Source: §5.1 · pp. 19–20
  • Uncommon hallucinations, particularly from lower-performing models such as gpt-4o, included experimental results that never occurred, risking misinformation in automated research.Future work must address these hallucinations to prevent automated research tools from propagating misinformation.Source: §5.1 · p. 20

5.2. Common failure modes

  • 0% accuracy across tested methods sometimes remained uncorrected when mle-solver exhausted its steps; code execution also caused failures and host-system risks.mle-solver disproportionately edited line 0, making replace more likely to compile successfully, and generated exit() calls requiring manual removal to prevent process termination. It also executed host commands through subprocess.run(); no problems were observed, but safeguards were recommended.Source: §5.2 · p. 20
  • 5 searches became paper-solver’s limit after arXiv retrieval sometimes required up to 100 attempts; literature review also suffered instruction-following and token-limit failures.gpt-4o, o1-mini, and o1-preview repeatedly invoked summarize until phase-step exhaustion terminated runs. Retrieved papers and printed data-preparation or experimental outputs could exhaust token limits.Source: §5.2 · p. 20

5.3. Ethical considerations

  • Automated research could lower barriers to substandard or misleading outputs, overwhelm peer review, and amplify dataset or algorithm biases.Transparent disclosure of AI involvement is important for mitigating these risks and maintaining accountability.Source: §5.3 · pp. 20–21
  • Misuse could enable harmful technologies, bypass ethical oversight, or produce biased environmental analyses, motivating safeguards and robust governance.Examples include automated malware creation (Begou et al. (2023); Francia et al. (2024); Happe & Cito (2023); Xu et al. (2024)). Misuse risks may grow as the platform matures without safeguards aligned with ethical research standards (Jiao et al. (2024); Watkins (2024)).Source: §5.3 · p. 21

6. Discussion

Agent Laboratory integrates research stages into a human-centered pipeline that adapts to researchers’ desired involvement and available compute. Human feedback improved output quality, while differences across LLM backends and gaps between automated and human reviews highlight the importance of human evaluation.

6. Discussion

  • Agent Laboratory integrates research stages into an open-source co-pilot pipeline that adapts to researchers’ desired interaction and compute availability.Specialized agents handle repetitive work, including data preprocessing and coding, allowing researchers to focus on conceptual design and critical thinking under human oversight.Source: §6 · pp. 21–22
  • Human feedback in co-pilot mode produced higher-quality outputs than autonomous mode across most metrics, with users reporting high utility and usability.Most participants decided to continue using Agent Laboratory after their experience.Source: §6 · p. 21
  • o1-preview was rated most useful, o1-mini achieved the highest experimental quality, and gpt-4o offered the fastest execution and lowest costs.MLE-Bench evaluations also showed that mle-solver improved general machine-learning problem solving over previous methods.Source: §6 · p. 21
  • Autonomous outputs fell short of human expectations for clarity and soundness, while automated reviewer scores did not predict human reviewer scores.These findings demonstrate the importance of human evaluations in automated research.Source: §6 · p. 21
  • Agent Laboratory addresses prior systems’ limitations in human interaction, current knowledge, paper generation, and open-ended research, although further improvements remain necessary.The AI Scientist (Lu et al. (2024b)) lacks human-computer interaction; Virtual Lab (Swanson et al. (2024)) lacks up-to-date knowledge and paper generation and was demonstrated only for nanobody design. ChemCrow (M. Bran et al. (2024)) and Coscientist (Boiko et al. (2023)) cannot solve open-ended research problems.Source: §6 · p. 22
  • Future work should compare researchers’ long-term outcomes with and without Agent Laboratory and explore automatic workflow and agent generation to optimize the framework.Current human evaluations provide only a snapshot of utility; longitudinal comparisons could assess sustained effects on research efficiency and scientific discovery.Source: §6 · p. 22

A. Agent Laboratory configuration · A.1. Hyperparameters

Agent Laboratory’s configuration section identifies Table 1 as its hyperparameter table; the supplied passage contains only the caption.

A.1. Hyperparameters

  • Table 1 covers Agent Laboratory’s hyperparameters, but no parameter names or values are supplied.The caption alone does not support a more detailed configuration summary.Source: Table 1 · p. 31

A.2. Hardware

All experiments used a 2023 MacBook Pro with an Apple M3 Max processor and 36 GB of memory.

A.2. Hardware

  • All experiments ran on a 2023 MacBook Pro with an Apple M3 Max processor and 36 GB of memory.Source: §A.2 · p. 31

B. Prompts

Agent Laboratory’s prompts combine role assignments, research context, phase-specific instructions, and constrained commands to guide literature review, planning, data preparation, code generation and revision, interpretation, and report writing. They also specify editing tools, reporting limited to recorded experimental results, section-specific writing guidance, and structured critical reviews with prescribed JSON assessments.

B.1. Base Inference Prompt

  • The base prompt combines the agent’s role, phase instructions, available commands, research objective, interaction history, progress, feedback, and notes while requesting one new command.70% progress toward the step limit triggers an instruction encouraging submission; the completion string is otherwise normally empty. The prompt also requires output different from the previous command.Source: §B.1 · p. 32

B.2. Context Prompts

  • Context prompts supply phase-specific research artifacts, including literature reviews, plans, dataset and experiment code, results, interpretations, and previous-round reports and reviewer responses.Planning receives the literature review; data preparation additionally receives the plan. Interpretation receives code and results, while report refinement additionally receives the interpretation.Source: §B.2 · p. 33

B.3.1. PhD Student phase; B.7.1. PhD Student Role Description

  • The PhD student reviews literature, develops simple experimental plans, directs data preparation, interprets results, and pursues favorable ICLR reviews.The role is a computer science PhD student at a top university, with arXiv summary search and full-text retrieval; plans specify models, datasets, and experiment details. Data preparation prioritizes simple code, and interpretations must accurately convey numbers, relevant metrics, and significance measures without delaying submission.Source: §B.3.1 · pp. 34–35, §B.7.1 · p. 40

B.4. Machine Learning Engineer Phase Descriptions; B.6.2. ML Engineer Agent Command Description; B.7.2. Machine Learning Engineer Role Description

  • The machine learning engineer prepares simple dataset code under PhD supervision, using Python execution, dialogue, and HuggingFace dataset search.The engineer works at a top university, integrates the literature review and plan, and receives execution errors and printed output. Commands must be fenced and issued individually; code must use an external HuggingFace dataset, avoid a main function, and keep function variables local or explicitly passed.Source: §B.4 · p. 35, §B.6.2 · pp. 38–39, §B.7.2 · p. 40

B.6.1. PhD Student Command Description

  • PhD commands enforce searching paper summaries, reading full texts before adding papers, and submitting finalized dataset code after dialogue with the engineer.SUMMARY, FULL_TEXT, and ADD_PAPER support literature collection, with one paper added at a time and summaries emphasizing experimental results. SUBMIT_CODE requires an external HuggingFace dataset import, local or passed function variables, no main function, and prompt submission; each inference executes only one fenced command.Source: §B.6.1 · pp. 36–38

B.5. Postdoc Phase Descriptions; B.6.3. Postdoc Agent Command Description; B.7.4. Postdoc Agent Role Description

  • The postdoc guides the PhD student through experimental planning and results interpretation, using dialogue before submitting PLAN or INTERPRETATION commands.Defined as a computer science postdoctoral student at a top university, the agent favors simple experiments with explicit models, datasets, and procedures. Interpretation draws on literature, code, plans, and results, accurately preserves metrics and significance measures, and must be submitted promptly.Source: §B.5 · pp. 35–36, §B.6.3 · pp. 39–40, §B.7.4 · p. 41

B.8.1. Tools

  • Code tools support line-range editing and full replacement, testing proposed changes before installation and rejecting changes that produce errors.EDIT replaces an inclusive line range and is preferred over wholesale REPLACE operations. Automated repair aims to eliminate the reported and other potential errors while preserving output and matching the original code closely.Source: §B.8.1 · pp. 41–42

B.8.1. Tools; B.7.3. Professor Agent

  • 0 to 1 is the professor agent’s scoring range for following the research plan, building the code, and producing proper output.The role is a computer science professor at a top university acting as an expert reward model. Scores must be returned as floating-point values through the SCORE command.Source: §B.7.3 · p. 40, §B.8.1 · p. 42

B.8.1. Tools

  • Coding prompts enforce single, properly fenced commands, prohibit TensorFlow and Keras, and require accuracy checks, explanatory experiment printouts, and at least two result figures.Figures should be named Figure_1.png and Figure_2.png; zero accuracy must be investigated and corrected. The agent is instructed to avoid repeatedly issuing the same commands.Source: §B.8.1 · pp. 43–44
  • The mle-solver must integrate the research plan and literature review into simple code that runs every planned experiment.The prompt supplies the plan and dataset description; dataset code is prepended automatically. It prohibits function definitions and installing new libraries.Source: §B.8.1 · pp. 43–44
  • Reflection prompts request line-by-line error fixes, refinements after successful execution, specific performance and test-generalization improvements, and generalizable insights across code variants.Error feedback includes the previous command and its returned error, alongside a history containing up to five previous errors. Reflections request editing suggestions rather than complete code rewrites.Source: §B.8.1 · pp. 43, 45

B.9. paper-solver Prompts

  • The paper-solver primarily edits inclusive LaTeX line ranges, while full replacement is available; proposed changes are tested and rejected if they produce errors.The prompts discourage frequent full rewrites and encourage substantive paragraph edits. Initial section drafting also uses arXiv searches for relevant papers and permits citations through parenthesized arXiv IDs.Source: §B.9 · pp. 45–46
  • The paper-solver receives the literature review, original research plan, experimental code, observed results, and an interpretation of those results.These materials provide the context for the report-writing task.Source: §B.9 · p. 47
  • Report generation begins with a compilable eight-section LaTeX scaffold and targets an objective, comprehensive paper of eight pages or 4000 words.The scaffold contains Abstract, Introduction, Background, Related Work, Methods, Experimental Setup, Results, and Discussion. Subsequent section prompts request only section content, without packages, titles, or section commands.Source: §B.9 · pp. 47–48

B.9.1. Per section tips

  • Writing tips adapted from Lu et al. (2024b) call for a single-paragraph abstract and an introduction covering motivation, difficulty, contributions, and experimental verification.The introduction should explicitly list contributions and may discuss future work when space permits.Source: §B.9.1 · pp. 48–49
  • Technical-section tips require comparisons with related work, explicit problem formalism and assumptions, precise mathematical methods, and experimental setup details without invented hardware specifications.Applicable related methods should receive experimental comparisons; otherwise, their inapplicability needs explanation. Setup descriptions should include datasets, evaluation metrics, important hyperparameters, and implementation details.Source: §B.9.1 · pp. 49–50
  • Results should connect the method to the stated problem and experimental setup, including hyperparameters and potential fairness issues.Discussion guidance calls for a brief recap of the paper and frames future work as potential academic offspring.Source: §B.9.1 · p. 50
  • Results must report only experiments actually run and logged, give numerical findings, and include all experimental results and relevant figures.When results exist, compare against baselines, include statistics and confidence intervals, and provide ablations showing individual components’ relevance. Discuss methodological limitations.Source: §B.9.1 · p. 50

B.9.2. paper-solver Reviewer prompt

  • The reviewer prompt, taken from Lu et al. (2024b), directs the reviewer to be critical and cautious.Given the research plan and generated LaTeX, the reviewer first records paper-specific evaluation reasoning rather than generic comments. Scores must be reduced when abstract, introduction, methods, results, or discussion sections are missing.Source: §B.9.2 · pp. 50, 51, 55, 56
  • The review must use precisely formatted JSON containing a summary, strengths, weaknesses, ratings, questions, limitations, ethical concerns, and an Accept-or-Reject decision.Originality, Quality, Clarity, and Significance use ratings from 1 to 4, as do Soundness, Presentation, and Contribution. Overall uses 1 to 10, Confidence uses 1 to 5, and Decision permits only Accept or Reject.Source: §B.9.2 · p. 51
  • Reviewers assess originality, technical quality, clarity, and significance, examining novelty, evidential support, reproducibility, and the importance of contributions.The summary should describe contributions without critique; strengths and weaknesses should assess how the work differs from prior contributions and whether its claims are supported.Source: §B.9.2 · p. 52
  • Reviewers should ask actionable clarifying questions, constructively assess limitations and negative societal impacts, and flag ethical issues for ethics review.Questions should target responses that could change the assessment or resolve confusion. Authors should be rewarded rather than punished for openly acknowledging limitations and potential negative impacts.Source: §B.9.2 · pp. 52–53

C. Survey questions · C.1. Grading Research Report Autonomous Mode Preselected Topics · Grading Research Report #1

The survey asks readers to evaluate an AI-generated report on whether gender role play affects language-model math accuracy as a foundation for further research. It combines quality and usefulness ratings with a NeurIPS 2025 reviewer-style assessment.

Grading Research Report #1

  • Readers assess the AI-generated report as a foundation for their own research, rather than comparing it with research performed by a human.Readers must read the entire report. The assistant independently searched literature, coded and executed experiments, collected data, analyzed findings, and wrote the report on gender role play and language-model math accuracy.Source: Grading Research Report #1 · p. 57
  • Experimental results receive quality ratings from 1 - Very Low Quality to 5 - Very High Quality.These are requested survey ratings, not reported evaluation outcomes.Source: Grading Research Report #1 · p. 58
  • Writing quality and assistant usefulness receive separate ratings.Writing quality ranges from 1 - Very Low Quality to 5 - Very High Quality; usefulness ranges from 1 - Very Low Usefulness to 5 - Very High Usefulness.Source: Grading Research Report #1 · pp. 58–59
  • From a NeurIPS 2025 reviewer perspective, respondents assess technical quality and clarity, including evidential support, methodological appropriateness, completeness, organization, and reproducibility.Quality also considers whether authors carefully and honestly evaluate their work’s strengths and weaknesses.Source: Grading Research Report #1 · pp. 59–60
  • Significance and contribution assess importance, usefulness to others, advances over prior work, originality, and value to the broader NeurIPS community.Significance includes unique data, conclusions, or approaches; contribution considers the importance of the research questions and originality of ideas or execution.Source: Grading Research Report #1 · pp. 61–62
  • Soundness assesses technical claims, methodology, and supporting evidence; presentation assesses writing style, clarity, and contextualization relative to prior work.Both dimensions request numerical ratings.Source: Grading Research Report #1 · pp. 61–62
  • Respondents provide an overall submission score and a separate binary Accept or Reject decision.Shown overall-score choices include 8: Strong Accept, 7: Accept, 4: Borderline reject, 3: Reject, and 2: Strong Reject.Source: Grading Research Report #1 · pp. 63–64
  • Respondents also report confidence in their assessment, distinguishing an educated guess from absolute certainty backed by familiarity and careful checking.Confidence 1 denotes an educated guess with unchecked details; confidence 5 denotes absolute certainty, strong familiarity with related work, and carefully checked details.Source: Grading Research Report #1 · p. 64

C.2. Co-Pilot Grading Research Report Preselected Topics

This section specifies a preselected-topic evaluation of Agent Laboratory as a research co-pilot, combining user-experience and report-quality questions with NeurIPS 2025-style paper review. Participants judge the output as a foundation for their research rather than against unaided human work.

Preselected-topic evaluation

  • Participants run Agent Laboratory on a preselected topic, read its complete PDF report, and judge whether the output provides a useful foundation for their research.Choices cover language-model cognitive biases, differential diagnosis for MedQA accuracy, word-order sensitivity in multiple-choice benchmarks, gender role-play effects on math accuracy, and image-transformer versus convolutional-network pixel-noise sensitivity. The assistant performs literature search, experimentation, coding, execution, data collection, analysis, and report writing; participants assess usefulness for further research rather than equivalence to unaided human work.Source: §C.2 · p. 66

User experience

  • User-experience questions assess project-building ease, enjoyment, research usefulness, and likelihood of using Agent Laboratory again.Participants can optionally suggest improvements for their research.Source: §C.2 · pp. 67–68

Report assessment

  • Participants separately assess experimental-result quality, report-writing quality, and the usefulness of the AI assistant presented in the report.Each assessment uses a 1–5 rating scale.Source: §C.2 · pp. 69–70

Reviewer assessment

  • Participants adopt a NeurIPS 2025 reviewer perspective to assess the paper’s technical quality, clarity, and significance.Quality considers supporting evidence, methodological appropriateness, completeness, and acknowledgment of strengths and weaknesses. Clarity considers organization and sufficient information for expert reproduction, while significance asks whether results are important.Source: §C.2 · pp. 70–71
  • The NeurIPS 2025 reviewer assessment also requests an overall score, evaluation confidence, an Accept/Reject decision, and optional additional feedback.Source: §C.2 · pp. 70, 73, 74
  • Additional review dimensions assess soundness of claims and methodology, presentation quality, and the paper’s overall research contribution.Soundness concerns evidential support; presentation includes writing, clarity, and positioning relative to prior work. Contribution considers question importance, originality of ideas or execution, and value to the NeurIPS community.Source: §C.2 · pp. 72–73

C.3. Co-Pilot Grading Research Report Custom Topics · Co-Pilot Grading Research Report

The custom-topic co-pilot assignment evaluates Agent Laboratory as a foundation for users’ own research, rather than a replacement for researchers. Its questionnaire covers user experience, report quality, and conference-style paper assessment.

C.3. Co-Pilot Grading Research Report Custom Topics

  • Users build and run Agent Laboratory on a chosen machine-learning question, read the resulting PDF completely, and provide their exact research question.The research topic can be anything machine-learning related.Source: Co-Pilot Grading Research Report · p. 76

Co-Pilot Grading Research Report

  • Users should judge whether Agent Laboratory provides useful foundations for their research, rather than compare its output with independently conducted human research.The assistant performs literature searches, coding, code execution, data collection, analysis, and report writing.Source: Co-Pilot Grading Research Report · p. 76
  • User-experience questions assess project-building ease, enjoyment, research usefulness, and likelihood of using Agent Laboratory again.An optional question asks how the system could be improved for the respondent’s research.Source: Co-Pilot Grading Research Report · pp. 77–78
  • Five-point scales assess experimental-result quality, report-writing quality, and the perceived usefulness of the AI assistant.These questions distinguish the quality of the research outputs from the usefulness of the tool.Source: Co-Pilot Grading Research Report · pp. 79–80
  • Participants adopt a NeurIPS 2025 reviewer perspective to assess technical quality and clarity.Quality considers supporting evidence, methodological appropriateness, completeness, and honest evaluation; clarity considers organization and sufficient information for an expert to reproduce results.Source: Co-Pilot Grading Research Report · pp. 80–81
  • Significance and contribution assessments examine research importance, usefulness to others, advances over prior work, originality, and value to the broader research community.Significance includes unique data, conclusions, or approaches; contribution includes the importance of the questions and originality of ideas or execution.Source: Co-Pilot Grading Research Report · pp. 81, 83
  • Separate soundness and presentation ratings examine evidential support for technical claims and methodology, alongside writing clarity and contextualization relative to prior work.Soundness explicitly asks whether the paper’s central claims are adequately supported.Source: Co-Pilot Grading Research Report · p. 82
  • The review concludes with an overall score, assessment confidence, a binary accept-or-reject decision, and optional additional feedback.Confidence records how certain the respondent is about their evaluation.Source: Co-Pilot Grading Research Report · pp. 83–84

Related papers

Loading 2501.04227v2…