Source-linked AI summary
Agent Laboratory: Using LLM Agents as Research Assistants
Samuel Schmidgall, Yusheng Su, Ze Wang, Ximeng Sun, Jialian Wu, Xiaodong Yu, Jiang Liu, Michael Moor, Zicheng Liu, Emad Barsoum
TL;DR
Researchers face time and resource constraints that leave many ideas unexplored. Agent Laboratory addresses this by taking a human research idea through literature review, experimentation, and report writing with autonomous language agents and optional human feedback. The framework reports strong backend, co-pilot, MLE-Bench, and cost results, while acknowledging limitations in self-evaluation and ethical misuse risks.
Problem
Time and resource constraints limit how many research ideas scientists can explore, leaving many high-quality ideas unexplored.
Method
Agent Laboratory is an open-source, human-centered LLM-agent framework that takes a human research idea through literature review, experimentation, and report writing to produce a report and code repository.
Results
Human feedback produced higher-quality outputs than autonomous mode, while o1-preview was rated most useful and the mle-solver achieved higher consistency and scoring than other solvers on selected MLE-Bench challenges.
Takeaways & Limitations
Agent Laboratory may help researchers spend less time on repetitive coding and writing and more time on creative, conceptual research work.
Takeaways & Limitations
The paper-solver relies on LLM-based reviewer scores despite evidence that LLM self-evaluation agrees less reliably with human reviewers, and the platform raises potential misuse risks without safeguards.
Abstract
from arXiv · showhide
Historically, scientific discovery has been a lengthy and costly process, demanding substantial time and resources from initial conception to final results. To accelerate scientific discovery, reduce research costs, and improve research quality, we introduce Agent Laboratory, an autonomous LLM-based framework capable of completing the entire research process. This framework accepts a human-provided research idea and progresses through three stages--literature review, experimentation, and report writing to produce comprehensive research outputs, including a code repository and a research report, while enabling users to provide feedback and guidance at each stage. We deploy Agent Laboratory with various state-of-the-art LLMs and invite multiple researchers to assess its quality by participating in a survey, providing human feedback to guide the research process, and then evaluate the final paper. We found that: (1) Agent Laboratory driven by o1-preview generates the best research outcomes; (2) The generated machine learning code is able to achieve state-of-the-art performance compared to existing methods; (3) Human involvement, providing feedback at each stage, significantly improves the overall quality of research; (4) Agent Laboratory significantly reduces research expenses, achieving an 84% decrease compared to previous autonomous research methods. We hope Agent Laboratory enables researchers to allocate more effort toward creative ideation rather than low-level coding and writing, ultimately accelerating scientific discovery.
1. Introduction
Agent Laboratory is introduced to help researchers explore more machine-learning ideas by combining human research direction with autonomous language-agent workflows. The framework produces research reports and code while evaluating model backends, human feedback, research quality, and cost.
- Scientists prioritize research ideas under time and resource constraints, leaving many high-quality ideas unexplored.
- Agent Laboratory accepts a human research idea and uses language agents to produce a research report and code repository with adjustable human involvement.Feedback can be provided at a frequency determined by the user.
- Human evaluators found o1-preview most useful, while o1-mini achieved the highest experimental quality and gpt-4o trailed across metrics.
- Automated evaluations overestimated research quality, scoring 6.1/10 overall versus 3.8/10 from human reviewers.
- Co-pilot papers received higher overall scores than autonomous-mode papers, although experimental quality and usefulness involved trade-offs related to researcher-intent alignment.
- Agent Laboratory reduced research cost to $2.33 USD per paper with a gpt-4o backend and achieved state-of-the-art performance on a subset of MLE-Bench challenges.
2. Background & Related Work
Prior work shows broad progress in LLM-assisted research, agents, automated machine learning, and scientific discovery, but ideation and human-AI collaboration remain unsettled. These limitations motivate systems that combine human guidance with LLM-based research workflows.
- LLM agents augment language models with structured frameworks, iterative refinement, self-improvement, and external tools for complex task execution.
- Automated machine-learning benchmarks use Kaggle challenges to assess agents on data preparation, model development, submissions, feature implementation, bug fixing, and refactoring.
- LLMs support code generation, software development, literature search, paper reviewing, and experiment-outcome prediction across research tasks.
- LLM ideation findings are mixed, ranging from greater novelty to reduced creativity and more homogeneous ideas that may limit creative discovery without human guidance.
- Human-AI collaboration research also reaches mixed conclusions about idea novelty, supporting human-guided ideation combined with LLM workflows.
- Autonomous research systems have extended LLM use to scientific ideation, experimentation, coding, and automated paper generation across chemistry, biology, and other domains.
- Agent Laboratory’s workflow combines literature review, experimentation, and report writing with human input and specialized tools such as mle-solver and paper-solver.
3. Agent Laboratory
Agent Laboratory organizes research into literature review, planning and preparation, experimentation, and report writing, using specialized agents and iterative tool-based refinement. Its solvers retrieve literature, generate and repair code, score programs, edit reports, compile LaTeX, and review outputs.
- Overview: The workflow proceeds through Literature Review, Experimentation, and Report Writing, beginning with paper collection and analysis and ending with automated experiments and report generation.
- Literature Review: The literature-review phase uses arXiv retrieval, summaries, full text, and curated paper selection to build references for later stages.
- Plan Formulation: PhD and Postdoc agents collaborate to formulate an actionable plan specifying models, datasets, and high-level experimental steps.
- Data Preparation: The ML Engineer prepares data with Python and HuggingFace datasets, while the SW Engineer submits finalized preparation code.
- Running Experiments: mle-solver iteratively generates, executes, repairs, scores, and self-reflects on machine-learning programs based on the research plan.
- Running Experiments: Performance stabilization maintains top-scoring programs and applies parallel modifications to balance exploration with refinement.
- Report Writing: paper-solver builds LaTeX report scaffolds, optionally researches through arXiv, edits sections, compiles changes, and evaluates reports with reward-based review.
- Report Writing: Example reviews assess generated reports using strengths, weaknesses, originality, quality, clarity, significance, questions, and limitations.
4. Results
Agent Laboratory’s evaluations show backend- and topic-dependent quality, with o1-preview strongest for usefulness and o1-mini strongest for experimental quality. Human guidance improves external evaluations, but autonomous and co-pilot papers remain below the reported NeurIPS acceptance average.
- Backend evaluation: o1-preview received the highest usefulness rating at 4.4/5, while o1-mini achieved the highest experimental quality at 3.2/5.o1-preview also led report quality at 3.4/5; o1-mini’s report quality was 3.2/5.
- Topic effects: Topic choice affected results, with word order reaching 3.8/5 report quality and 4.5/5 usefulness, while cognitive bias reached 3.2/5 experimental quality.Image noise showed substantial backend variation, from 1.5/5 experimental quality with gpt-4o to 4.0/5 with o1-mini.
- Human versus automated review: Automated reviewers overestimated generated-paper quality: their average overall score was 6.1/10 versus substantially lower human assessments.For clarity, automated reviewers averaged 3.6/4 compared with 2.4/4 from human evaluators.
- Co-pilot evaluation: Co-pilot self-evaluations averaged 4.13/10 versus 3.8/10 for autonomous papers, while external evaluations rose further to 4.38/10.External reviewers increased quality by +0.62 but decreased clarity by -0.25.
- Co-pilot evaluation: Co-pilot papers scored higher than autonomous papers externally, improving overall score by +0.58, quality by +0.75, and soundness by +0.48.The co-pilot overall average was 4.38, still -1.45 points below the reported 5.85 NeurIPS 2024 acceptance average.
5. Limitations
Agent Laboratory’s limitations include weaknesses in self-evaluation, rigid workflow structure, code and execution management, hallucinated results, and broader ethical risks. These constraints affect evaluation reliability, output flexibility, runtime robustness, and responsible deployment.
- Challenges with self-evaluation: LLM-based self-evaluation can diverge from human rankings, limiting assessment of research ideas and reports.The framework uses LLM-emulated NeurIPS reviewers, although LLMs may rely on superficial patterns rather than robust evaluation criteria.
- Challenges with automated structure: The fixed workflow structure restricts paper organization and limits mle-solver and paper-solver to generating two figures.Future work could incorporate figures generated by mle-solver without this restriction.
- Challenges with automated structure: Agent Laboratory cannot independently manage repository-level code because files are supplied and saved separately by workflow phase.Flexible repository-level file modification and execution is identified as a next step.
- Common failure modes: Runtime failures include instruction-following breakdowns, token-limit terminations, 0% experimental accuracy, unsafe process termination, and unguarded host commands.These failures were observed across literature review and experimentation, including models such as gpt-4o, o1-mini, and o1-preview.
- Ethical considerations: Autonomous generation may lower barriers to substandard or misleading outputs, amplify bias, and enable unethical misuse without robust governance.The paper calls for transparent disclosure of AI involvement and safeguards aligned with ethical research standards.
6. Discussion
Agent Laboratory combines autonomous research agents with adjustable human oversight in a continuous machine-learning research pipeline. Its evaluations show varied backend performance, stronger co-pilot results, reduced costs, and a need for longer-term assessment.
- Human-centered evaluation: Human feedback in co-pilot mode produced higher-quality outputs than autonomous mode across most metrics, while researchers rated the feature highly usable and useful.Most participants decided to continue using the feature after their experience.
- Backend evaluation: Performance varied across language-model backends: o1-preview was rated most useful, whereas o1-mini achieved the highest experimental quality.Autonomous outputs were generally well received but showed gaps in clarity and soundness relative to human expectations.
- Evaluation: Automated reviewer scores did not predict human reviewer scores, demonstrating the importance of human evaluations in assessing research quality.The study therefore compares automated and human-centered assessments rather than relying on reviewer automation alone.
- Efficiency: Agent Laboratory offers automatic research at greatly reduced cost, with the reported cost reaching only $2.33 USD per paper for a model backend.The paper also reports runtime and cost statistics broken down by research phase and backend.
- Pipeline and positioning: Agent Laboratory integrates literature review, experimentation, and report writing into one continuous pipeline that adapts to researcher interaction and compute availability.The framework is designed to handle tedious tasks while researchers focus on conceptual design and critical thinking.
- Limitations and future work: The authors position Agent Laboratory as complementary to researchers because prior work identifies feasibility and implementation limitations for LLM-based ideation.They propose longitudinal studies comparing research outcomes with and without the framework because current human evaluations provide only a snapshot of utility.
A.1. Hyperparameters
Table 1 lists the hyperparameters used for Agent Laboratory.
- Table 1 presents the hyperparameters for Agent Laboratory.
A.2. Hardware
All experiments were run on a 2023 MacBook Pro equipped with an Apple M3 Max processor and 36 GB of memory.
- All experiments ran on a 2023 MacBook Pro with an Apple M3 Max processor and 36 GB of memory.
B. Prompts
The prompts coordinate agents across literature review, planning, data preparation, experimentation, interpretation, and report refinement. They encode task objectives, dialogue-based supervision, iterative context, command formats, and reviewer criteria.
- B.1. Base Inference Prompt: Base inference prompts provide each agent’s role, phase instructions, command descriptions, history, feedback, notes, and previous command context.They also require a single new command and can append a completion string when progress reaches 70%.
- Workflow integration: The prompts collectively operationalize a staged workflow in which agents gather literature, formulate plans, prepare data, execute experiments, interpret results, and write reports.
- B.2. Context Prompts: Context prompts pass current literature, plans, datasets, experiment code, results, interpretations, previous reports, and reviewer responses between workflow stages.Separate prompts support report refinement, plan formulation, data preparation, and results interpretation.
- B.3.1. PhD Student phase: The PhD-student prompts assign literature review, planning, data preparation, and results interpretation as supervised, dialogue-based research tasks.They emphasize simple experiments, literature integration, executable data preparation, and interpretations containing accurate metrics and significance measures.
- B.4. Machine Learning Engineer Phase Descriptions: The machine-learning-engineer prompts direct simple data-preparation code that integrates the literature review and finalized research plan.The engineer can execute Python code and search HuggingFace datasets.
- B.5. Postdoc Phase Descriptions: The postdoc prompts guide PhD students in specifying models, datasets, experimental details, and interpretations through dialogue.Their plans are intended to be simple, actionable, and suitable for achieving the research task.
- B.6.1. PhD Student Command Description: The PhD literature-review command prompt supports semantic summary searches, full-paper retrieval, and adding relevant papers after reading them.It restricts the agent to one command per inference and limits repeated summary searches.
- B.9.2. paper-solver Reviewer prompt: The reviewer prompt asks for a summary plus assessments of originality, quality, clarity, and weaknesses while emphasizing technically supported and honest evaluation.It also notes that some reviews and meta-reviews may be made public after decisions.
Grading Research Report #1
The first grading report asks readers to evaluate an AI-generated research report and assistant across writing, usefulness, experimental quality, and review criteria. It also requests numerical assessments of quality, clarity, significance, soundness, presentation, contribution, overall score, confidence, and decision.
- Evaluation scope: The assignment asks readers to read an AI-generated research report in full and rate its writing quality, usefulness, and experimental results.The report concerns whether gender role play affects language-model accuracy on math questions.
- Review criteria: Reviewers assess the submission’s quality, clarity, significance, soundness, presentation, and contribution using numerical rating scales.The criteria include technical support, organization, reproducibility, importance, originality, and value to the broader NeurIPS community.
- Final judgments: The evaluation form additionally requests overall score, confidence score, and an Accept-or-Reject decision.Confidence ranges from an educated guess to absolute certainty based on familiarity with related work and checking technical details.
C.2. Co-Pilot Grading Research Report Preselected Topics
The preselected-topics survey evaluates Agent Laboratory as a research co-pilot through user ratings and paper-review criteria. It asks participants to choose a machine-learning research topic, assess the resulting work, and judge whether the tool is useful to build upon.
- Topic selection: Participants are asked to select a machine-learning research topic and use Agent Laboratory to build a project.The listed options include questions about cognitive biases, MedQA, word order, gender role play, and robustness to pixel noise.
- Output evaluation: Participants rate the generated report and paper using criteria covering experimental quality, writing quality, technical soundness, clarity, significance, presentation, contribution, confidence, and overall score.The evaluation is framed around whether the output is useful to build from rather than equivalent to unaided human performance.
Co-Pilot Grading Research Report
The co-pilot grading report records participant experience with Agent Laboratory and evaluates the resulting research report. Its review lens emphasizes usefulness as a foundation for further research rather than comparison with unaided human performance.
- Evaluation lens: Reviewers are instructed to judge whether the assistant’s output is useful to build off rather than compare it with what a person would produce alone.This framing treats the assistant as a research foundation intended to accelerate the user’s own research.
- Participant experience: Participants report how easy it was to build a project with Agent Laboratory and how much they enjoyed using it.The form uses five-point scales for ease and enjoyment.
- Research assistance: The survey asks how useful Agent Laboratory was for assisting research and how likely participants are to use it again.Both usefulness and reuse likelihood are recorded on five-point scales.
- Report quality: Participants evaluate the report’s experimental results and writing quality using five-point quality scales.The form separately asks for perceptions of experimental quality and report-writing quality.
- Paper review: The paper review covers technical quality, clarity, significance, soundness, presentation, contribution, overall score, confidence, and decision.The criteria ask whether claims are supported, methods are appropriate, results are reproducible, and contributions are valuable and original.