MLR-Copilot: Autonomous Machine Learning Research based on Large Language Models Agents

1 Introduction

• Traditional research workflows require labor-intensive literature review, method formulation, experiment design, implementation, and execution, under time pressure that can increase decision errors.The paper presents LLM agents as an opportunity to support phases of this workflow and improve efficiency and productivity.Source: §1 · p. 1
• Prior LLM research efforts often address either idea generation or auto-experimenting, with limits including broad ideas, predefined tasks, small code edits, and weak error feedback.The cited idea-generation efforts are not tailored to ML research, while auto-experimenting efforts commonly start from a predefined task and mature code template.Source: §1 · p. 2
• MLR-COPILOT takes research papers and extracted problems as input, then uses IdeaAgent and ExperimentAgent to generate research plans, implement experiments, and run them with execution and debugging feedback.Optional human feedback supports refinement of experiment implementations.Source: §1 · p. 2
• Evaluations and case studies on five ML research papers/problems indicate that MLR-COPILOT can generate novel, feasible hypotheses and complete the research process with significant results and conclusions.The authors report manual and automatic evaluations of hypotheses and experimental executions/results.Source: §1 · p. 2

2 MLR-COPILOT Framework

• MLR-COPILOT organizes research-idea generation, experiment implementation, and implementation execution into three integrated phases using LLM agents.IdeaAgent generates research methodologies and experimental plans; ExperimentAgent implements and executes experiments.Source: §2 · p. 2, Figure 2 · p. 3

2.1 Research Idea Generation

• IdeaAgent uses paper content and recent findings to generate a methodology and experiment plan, with supervised fine-tuning and reinforcement learning guided by novelty, feasibility, and effectiveness feedback.The supervised fine-tuning dataset is derived from 1,000 papers and extracted research ideas and plans.Source: §2.1 · p. 3

2.2 Experiment Implementation

• ExperimentAgent adapts prototype code from the original paper and can retrieve task-appropriate models and datasets before integrating them into an executable experimental setup.Retrieved datasets are checked for alignment with experimental requirements.Source: §2.2 · pp. 3–4

2.3 Implementation Execution

• ExperimentAgent runs experiments, monitors their progress and performance, and supports human input and iterative debugging so implementations can be refined during execution.Execution results can also inform refinement of methodologies and experimental designs.Source: §2.3 · p. 4

3.1 Experimental Setup and Datasets

• Five tasks span semantic relatedness, feedback prediction, sentiment, survival, and contrail classification, using Pearson correlation, MCRMSE, and accuracy as metrics.The tasks include SemRel, ELLIPSE feedback, IMDB, Spaceship-Titanic, and Identify-Contrails.Source: §3.1 · p. 4

3.2 Evaluation and Results

• The evaluation separately assesses hypothesis generation and experiment implementation and execution, using BaseLLM and ResearchAgent as idea-generation baselines.Five domain experts and GPT-4 assess hypotheses and experimental designs on specified criteria using five-point scales.Source: §3.2 · p. 4, §3.2.1 · p. 4

3.2.1 Research Idea Generation

• IdeaAgent outperforms BaseLLM and ResearchAgent in manual and automated assessments, and its hypotheses have lower similarity to existing ones.The reported comparisons cover generated hypotheses and experimental designs.Source: Table 1 · p. 4, §3.2.1 · p. 5, Table 2 · p. 5

3.2.2 Experiment Implementation and Implementation Execution

• Execution experiments measure average task-performance improvement and success across eight trials with human instructions, comparing against one-pass code modification without iterative feedback.Each assessment starts from retrieved state-of-the-art prototype code.Source: §3.2.2 · p. 5
• Claude-3.7 achieved the highest average improvement and 50.0% success, versus GPT-4 at 40% and Claude-2.1 at 27.5%; GPT-4 and Claude both outperform 1-Prompt.Table 4 defines success as at least 10% improvement over the SOTA prototype; Table 3 reports average percentage improvement.Source: Table 3 · p. 5, Table 4 · p. 5, §3.2.2 · p. 6

• The case study applies the framework to generate hypotheses and run sentiment-analysis experiments on ELLIPSE.The action sequences illustrate the system's use in conducting experiments.Source: §4 · p. 6
• ExperimentAgent interacts with the Action Executor and Utility Modules while inspecting scripts, executing models, retrieving models, and analyzing results.Source: §4 · p. 6
• Reasoning, objectives, observations, and feedback guide actions that refine the model and experimental design.Researchers can understand, modify, and execute scripts in the process.Source: §4 · p. 6

• Earlier systems explore literature-based idea generation or AutoML, while MLAgentBench focuses on predefined tasks and existing codebases without interaction; MLR-COPILOT supports broader ML hypothesis generation.The cited idea-generation efforts include Zhong et al. (2023), Qi et al. (2023b), Yang et al. (2023), Wang et al. (2024), and Li et al. (2024b).Source: §5 · p. 6
• AI SCIENTIST introduced a similar staged autonomous-research design with paper writing and reviewing; subsequent systems add tree-search, role specialization, multiagent collaboration, or expert feedback.The passage describes these later developments, including AI-SCIENTIST V2, AGENTLAB, AI-RESEARCHER, AI CO-SCIENTIST, RESEARCHTOWN, CODESCIENTIST, and DEEPREVIEW.Source: §5 · p. 6
• MLR-COPILOT uses natural-language model and data retrieval to translate research questions into model requirements and retrieve models for hypothesis testing and validation.The passage identifies Hugging Face’s Datasets and Model Hub as repositories supporting retrieval.Source: §5 · p. 6

• MLR-COPILOT automates machine-learning research using LLM agents.Source: §6 · p. 6
• The framework generates research ideas, implements and executes experiments, and refines implementations using automatic and human feedback.Source: §6 · p. 6
• Domain experts highlight MLR-COPILOT as a powerful tool for research idea generation and experimentation.Source: §6 · p. 6

• The pipeline treats its stages largely sequentially, while failed or suboptimal experiments may call for revising the original hypotheses.The authors identify backward transitions from implementation or execution to ideation as a closer fit to iterative discovery.Source: §7 · p. 7
• Usability and accessibility remain limited, especially for researchers with little experience in LLM prompting or code debugging.Source: §7 · p. 7
• The authors suggest a better interface and more intuitive interactions to broaden adoption among ML practitioners.Source: §7 · p. 7

A Details of Case Study: Sentiment Analysis Research

• The appendix summarizes the running logs and notes that full logs are available in the repository.Source: A Details of Case Study: Sentiment… · p. 10
• The prompts guide hypothesis and experiment generation, emphasizing clarity, innovation, validity, and reproducibility.They draw on the research problem, existing studies, and relevant entities.Source: A Details of Case Study: Sentiment… · p. 10

Generated Research Idea

• The generated idea proposes aspect-level sentiment analysis using a hybrid CNN, BiLSTM, and BERT model with enhanced linguistic features.It also proposes additional data collection and annotation refinement, cross-validation against baselines, and iterative improvement.Source: Generated Research Idea · p. 10
• The training script preprocesses and splits data, trains a BERT-based regression model, and validates it; the evaluation script loads parameters and calculates test metrics.The scripts also generate submissions and are described as streamlining training, evaluation, and deployment.Source: Generated Research Idea · p. 10

B.1 Dataset Analysis

• Figures 4 and 5 provide an overview of the IdeaAgent training data distribution and top 10 keywords, while Figure 4 gives rating-distribution statistics for the dataset.Source: §B.1 · p. 10, Figure 4 · p. 10

B.2 Definition of Novelty, Feasibility, and Effectiveness

• Novelty, feasibility, and effectiveness are the three dimensions used to evaluate research ideas and provide feedback for IdeaAgent training.Source: §B.2 · p. 10

B.2.1 Novelty

• Novelty measures how different an idea is from existing work, with the rubric spanning ideas identical to existing work to highly different and creative ideas.Source: §B.2.1 · pp. 10–11

B.2.2 Feasibility

• Feasibility measures whether an idea can be executed within 1–2 months, assuming ample OpenAI/Anthropic API access and limited GPU resources.The scale ranges from impossible to easy to implement.Source: §B.2.2 · p. 11

B.2.3 Effectiveness

• Effectiveness scores the likelihood that an idea will achieve meaningful experimental performance improvement, from extremely unlikely to definitely effective.Source: §B.2.3 · p. 11