ML Research Benchmark

1 Introduction

• The benchmark comprises seven recent conference competition tasks spanning AI research activities, and offers a baseline domain-specific agent for machine learning research.Tasks cover activities including pretraining, fine-tuning, pruning, compression, and model merging.Source: §1 · pp. 1–2

2 Related Works

• Prior agent benchmarks address general ML tasks, while the proposed competition benchmark focuses on frontier research challenges beyond canonical ML tasks.MLAgentBench includes 13 tasks such as image classification, segmentation, time-series modeling, and text classification [3].Source: §2 · p. 2

2.1 AI Research and Development Agents

• The domain-specific baseline agent is modular, with research tools intended to extend its machine-learning task capabilities.Related agents include AutoML-GPT [18], MLcopilot [19], HuggingGPT [12], MLAgentBench [3], and AI Scientist [20].Source: §2.1 · p. 2

3 ML Research Benchmark: Utilizing Conference Competitions to Benchmark Agent Performance

• Competition tracks test emerging AI challenges with standardized evaluation and complex, multifaceted criteria, bridging general agent evaluations and AI research demands.The benchmark builds on general software-development and machine-learning benchmarks [3].Source: §3 · p. 2

4 Agent Conference Competition Tasks

• Task prompts provide instructions, applicable data, and an example output, but omit baseline starter code to encourage problem-solving.The benchmark's challenges span subfields and assess skills from efficient training and compression to reasoning and cross-domain generalization.Source: §4 · pp. 2–3

5 Defining Tasks

• Competition tasks permit continuing performance improvement rather than the rapid saturation and binary measurement of conventional benchmarks.Tasks can become progressively harder as the baseline for success rises.Source: §5 · p. 3

6 Tasks Constraints

• Training runs are constrained to one A100 40GB GPU and 24 hours; tasks were selected from recent competitions and evaluated using LM-Evaluation Harness or Agent-Eval methods.Agent-Eval includes memory and throughput measurement and task-specific evaluations such as BLEU and ROUGE for math reasoning.Source: §6 · p. 3, §7 · p. 3

8.1 MiniPile Challenge

• MiniPile asks agents to pretrain a model for SuperGLUE performance within 24 hours, using a dataset with one million training samples, 500 validation samples, and 10,000 test samples.Evaluation averages accuracy across eight SuperGLUE tasks: BoolQ, CB, COPA, MultiRC, ReCoRD, RTE, WiC, and WSC.Source: §8.1.1 · p. 4, §8.1.2 · p. 4, §8.1.3 · p. 4

8.2 LLM-Merging Competition

• LLM merging targets a generalist model evaluated on MMLU subcategories, with merging and evaluation under one hour and no direct MMLU training.Agents use publicly available models up to 8GB, open-source data, and provided example code.Source: §8.2.1 · p. 4, §8.2.2 · p. 4, §8.2.3 · p. 4

8.3 Edge LLMs Challenge: Compression Track

• The Edge LLM compression task compresses microsoft/phi-2 for a 12 GB DRAM device while preserving MMLU performance, without training, quantization, or distillation.Evaluation uses average accuracy across MMLU subcategories and their subjects.Source: §8.3.1 · p. 4, §8.3.2 · p. 4, §8.3.3 · p. 5

8.4 Edge LLMs Challenge: Training from Scratch Track

• The Edge LLM training task requires training from scratch for a 1 GB DRAM device, using only C4 and Alpaca and no pre-trained models or quantization.Models are assessed by average accuracy across eight SuperGLUE tasks.Source: §8.4.1 · p. 5, §8.4.2 · p. 5, §8.4.3 · p. 5

8.5.1 Math Reasoning Goal

• The autoformalization task trains agents to translate natural-language math problems and proofs into Lean 3 formal statements and proofs.The supplied dataset has 3,963 samples; evaluation includes BLEU, ROUGE-L, and the number of scripts compiling into working Lean 3 code out of 100.Source: §8.5.1 · p. 5, §8.5.2 · p. 5, §8.5.3 · p. 5

8.6 NeurIPs 2023 1LLM + 1GPU + 1Day: LLM Efficiency Challenge [27]

• The LLM Efficiency Challenge fine-tunes an approved base model on open-source data using one A100 40GB GPU within 24 hours, excluding MMLU training.The goal is performance on a subset of MMLU while balancing performance improvements with resource constraints.Source: §8.6 · pp. 5–6

8.7 BabyLM Challenge

• BabyLM requires training from scratch on the provided corpus of approximately 10 million words, with no pre-trained models or fine-tuning on BLiMP.Agents may choose their architecture; the resulting model is evaluated on the BLiMP suite, with average accuracy over its subtasks.Source: §8.7.1 · p. 6, §8.7.2 · p. 6, §8.7.3 · p. 6

9 Baseline Agent

• A supervisor manages task instructions and results while worker agents execute tasks, with parallel workers available for efficiency and scalability.The baseline supports OpenAI and Anthropic models and uses their function-calling capabilities.Source: §9 · p. 7, §9.1 · p. 7

9.1 Agent Tools

• Workers use modular tools for Python and Bash, code and GitHub management, and academic-paper search, while recording intermediate reasoning in a ReAct-like process [29].The agent reports task number, run ID, submission status, model path, tokens, interaction turns, and time taken.Source: §9.1 · p. 7, §9 · p. 7

9.2 Agent Environment

• Each agent receives a Docker environment with preinstalled ML packages and evaluation libraries, an NVIDIA A100 40GB GPU, 30 CPU cores, 214.7 GB RAM, and 549.8 GB storage.The environment is provided through Lambda Labs cloud GPU instances.Source: §9.2 · p. 7

10 Results and Discussion

• Claude Sonnet 3.5 outperformed GPT-4o in five of seven challenges, although the results are based on a limited number of runs.Source: §10 · p. 8

10.1 MiniPile Task

• On SuperGLUE, Claude averaged 0.541 vs GPT-4o’s 0.457; GPT-4o succeeded in one of five runs, while Claude succeeded in four of five.Source: §10.1 · p. 8

10.2 LLM Merging Task

• On MMLU, Claude scored 0.273 while GPT-4o produced no valid result.Source: §10.2 · p. 8

10.3 Edge LLM Compression

• On MMLU, Claude scored 0.532 vs GPT-4o’s 0.326, pruning phi-2’s non-zero parameters from 2,779,683,840 to 1,984,822,594 while maintaining accuracy.Source: §10.3 · p. 9

10.4 Edge LLM Training

• On SuperGLUE, Claude scored 0.494 vs GPT-4o’s 0.459; both used GPT-2, and Claude’s winning solution trained on combined C4 and Alpaca data.Source: §10.4 · p. 9

10.5 Math Reasoning

• Across five runs, GPT-4o averaged 0.08725 vs Claude’s 0.02842 on Math Reasoning, but no model from either agent produced compilable code.Source: §10.5 · pp. 9–10

10.6 LLM Efficiency

• On MMLU, Claude scored 0.310 vs GPT-4o’s 0.234; Claude’s best results used an 8-bit-quantized Llama-2 model fine-tuned on OASST1 with LoRA.Source: §10.6 · p. 10

10.7 BabyLM

• On BLiMP, GPT-4o scored 0.535 vs Claude’s 0.527, with both agents producing similar baselines and predominantly using GPT-2.Source: §10.7 · p. 10

• Both agents produced baselines across tasks, but neither demonstrated non-trivial model development or research.Source: §11 · p. 10
• Some runs did not converge within 24 hours, and agents sometimes failed to checkpoint their models.Source: §11 · p. 10
• Claude Sonnet 3.5 generally performed better than GPT-4o across the study’s tasks.Source: §11 · p. 10

12 Limitations and Future Work

• The evaluation uses only five runs per task, limiting statistical significance, and model or scaffold updates may make results outdated.The authors recommend more runs to improve robustness.Source: §12 · p. 10
• The selected tasks may not cover all relevant AI capabilities, motivating broader task coverage and more complex real-world scenarios.The authors propose expanding the task range and testing more complex scenarios.Source: §12 · p. 11
• Experiments used one A100 40GB GPU and a 24-hour task limit; varying either may yield different results, while costs averaged $42.89 per run.The reported total was $300.23 per agent across all tasks, and the authors flag cost-production tradeoffs for future comparisons.Source: §12 · p. 11
• The two agent scaffolds produced baselines on many tasks but did not demonstrate non-trivial model development or research.The authors describe the findings as initial insights into agent performance across AI challenges.Source: §12 · p. 11

13 Conclusion

• The agents produced baselines across many tasks but did not demonstrate non-trivial model development or research.The authors report performance differences between GPT-4o and Claude Sonnet 3.5 across tasks.Source: §13 · p. 11
• The benchmark provides a framework for comparing agents on tasks intended to mirror real-world AI research and development challenges.The authors identify non-trivial research and model development as an area for future development.Source: §13 · p. 11
• Performance differences across tasks underscore the need for versatile systems that can adapt to varied challenges.The authors also call for comprehensive benchmarks that evaluate agent capabilities across diverse tasks.Source: §13 · p. 11

15.1 Supervisor System Prompt

• The supervisor prompt casts the agent as an AI researcher tasked with completing a goal efficiently and effectively.It directs the agent to use available tools, prefer coding, track progress in a scratchpad, and save the model before reporting completion.Source: §15.1 · p. 14
• The supervisor prompt encourages immediate execution, trying new approaches after repeated failures, and finding the working directory before starting.It also tells the agent to make assumptions when necessary and complete tasks sequentially or combine them toward the goal.Source: §15.1 · p. 14

15.2 Worker System Prompt

• The worker prompt supplies the task goal, working directory, elapsed and remaining time, plan outline, recent actions, and prior attempt outputs or errors.The provided template includes the last ten actions and fields for prior attempts, outputs, and errors.Source: §15.2 · p. 15
• The worker instructions require recording key information, reporting an evaluation metric after training, and saving the model before returning it.They also tell the worker to prioritize necessary coding and consider available compute and time.Source: §15.2 · p. 15

15.3 Worker Prompt

• The task-specific prompt template specifies the goal, dataset and its shape, allowed models, compute, time limit, expected result, evaluation, and rules.The task is presented to the worker as a task description followed by task details.Source: §15.3 · p. 15
• The worker is told to optimize the rubric score using the specified task conditions and to provide the best possible model.The prompt notes installed packages and that evaluation follows additional conditions supplied in the task.Source: §15.3 · p. 15

1LLM + 1GPU + 1Day: LLM Efficiency Challenge

• The LLM Efficiency Challenge fine-tunes an approved base model on open-source data within 24 hours on one A100 40GB GPU, targeting a subset of MMLU metrics.MMLU is for evaluation only, and the evaluation data is not available to agents.Source: §16 · p. 16

BabyLM Challenge (Strict-Small)

• BabyLM requires training from scratch on an approximately 10-million-word corpus and evaluates performance on BLiMP.Agents must use the provided dataset and cannot fine-tune a pretrained model or train directly on BLiMP.Source: §16 · p. 17

MiniPile Challenge

• MiniPile asks agents to pretrain a language model on the MiniPile dataset within 24 hours and maximize SuperGLUE performance.The task permits a model of the agent’s choosing.Source: §16 · p. 17

LLM-Merging Competition Task

• LLM-Merging asks agents to merge publicly available models up to 8 billion parameters into a generalist model evaluated on CosmosQA and XSUM.Agents must use the provided example code; MMLU is evaluation-only, and merging, fine-tuning, and evaluation must take under one hour.Source: §16 · p. 18

Edge LLMs Challenge: Compression

• Edge LLMs Compression requires compressing microsoft/phi-2 to run on a 12 GB DRAM device while maximizing MMLU performance.Only compression is allowed; training, quantization, and distillation are prohibited.Source: §16 · p. 19

Edge LLMs Challenge: Training from Scratch

• Edge LLMs Training requires training a model from scratch to run on a 1 GB DRAM device and maximize SuperGLUE performance.Only C4 and Alpaca datasets are allowed, and quantization and pretrained LLMs are prohibited.Source: §16 · p. 19

ICML 2024 Challenges on Automated Math Reasoning: Track 1-1 Auto-formalization

• ICML 2024 Auto-formalization asks agents to generate Lean 3 statements and proofs from natural-language problem statements and proofs.Open-source models and datasets are allowed, and agents must evaluate on part of the provided data.Source: §16 · p. 20

• The available code includes the baseline agent implementation and evaluation scripts.Source: §17 · p. 20
• Researchers are encouraged to use, adapt, and build upon the framework.Source: §17 · p. 20

• Approximately 1,295 GPU-hours were used for the study.Source: §18 · p. 20
• Experiments ran on NVIDIA A100 40GB GPUs on Lambda Labs.Source: §18 · p. 20

• The authors warn that work assessing AI R&D capabilities could potentially contribute to accelerating AI development.Source: §19 · p. 20
• They identify fast takeoff scenarios as a concern if AI systems rapidly improve themselves or create more advanced systems.Source: §19 · p. 20
• The authors state that potential risks require thoughtful consideration and mitigation strategies.Source: §19 · p. 20

20 Example Run: BabyLM

• The loaded BabyLM dataset has train, dev, and test splits, each reported as six rows with filename and content features.Source: §20 · p. 21
• The agent configures a GPT-2 language model and scripts three training epochs at a learning rate of 5e-5, but no outcome is shown.A later script uses Hugging Face Trainer with three epochs and saves the model; the passages provide no reported training or benchmark metrics.Source: §20 · pp. 22–25
• The pipeline tokenizes the text with GPT-2, pads sequences to 1,024 tokens, and prepares shuffled training batches of four.Source: §20 · p. 22

• The GPT-2 tokenizer lacked a padding token, so the code set pad_token to eos_token.Source: §20 · pp. 25, 27
• Training logs report train_loss 3.453526814778646 and epoch 3.0, with train_runtime 4.4333 on one run and 5.1055 on a later run.Source: §20 · p. 25
• After identifying a missing labels column, the tokenization code copied input_ids into labels.Source: §20 · p. 29

20 Example Run: BabyLM

• 9.997876167297363 to 9.634803771972656 eval_loss across epochs 1.0 to 3.0 shows improvement during the initial training run.The run reported train_runtime of 4.434 seconds.Source: §20 · p. 31
• 0.0100 acc on blimp_matrix_question_npi_licensor_present and 0.9770 acc on blimp_sentential_negation_npi_licensor_present illustrate the task-level spread.Both are version 1, zero-shot task results.Source: §20 · p. 33
• 0.5226 ± 0.0018 was the BLiMP version 2 overall acc with no filter.The reported overall result is for the aggregate BLiMP task.Source: §20 · pp. 32–34