PaperlayerView exact revision on arXiv

Source-linked AI summary

MLAgentBench: Evaluating Language Agents on Machine Learning Experimentation

Qian Huang, Jian Vora, Percy Liang, Jure Leskovec

arXiv:2310.03302v2cs.LGcs.AI

TL;DR

Machine learning experimentation requires iterative method design, execution, evaluation, and revision, motivating the question of whether language-model agents can perform it autonomously. The paper introduces MLAgentBench and a prompting-based agent for 13 executable tasks, finding Claude v3 Opus achieves the best success rate at 37.5% on average, while performance varies sharply across tasks. The authors identify long-horizon planning and hallucination as key challenges and note that reliable success remains limited.

  • Problem

    Machine learning experimentation requires extensive prior knowledge, functional code, and interpretation of results, motivating evaluation of whether language-model agents can conduct it autonomously.

  • Method

    The paper introduces MLAgentBench, a 13-task benchmark with executable environments, and a prompting-based agent that generates plans, reasoning, and actions while interacting with workspaces.

  • Results

    37.5% average success rate is achieved by the Claude v3 Opus agent, the best success rate reported across the evaluated agents, with results ranging from 100% on house-price to 0% on BabyLM.

  • Takeaways & Limitations

    The benchmark demonstrates feasibility of language-model agents for machine learning experimentation while showing that they do not yet succeed reliably across tasks.

  • Takeaways & Limitations

    The authors recommend close human supervision because autonomous agents can arbitrarily modify and run pipelines, producing unpredictable outcomes such as dangerous system code.

Abstract

from arXiv · show

A central aspect of machine learning research is experimentation, the process of designing and running experiments, analyzing the results, and iterating towards some positive outcome (e.g., improving accuracy). Could agents driven by powerful language models perform machine learning experimentation effectively? To answer this question, we introduce MLAgentBench, a suite of 13 tasks ranging from improving model performance on CIFAR-10 to recent research problems like BabyLM. For each task, an agent can perform actions like reading/writing files, executing code, and inspecting outputs. We then construct an agent that can perform ML experimentation based on ReAct framework. We benchmark agents based on Claude v1.0, Claude v2.1, Claude v3 Opus, GPT-4, GPT-4-turbo, Gemini-Pro, and Mixtral and find that a Claude v3 Opus agent is the best in terms of success rate. It can build compelling ML models over many tasks in MLAgentBench with 37.5% average success rate. Our agents also display highly interpretable plans and actions. However, the success rates vary considerably; they span from 100% on well-established older datasets to as low as 0% on recent Kaggle challenges created potentially after the underlying LM was trained. Finally, we identify several key challenges for LM-based agents such as long-term planning and reducing hallucination. Our code is released at https://github.com/snap-stanford/MLAgentBench.

1. Introduction

MLAgentBench addresses whether language-model agents can conduct machine learning experimentation autonomously, an iterative process requiring domain knowledge, functional code, and result interpretation. It introduces a 13-task benchmark and a prompting-based agent, with Claude v3 Opus achieving the strongest reported success rate but substantial variation across tasks.

  • Machine learning experimentation iteratively develops methods, runs experiments, evaluates outcomes, and revises methods to improve performance.
  • MLAgentBench is proposed as the first benchmark for evaluating agents capable of end-to-end machine learning experimentation.Tasks provide descriptions, starter files, evaluators, and workspace actions; agents are evaluated for competence and efficiency.
  • The benchmark includes 13 tasks spanning established datasets such as CIFAR-10 and newer Kaggle and research datasets.Tasks test whether agents can extrapolate to datasets potentially unseen during pretraining.
  • The agent constructs prompts summarizing task information and prior actions, then generates reflections, plans, fact checks, reasoning, and the next action.Available actions include reading and writing files and executing Python code.
  • 37.5% average success rate is achieved by the Claude v3 Opus agent, the best reported success rate among the evaluated language-model agents.Success reaches 100% on the house-price task but ranges from 0–25% on Kaggle challenges and BabyLM.

2. MLAgentBench: Benchmarking ML experimentation

MLAgentBench represents machine learning experimentation as repeated agent–environment interaction over executable tasks with automatic evaluation. Its benchmark spans diverse modalities and task recency, while its agent plans, acts, observes, updates memory, and is assessed for competence and efficiency.

  • Task specification: MLAgentBench specifies each task with a description, starter files, and an evaluator in a general task-independent environment.The benchmark includes 13 concrete and diverse machine learning tasks.
  • Task specification: Task descriptions define goals, submission formats, constraints, and occasional required approaches such as fine-tuning a pretrained BERT model.
  • Task specification: Starter files contain data, descriptions, metric information, and framework-specific baseline code, while evaluators assign scores to final submissions.A typical evaluator computes test accuracy from predictions in submission.csv.
  • General environment: At each time step, the agent produces a rationale and action, the environment updates the workspace and returns an observation, and the agent updates its memory.The agent can repeat actions until submitting a final answer or reaching action or time limits.
  • General environment: Available actions include file operations, arbitrary Python execution, final-answer declaration, and compound LM-assisted actions for understanding and editing files.Segmented script editing is particularly useful for large codebases such as CLRS and BabyLM.
  • Evaluation: Competence is measured by whether the final metric improves at least 10% over baseline, while efficiency uses wall-clock time and token consumption.Aggregated metrics include success rate and average performance improvement across runs.
  • Task diversity: The 13 tasks cover text, image, time-series, graph, and tabular data, combining well-studied datasets with newer challenges to test generalizability and mitigate contamination.Canonical tasks include CIFAR-10, IMDB, and ogbn-arxiv.

3. Our LM-based Agent

The LM-based agent iteratively constructs prompts from task context and recent interaction history, then generates a rationale and parses an action for execution. Its structured rationale supports planning, status tracking, fact checking, and human interpretation while addressing hallucinated progress.

  • At each step, the agent prompts the language model to generate a rationale and action, parsing the action from the model response.The prompt includes available actions, the task description, an output template, and the last three interaction steps.
  • The agent’s response format requires Reflection, Research Plan and Status, Fact Check, Thought, Action, and Action Input entries.The format lets the model plan and think before proposing an action.
  • Reflection revisits the previous step, Research Plan and Status tracks ongoing planning, Fact Check verifies claims, and Thought selects the next action.These components are inspired by Reflexion and ReAct and are designed to improve planning and reduce unsupported updates.
  • Detailed Research Plan and Status entries make the agent’s exploration process interpretable and enumerate the steps it intends to take.Fact Check is used to verify whether plan updates are factual, addressing hallucinated improvement observed during preliminary experiments.

4. Experiments

The experiments evaluate LM-based agents across tasks, success criteria, performance over time, research-process behaviors, and efficiency. Claude v3 Opus leads success rate overall, but outcomes vary widely across tasks and longer runs can degrade performance.

  • Competence: 37.5% average success rate makes Claude v3 Opus the strongest agent over most tasks, ranging from 100% on house-price to 0% on BabyLM.The reported success rates vary substantially across tasks.
  • Competence: The benchmark evaluates agents across tasks using success rate and average percentage improvement over baseline, with success defined as exceeding 10% improvement.Success rate is computed over 8 trials, while average improvement uses valid final submissions.
  • Competence: The proposed agent achieves a higher average success rate than baseline agents based on GPT-4-turbo and Claude v3 Opus.LangChain with Claude v3 remains competitive, partly because it avoids changing the submission format incorrectly.
  • Research Process: Longer runs generally degrade performance, although Claude v3 Opus is an exception; agents can regress after earlier improvements.Performance is evaluated at intermediate time steps as well as the final step.
  • Research Process: The analyzed failure modes include hallucination, bad planning, response-format errors, submission-format errors, and improvements below 10%.Bad plans often arise initially, making later recovery difficult.
  • Efficiency: GPT-4-turbo uses 51.0% fewer tokens than an average agent, while Claude v3 Opus uses nearly the most tokens and wall-clock time.The full benchmark with GPT-4-turbo took 6 million tokens, about 60 dollars, while the expected cost per successful task was estimated at 231 dollars given a 26% average success rate.

5. Related Work

Prior work explores language-model agents and automated experimentation across interactive environments, AutoML, and domain-specific scientific systems. MLAgentBench positions itself as a testbed for more general, open-ended ML experimentation.

  • Language-model agents have been explored for human-interaction simulation, Minecraft, robotics, general-purpose automation, and other interactive settings.
  • Related benchmarks evaluate agents in diverse environments, web interactions, and realistic high-stakes scenarios.
  • MLAgentBench examples and analyses examine agent traces, task outcomes, token efficiency, and success rate across experimentation settings.
  • AutoML-oriented systems use language models for hyperparameter tuning or prediction of categorized settings, whereas this work targets more open-ended experimentation.
  • Existing closed-loop scientific systems are often tailored to specific data types, tasks, and domains rather than general-purpose decision-making.

6. Conclusion

The paper introduces MLAgentBench and a prompting-based agent for end-to-end machine learning experimentation. The benchmark spans diverse tasks, and the agent succeeds at many of them with varying success rates.

  • MLAgentBench benchmarks language-model agents on performing machine learning experimentation end-to-end.
  • The authors develop a prompting-based agent that accomplishes many MLAgentBench tasks with varying success rates.
  • Future work targets a more robust agent, more complex and creative benchmark tasks, and human-AI collaboration studies.

Impact Statement

Language-model agents for ML experimentation may broaden access and support reproducibility, but autonomous pipeline modification introduces safety, adaptation, employment, and skill-displacement concerns. These risks motivate close supervision and involvement of ML researchers and engineers.

  • Language-model tools for ML experimentation may broaden participation by helping researchers without extensive coding or ML expertise engage in experimentation.
  • Autonomous modification and execution of ML pipelines can produce unpredictable outcomes, including dangerous system code, motivating close human supervision.
  • More capable agents may replace or diminish human engineers’ and researchers’ roles in some ML experimentation tasks, creating employment and skill-displacement concerns.
  • The impact statement compares the presented framework with other agent frameworks, including LangChain and AutoGPT, using GPT-4-turbo and Claude v3 Opus.

B. Benchmark Details

Benchmark tasks evaluate generated predictions or saved model checkpoints against task-specific requirements. Starter training code provides baseline or dummy outputs that can produce the required files.

  • Most benchmark task groups require a submission.csv prediction file, while CLRS and BabyLM are evaluated through saved model checkpoints.
  • The provided train.py starter code can generate required submission files using a baseline model or dummy predictions.

C. Quantitative Analysis

The section presents analyses of agents’ action usage and interaction length, alongside examples intended to illustrate component benefits and failure modes.

  • Figure 7 measures the percentage of time agents spend on each action and the distribution of the numbers of steps they use.
  • The section introduces examples demonstrating the benefits of individual agent components and observed failure modes.

D.1. Research Plan and Status

The agent maintains a detailed Research Plan and Status that guides iterative ML experimentation and remains interpretable to human readers. However, planning overly complex edits can lead to debugging dead ends.

  • Research Plan and Status entries are detailed and interpretable, helping guide exploration and support human understanding.The paper illustrates this with a Claude v1.0 CIFAR-10 training example.
  • The agent’s plan starts by understanding the script and dataset, then proposes improvements, establishes a baseline, and evaluates trials against it.
  • 52.53% baseline test accuracy after 5 epochs increased to 64.31% after 5 epochs after expanding convolutional filter counts.
  • The agent saved test-set class probabilities to submission.csv after achieving at least a 10% improvement within 10 epochs.
  • 40% of Claude v1.0 runs became stuck debugging after the agent planned an overly complex edit.Reflection can refocus the agent on the high-level problem but can also lead to repeated reflection without action.

D.2. Fact Check

The Fact Check entry is designed to reduce hallucinated claims about experiment results by marking unexecuted evaluations as unknown, but hallucinations persist.

  • The Fact Check entry flags performance as unknown when an updated file has not yet been executed, addressing hallucinated improvement claims.
  • The agent still hallucinated file knowledge or improvement claims in 20% of Claude-1 runs.One example claimed 26.35% improved over a 51.80% baseline even though the claimed result was lower.
  • In one case, execution of train_lr03.py confirmed the claimed result through observed output.

D.3. Problem Misspecification

The paper identifies problem specification as a critical determinant of agent performance. Ambiguous metric direction can cause the agent to optimize for the wrong objective.

  • The agent tried to increase SMAPE on the Parkinson’s disease progression dataset because the specification did not explain that lower SMAPE is better.
  • A tuned random forest achieved a cross-validation score of 76.16 versus a baseline score of 61.40 after reshaping the 1D data to 2D and tuning hyperparameters.
  • MLAgentBench compares agents’ average token and time usage for each task, with total tokens dominated by prompt tokens reused across steps.

F. Full Example

The agent iteratively edits and executes CIFAR-10 training scripts, comparing each result with a baseline. Architectural changes improved test accuracy, while higher learning rates and dropout produced worse reported results.

  • Hyperparameter trial: 9.99% test accuracy followed increasing the learning rate to 0.3, compared with the 52.53% baseline.The agent’s reflection attributed the result to convergence to a poor local minimum.
  • Research plan: The agent’s plan examines the training script, proposes hyperparameter, regularization, and architecture changes, then evaluates each trial against a baseline.The plan also requires saving test-set probabilities after achieving at least 10% improvement.
  • Regularization trial: 49.34% test accuracy followed adding dropout, lower than the 52.53% baseline despite the agent’s claim of improvement.The reported reflection also suggested that higher variance might explain the decrease.
  • Baseline: 52.53% baseline test accuracy was recorded after 5 epochs for the original model.The baseline came from running train.py.
  • Architecture trial: 64.31% test accuracy after 5 epochs followed increasing convolutional filter counts to 32 and 64, improving performance by over 11%.The modified architecture was described as having higher capacity and better generalization performance; the agent saved submission.csv.
Loading 2310.03302v2…