PaperlayerView exact revision on arXiv

Source-linked AI summary

AIBuildAI: An AI Agent for Automatically Building AI Models

Ruiyi Zhang, Peijia Qin, Qi Cao, Li Zhang, Pengtao Xie

arXiv:2604.14455v1cs.AI

TL;DR

Developing high-performing AI models is labor-intensive, and existing automation leaves much of the development lifecycle to human expertise. AIBuildAI uses a hierarchical team of specialized LLM agents to build models from task descriptions and data. It ranks first on MLE-Bench, with a 63.1% medal rate.

  • Problem

    Developing high-performing AI models requires specialized expertise, while existing automation is limited to parts of the process and single-call coding agents face constraints.

    Source: §1 · pp. 1, 3

  • Method

    AIBuildAI uses a manager to coordinate LLM-based designer, coder, and tuner agents that develop models from task descriptions and training data.

    Source: §1 · p. 3, §2.1 · p. 3

  • Results

    On MLE-Bench, AIBuildAI ranked first with a 63.1% medal rate, outperforming all existing autonomous AI development systems.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The results demonstrate that hierarchical agent systems can automate AI model development from task specification to deployable model with minimal human intervention.

    Source: §1 · p. 3

    The evaluation uses MLE-Bench’s single-GPU, 24-hour protocol, while real-world tasks may require multi-GPU training and longer horizons.

    Source: §3 · p. 12

Abstract

from arXiv · show

AI models underpin modern intelligent systems, driving advances across science, medicine, finance, and technology. Yet developing high-performing AI models remains a labor-intensive process that requires expert practitioners to iteratively design architectures, engineer representations, implement training pipelines and refine approaches through empirical evaluation. Existing AutoML methods partially alleviate this burden but remain limited to narrow aspects such as hyperparameter optimization and model selection within predefined search spaces, leaving the full development lifecycle largely dependent on human expertise. To address this gap, we introduce AIBuildAI, an AI agent that automatically builds AI models from a task description and training data. AIBuildAI adopts a hierarchical agent architecture in which a manager agent coordinates three specialized sub-agents: a designer for modeling strategy, a coder for implementation and debugging, and a tuner for training and performance optimization. Each sub-agent is itself a large language model (LLM) based agent capable of multi-step reasoning and tool use, enabling end-to-end automation of the AI model development process that goes beyond the scope of existing AutoML approaches. We evaluate AIBuildAI on MLE-Bench, a benchmark of realistic Kaggle-style AI development tasks spanning visual, textual, time-series and tabular modalities. AIBuildAI ranks first on MLE-Bench with a medal rate of 63.1%, outperforming all existing baseline methods and matching the capability of highly experienced AI engineers. These results demonstrate that hierarchical agent systems can automate the full AI model development process from task specification to deployable model, suggesting a pathway toward broadly accessible AI development with minimal human intervention.

1 Introduction

Building high-performing AI models remains labor-intensive and dependent on specialized expertise, while existing automation covers only parts of the development process. AIBuildAI coordinates specialized agents to automate model development and ranks first on MLE-Bench.

1 Introduction

  • Model development requires iterative strategy design, preprocessing, coding, training, optimization, debugging, and empirical refinement, making high-performing models labor-intensive and reliant on specialized expertise.The complexity limits practitioners’ ability to fully leverage AI for large-scale discovery and real-world applications.Source: §1 · p. 1
  • Traditional AutoML searches predefined candidates, while single-call coding agents face limits on modification complexity and must handle distinct implementation and tuning phases.Traditional AutoML also requires users to define candidates using prior knowledge.Source: §1 · pp. 1, 3
  • A manager coordinates designer, coder, and tuner agents that handle modeling strategy, implementation and debugging, and training and performance optimization.Each sub-agent can conduct multiple rounds of LLM calls and tool interactions, while the manager selects candidates using execution history.Source: Figure 1 · p. 2, §1 · p. 3
  • AIBuildAI ranks first on MLE-Bench with a 63.1% medal rate, outperforming existing autonomous AI development systems and matching highly experienced AI engineers.The benchmark spans visual, textual, time-series, and tabular tasks; the reported result is without task-specific customization.Source: §1 · p. 3

2 Results

AIBuildAI uses a manager-coordinated designer, coder, and tuner workflow to build models, and ranks first on MLE-Bench. Across modality-specific evaluations, it outperforms the two named baselines on the reported tasks.

2.1 Overview of AIBuildAI

  • A manager coordinates parallel candidate work by designer, coder, and tuner agents, while setup and aggregation support the pipeline.The system accepts a task description and training data and outputs trained checkpoints and an inference script.Source: §2.1 · p. 3

2.2 AIBuildAI ranks first on the MLE-Bench leaderboard

  • 63.1% medal rate puts AIBuildAI first among 26 recent MLE-Bench baseline methods or variants as of March 18, 2026.It also ranks first on the medium- and high-complexity splits.Source: Figure 2 · p. 4, §2.2 · p. 5

2.3 AIBuildAI demonstrates strong performance on image classification tasks

  • AIBuildAI beats MLEvolve and AIRA-dojo on all 10 image-classification tasks; its Alaska2 ensemble scores 0.8101 on the private test set.The example workflow proposes six candidates, advances four, and revises one before ensembling.Source: §2.3 · pp. 5, 7, Figure 3 · p. 6

2.4 AIBuildAI excels in detection, segmentation, and video understanding tasks

  • AIBuildAI outperforms both baselines on all 10 structured visual tasks; its contrails ensemble reaches a 0.660 Dice score on the private test set.The example workflow revises two candidates and ensembles the strongest pair.Source: §2.4 · p. 7, Figure 4 · p. 8

2.5 AIBuildAI achieves superior performance on language understanding and generation tasks

  • AIBuildAI outperforms both baselines on all 10 language tasks; the selected word-imputation model yields a private-test Levenshtein distance of 5.519.The example pipeline scores masked-word candidates with BERT and uses a confidence gate to decide whether to replace the missing word.Source: Figure 5 · p. 9, §2.5 · p. 10

2.6 AIBuildAI delivers strong performance on temporal and tabular tasks

  • AIBuildAI outperforms MLEvolve and AIRA-dojo on all six temporal and structured-data tasks, spanning three tasks in each modality.The EEG example combines spectrogram and raw-signal branches to produce six-class probabilities.Source: §2.6 · pp. 10, 12, Figure 6 · p. 11

3 Discussion

AIBuildAI uses specialized sub-agents to refine parallel solutions while limiting context growth, and its multi-step operations distinguish it from single-call coding approaches. The discussion also identifies higher token costs and boundaries of the current evaluation, alongside possible extensions and scientific applications.

3 Discussion

  • Specialized sub-agents keep each operation focused and avoid the context explosion of maintaining all intermediate reasoning in one agent.The coder tracks prior implementation attempts, while the tuner uses the coder’s correct code and focuses on training performance.Source: §3 · p. 12
  • Unlike AIRA and MLEvolve’s single-call code transformations, AIBuildAI’s sub-agents can perform multiple rounds of reasoning and tool use, a stated factor in its performance across diverse tasks.The paper identifies this shift as a key factor in producing high-quality solutions.Source: §3 · p. 12
  • Multiple interacting sub-agents increase token consumption relative to baseline approaches through repeated LLM calls and manager–sub-agent communication.The paper proposes routing simpler requests to cheaper models as a possible way to reduce cost without sacrificing performance.Source: §3 · p. 12
  • The evaluation uses one GPU and a 24-hour budget; proposed extensions include resource-aware distributed training and a retrievable, updateable AI-task knowledge base.The proposed knowledge base would collect modeling strategies, preprocessing techniques, architecture patterns, and outcomes from prior runs.Source: §3 · pp. 12–13
  • Systems like AIBuildAI could help scientists build predictive models from experimental data without deep AI engineering expertise.The discussion names genomics, materials science, chemistry, and drug discovery as potential application areas.Source: §3 · p. 13

4 Methods

An LLM-based agent extends single-shot prompting with iterative tool interaction. Its behavior is specified by its prompt and available tools, which shape its execution strategy.

4 Methods

  • An agent can request external tool execution, append returned observations to its context, and use them to produce its final answer.Examples include running code, querying a database, or modifying files.Source: §4.1 · p. 13
  • The agent repeats model calls when outputs request tools and terminates when an output has no tool invocation or the maximum number of steps is reached.Each observation is appended to the interaction history before the next call.Source: §4.1 · p. 13
  • An agent is fully specified by its task prompt and tool set; changing those components can produce different capabilities and execution strategies.The prompt describes the task and how to act on intermediate results, while the tools define available actions.Source: §4.1 · p. 13

4.2 AIBuildAI system design

AIBuildAI treats automated model development as constructing a complete runnable solution from a task description and dataset. It uses isolated repositories and specialized agents to develop and refine candidate solutions.

4.2 AIBuildAI system design

  • The task takes a natural-language description, evaluation protocol, and dataset as input, and aims to produce a runnable repository with trained checkpoints and an inference script for unseen data.Source: §4.2 · p. 13
  • AIBuildAI initializes multiple repositories with different modeling strategies, refines each incrementally, and selects or ensembles the best-performing solutions.Repositories hold data, a textual plan, code, configuration, and execution results such as validation performance and checkpoints.Source: §4.2 · pp. 13–14
  • Repository-scoped read, write, and execute tools let the agent inspect artifacts, revise plans, code, or configurations, and run training.A single agent with all tools could in principle do this, but managing heterogeneous operations across repositories would strain context limits and attention.Source: §4.2 · p. 14
  • A manager delegates solution design, code generation, and training or hyperparameter tuning to specialized LLM-based sub-agents with separate contexts and restricted tool subsets.The manager’s compact context supports unified control across the run, while sub-agents write their resulting artifacts to their repository.Source: §4.2 · p. 14

4.3 Manager

The manager orchestrates candidate repositories by reading their artifacts and assigning specialized sub-agents. Its decisions adapt to repository-specific results, and tool calls can run concurrently across repositories.

4.3 Manager

  • The manager selects repository-scoped designer, coder, and tuner tools based on context and reads plans, code, and results across repositories to compare candidates.These artifacts also help it determine which refinement a candidate needs.Source: §4.3 · p. 14
  • Asynchronous tool calls let the manager explore multiple repositories in parallel and overlap design, implementation, and training.Source: §4.3 · p. 14
  • The manager adaptively assigns design, coding, or tuning work according to whether a candidate’s weakness lies in its strategy, implementation, or training configuration.Underperforming or similar repositories may be dropped by ceasing further actions.Source: §4.3 · p. 14

4.4 Designer

The designer creates or revises a modeling plan based on the repository’s current contents and available results. It writes the updated plan back to the repository for the coder to implement.

4.4 Designer

  • When a plan and results exist, the designer analyzes training loss, validation metrics, and logs to diagnose potential failures and revise the modeling strategy.Revisions may change the architecture, data-processing pipeline, or training procedure.Source: §4.4 · p. 15
  • If no plan exists, the designer creates an initial strategy covering the modeling approach, data-processing pipeline, and training procedure.It usually uses web search for this initial plan.Source: §4.4 · p. 15
  • The designer observes the repository at each invocation and writes its updated plan there for the coder to consume.Source: §4.4 · p. 15

4.5 Coder

The coder implements or revises the solution plan within its assigned repository, then runs and debugs the code to verify that the pipeline works end to end. Its resulting code is made available to other sub-agents, especially the tuner.

4.5 Coder

  • The coder implements the plan, then runs a short end-to-end verification and iteratively debugs encountered errors until the code is bug-free.The initial implementation includes the model, training and inference code, and an example configuration file.Source: §4.5 · p. 15
  • When code already exists, the coder fixes errors, improves robustness, or incorporates plan updates.Source: §4.5 · p. 15
  • Each coder works only in its assigned repository, reading its plan and writing implementation artifacts there for subsequent sub-agents, especially the tuner.Source: §4.5 · p. 15

4.6 Tuner

The tuner trains models and improves performance by analyzing prior results, changing training configurations, and running new experiments within a manager-assigned time limit. It extends promising configurations to full training in later stages.

4.6 Tuner

  • Within its maximum tuning time, the tuner uses short preliminary runs for early performance signals, replaces stagnant or degrading configurations, and extends promising ones to full training.Source: §4.6 · p. 15
  • The tuner analyzes logs, validation metrics, and checkpoints to identify slow convergence, overfitting, or instability, then adjusts configurations and launches new training runs.Changes may include hyperparameters, schedules, or data-related settings.Source: §4.6 · p. 15
  • The tuner writes updated configurations, logs, checkpoints, and evaluation outputs to its repository, making validation results available to other agents.Source: §4.6 · p. 15

4.7 Setup and aggregation

A setup agent prepares the training environment before execution, while an aggregator finalizes the workflow by selecting or combining candidate solutions into a single submission. These two agents run once, outside the manager-controlled iterative loop.

4.7 Setup and aggregation

  • A setup agent runs at initialization and an aggregator at termination; neither participates in the iterative decision loop.They provide deterministic preparation and finalization of the workflow.Source: §4.7 · p. 15

Setup

  • The setup agent creates a conda environment and installs required packages with conda or pip before any sub-agent begins execution.Source: Setup · p. 15

Aggregator

  • The aggregator selects the best candidate or constructs a weighted ensemble based on solution diversity, validation performance, and remaining computation time.It adapts aggregation to the task, using rank-based blending, probability-map combinations, or weighted prediction averages as appropriate.Source: Aggregator · p. 16
  • When time is insufficient for ensemble inference or one candidate substantially outperforms the others, the aggregator directly selects that candidate’s prediction outputs.It does not retrain models or modify intermediate artifacts.Source: Aggregator · p. 16

4.8 Baseline methods

The section compares autonomous machine-learning engineering systems that search over code-based solutions, where candidate pipelines are evaluated by execution and search policies allocate computation between exploration and exploitation.

4.8 Baseline methods

  • Baseline methods search over executable code artifacts implementing complete training and inference pipelines, with evaluation requiring computationally expensive execution.Search selects an evaluated parent solution and expands it into variants conditioned on its result.Source: §4.8 · p. 16

AIRA

  • AIRA uses single-call Draft, Debug, and Improve operators, with global Memory and a default greedy policy that expands the highest-scoring solution.It also examines MCTS and evolutionary policies, while emphasizing the operator set as the primary determinant of effectiveness.Source: AIRA · p. 16

MLEvolve

  • MLEvolve replaces tree search with MCGS aggregation, uses intra- and cross-branch references instead of global Memory, and supplies a domain-specific knowledge base.Its knowledge base covers model, data, and strategy dimensions.Source: MLEvolve · pp. 16–17

4.9 Experimental settings

Experiments follow the MLE-Bench protocol and compare systems using different LLM backbones, time budgets, and hardware configurations.

4.9 Experimental settings

  • All experiments follow the MLE-Bench evaluation protocol, and AIBuildAI uses Claude Opus 4.6 across agents at temperature 1.0 without nucleus or top-k truncation.It initializes n = 7 parallel solution repositories per task.Source: §4.9 · p. 17
  • AIBuildAI is evaluated for 24 hours on 24 vCPUs, 256 GB RAM, and one NVIDIA A100 GPU.Source: §4.9 · p. 17
  • AIRA uses OpenAI o3 for 24 hours on 24 vCPUs, 120 GB RAM, and one NVIDIA H200 GPU; MLEvolve uses Gemini 3 Pro Preview for 12 hours on 21 vCPUs, 234 GB RAM, and one NVIDIA H200 GPU.Source: §4.9 · p. 17

Related papers

Loading 2604.14455v1…