Source-linked AI summary
AIBuildAI-2.5: Efficient Autonomous AI Model Development Through LLM-Guided Tree Search
Peijia Qin, Ruiyi Zhang, Qi Cao, Han Guo, Li Zhang, Pengtao Xie
TL;DR
Autonomous AI model-building agents face sparse and noisy search rewards, inefficient training-job scheduling, and costly LLM inference. AIBuildAI-2.5 combines LLM-guided tree search with resource-aware scheduling and role-based model routing. It ranks first on MLE-Bench and outperforms MLEvolve across evaluated AIRS-Bench tasks.
Problem
Autonomous AI model-building agents face sparse, noisy rewards for selecting candidates, no resource-aware training-job scheduling, and costly use of a single powerful LLM for every call.
Source: §1 · p. 2Method
AIBuildAI-2.5 uses a judge and selector for LLM-guided tree search, schedules jobs based on hardware status, and routes roles to the least costly capable model.
Source: §1 · p. 2Results
Takeaways & Limitations
Model routing matches or exceeds the all-frontier-model configuration's task scores at less than half its cost.
Source: §2.4 · p. 6The resource scheduler uses heuristic rules to decide whether to launch the next job based on current resource status.
Source: §3 · p. 7
Abstract
from arXiv · showhide
Autonomous agents that automatically build artificial intelligence (AI) models could broaden access to AI across science and engineering. A popular line of such agents frames model building as a code search problem and solves it by tree search, in which each node is a candidate program and the tree grows by generating a child program from a parent, and these agents now approach the capability of experienced AI engineers on realistic benchmarks. However, these agents have three weaknesses in efficiency that have not been fully addressed. First, only a small number of candidates can be executed within a realistic budget, so search rules that rank nodes by executed rewards, such as Monte Carlo-style tree search, rely on few and noisy scores and select the next node to explore less effectively. Second, no resource-aware strategy is used to schedule training jobs, which can lower hardware utilization and training efficiency. Third, every agent call is served by a single powerful model, which inflates inference cost. Here we introduce AIBuildAI-2.5, an agentic system that carries out the tree search with LLM agents and addresses each of the three issues. AIBuildAI-2.5 proposes a novel LLM-guided tree search, in which a judge scores each candidate on its expected improvement, grounding, and feasibility, and a selector ranks the pool of candidates from these scores and the state of the search. In addition, AIBuildAI-2.5 comprises a scheduler that launches training jobs with the current hardware resource status taken into account and a router that assigns lower-cost LLMs to less demanding tasks while reserving the most capable LLM for the most challenging sub-tasks in the AI model building workflow. AIBuildAI-2.5 ranks first on MLE-Bench with a medal rate of 73.3%, and outperforms a strong baseline on six autonomous AI research tasks from AIRS-Bench.
1 Introduction
AIBuildAI-2.5 addresses three efficiency gaps in autonomous AI model building with LLM-guided tree search, resource-aware scheduling, and cost-aware model routing. It ranks first on MLE-Bench and outperforms a strong baseline across six AIRS-Bench tasks.
1 Introduction
- Autonomous AI model building remains costly and expertise-dependent, motivating agents that can build models end-to-end from a task description and dataset.Tree-search agents represent candidate training-and-inference pipelines as programs and expand the search by generating children from parent nodes.Source: §1 · pp. 1–2
- Sparse, noisy rewards make existing reward-based selection less effective, while resource-unaware scheduling and single-model LLM calls can waste hardware and inference cost.A candidate may require hours of training and validation, limiting how many can be executed within a realistic budget.Source: §1 · p. 2
- AIBuildAI-2.5 uses a judge to score candidates on expected improvement, grounding, and feasibility, then a selector ranks them using those scores and search state.A scheduler accounts for current hardware resources when launching jobs, and a router assigns less costly models to roles they can handle while reserving the strongest model for harder roles.Source: §1 · p. 2
- 73.3% medal rate places AIBuildAI-2.5 first on MLE-Bench, and it scores better than MLEvolve on all six evaluated AIRS-Bench tasks.MLE-Bench covers 75 Kaggle-style tasks; AIRS-Bench tasks span molecular, temporal, and linguistic modalities.Source: §1 · p. 2
2 Results
AIBuildAI-2.5 combines LLM-guided tree search, hardware-aware scheduling, and model routing for autonomous model development. It ranks first on MLE-Bench, outperforms MLEvolve on all six AIRS-Bench tasks, and achieves comparable-or-better scores at lower routing cost.
2.1 Overview of AIBuildAI-2.5
- Role-specialized LLM agents build candidate programs, while a judge-selector prioritizes unexecuted leaves, a resource-aware scheduler launches jobs, and a router assigns model tiers.The judge scores expected improvement, grounding, and feasibility; the selector also considers the search state and remaining resources.Source: Figure 1 · p. 3, §2.1 · p. 4
2.2 AIBuildAI-2.5 ranks first on the MLE-Bench leaderboard
- 73.3% medal rate places AIBuildAI-2.5 first on the MLE-Bench leaderboard, surpassing prior autonomous AI development systems.The benchmark covers 75 realistic Kaggle-style tasks.Source: §2.2 · pp. 4–5, Figure 2 · p. 5
2.3 AIBuildAI-2.5 generalizes to autonomous AI research tasks
- AIBuildAI-2.5 scores better than MLEvolve on all six AIRS-Bench tasks under the same budgets and hardware.It lowers MAE by 20.8% on QM9-Cv and 29.8% on ZINC, improves accuracy by 5.1% on SICK-NLI and 1.4% on Yelp, and lowers Solar MAE by 4.0% and Web traffic MASE by 3.6%.Source: §2.3 · pp. 5–6, Figure 3 · p. 6
2.4 Model routing matches frontier-model performance at a fraction of the cost
- 56% lower total cost than Opus across three tasks, while routing scores comparably to or better than the strongest fixed tier on each task.On dog-breed, routing achieves the lowest log-loss among all configurations.Source: §2.4 · p. 6, Figure 4 · p. 7
3 Discussion
AIBuildAI-2.5 combines LLM-guided search, resource-aware scheduling, and model routing to improve autonomous model development under limited budgets. The discussion describes its benchmark results and potential application to research code generation and other scientific searches.
- 73.3% medal rate ranks AIBuildAI-2.5 first on MLE-Bench, and it outperforms a competing agent on all six AIRS-Bench tasks.Source: §3 · p. 7
- LLM-guided priors help rank candidates before costly execution, while scheduling fits evaluations to available resources and routing reduces inference cost.Source: §3 · p. 7
- The resource scheduler relies on heuristic launch rules; proposed extensions include job profiling, hardware sharing, and making generated jobs resource-aware.Source: §3 · p. 7
- The framework can apply to autonomous research code generation, where candidate experiment scripts are executed and scored under resource constraints.Source: §3 · p. 8
- Potential scientific applications include algorithm discovery, molecule and materials screening, simulations, and data-analysis pipelines, where candidate evaluation can be costly.Source: §3 · p. 8
4 Methods
The methods formulate AI model development as tree-based code search and divide expansion, selection, scheduling, and model assignment among specialized agents and system components. The experiments use standardized benchmark protocols and fixed compute budgets.
4.1 Problem formulation
- Model building is code search over executable training-and-inference programs, but hour-scale training yields few noisy rewards, making child quality and node selection critical.Source: §4.1 · p. 8
4.2 Agent-based tree search
- The designer proposes varied strategies, the reviser suggests concrete changes from node artifacts, and the coder implements and debugs programs; judge and selector rank pending leaves before execution.Source: §4.2 · p. 9
4.3 Auxiliary agents
- Setup prepares the shared runtime, data-split creates a global validation set for comparable rewards, and aggregator selects or ensembles completed nodes for submission.Source: §4.3 · pp. 9–10
4.4 Resource-aware job scheduler
- The scheduler launches ranked jobs when current hardware resources meet thresholds, allowing parallel execution while avoiding resource bottlenecks for running jobs.Source: §4.4 · p. 10
4.5 Model routing
- Models are assigned per role and search node; the router chooses the cheapest model it judges sufficient, adapting from within-run outcomes and cross-run role/task knowledge.Source: §4.5 · p. 10
4.6 Baselines
- The MLEvolve baseline uses a single-call tree expansion operator and a UCT rule over backed-up rewards, whereas AIBuildAI-2.5 uses specialized agents and LLM-guided selection [8].Source: §4.6 · p. 11
4.7 Experimental settings
- Both benchmarks use seven initial drafts and a 24-hour budget on a Linux machine with one NVIDIA A100 GPU; routing draws from three Claude models.Source: §4.7 · p. 11
Data availability
The MLE-Bench and AIRS-Bench data sources are publicly available.
- MLE-Bench data are publicly available from the MLE-Bench repository, and AIRS-Bench is publicly available from its repository.Source: Data availability · p. 11