PaperlayerView exact revision on arXiv

Source-linked AI summary

MLEvolve: A Self-Evolving Framework for Automated Machine Learning Algorithm Discovery

Shangheng Du, Xiangchao Yan, Jinxin Shi, Zongsheng Cao, Shiyang Feng, Zichen Liang, Boyuan Sun, Tianshuo Peng, Yifan Zhou, Xin Li, Jie Zhou, Liang He, Bo Zhang, Lei Bai

arXiv:2606.06473v1cs.AIcs.CL

TL;DR

Existing MLE agents struggle with information isolation, memoryless search, and fixed exploration during long-horizon optimization. MLEvolve combines progressive graph search, retrospective memory, and adaptive code generation, achieving a 65.3% average medal rate on MLE-Bench under a 12-hour budget and outperforming existing baselines.

  • Problem

    Existing MLE agents have isolated search branches, memoryless frameworks, and fixed exploration strategies, limiting long-horizon self-evolution.

  • Method

    MLEvolve combines Progressive MCGS for cross-branch information flow and adaptive exploration, Retrospective Memory, and hierarchical adaptive code generation.

  • Results

    65.3% average medal rate on MLE-Bench under a 12-hour budget establishes state-of-the-art performance, while MLEvolve achieves the best result on 11 of 15 mathematical optimization tasks.

  • Takeaways & Limitations

    MLEvolve’s self-evolving mechanism generalizes beyond MLE to broader algorithmic optimization problems.

Abstract

from arXiv · show

Large language model (LLM) agents are increasingly applied to long-horizon tasks such as scientific discovery and machine learning engineering (MLE), where sustained self-evolution becomes a key capability. However, existing MLE agents suffer from inter-branch information isolation, memoryless search, and lack of hierarchical control, which together hinder long-horizon optimization. We present MLEvolve, an LLM-based self-evolving multi-agent framework for end-to-end machine learning algorithm discovery. By extending tree search to Progressive MCGS, MLEvolve enables cross-branch information flow through graph-based reference edges and gradually shifts the search from broad exploration to focused exploitation with an entropy-inspired progressive schedule. To allow the agent to evolve with accumulated experience, we introduce Retrospective Memory, which combines a cold-start domain knowledge base with a dynamic global memory for task-specific experience retrieval and reuse. For stable long-horizon iteration, we further decouple strategic planning from code generation with adaptive coding modes. Evaluation on MLE-Bench shows that MLEvolve achieves state-of-the-art performance across multiple dimensions including average medal rate and valid submission rate under a 12-hour budget (half the standard runtime). Moreover, MLEvolve also outperforms specialized algorithm discovery methods including AlphaEvolve on mathematical algorithm optimization tasks, demonstrating strong cross-domain generalization. Our code is available at https://github.com/InternScience/MLEvolve.

1. Introduction

MLEvolve is an LLM-based self-evolving multi-agent framework for end-to-end MLE that addresses inter-branch information isolation, memoryless exploration, and limited hierarchical control. Its Progressive MCGS, Retrospective Memory, and adaptive code-generation components yield state-of-the-art MLE-Bench performance and generalize to mathematical algorithm optimization.

  • Motivation: Existing MLE agents struggle with inter-branch information isolation, memoryless exploration, and lack of hierarchical control during long-horizon self-evolution.Tree-structured or linear searches confine information within individual branches, limiting transfer of successful strategies across search trajectories.
  • Framework: MLEvolve unifies Progressive MCGS, Retrospective Memory, and Hierarchical Planning with Adaptive Code Generation for long-horizon end-to-end MLE optimization.The framework uses graph-based cross-branch information flow, accumulated task-specific experience, and coordinated planning and coding.
  • Core components: Progressive MCGS shifts search from broad exploration to focused exploitation through graph-based information reuse and an entropy-inspired progressive schedule.The schedule adaptively steers exploration over time while graph references enable cross-branch transfer of successful strategies.
  • Core components: Retrospective Memory combines a curated domain knowledge base for cold-start initialization with dynamic global memory that accumulates and retrieves task-specific experience.This design allows the agent to evolve using accumulated experience throughout the search.
  • Results: 65.3% average medal rate on MLE-Bench under a 12-hour budget establishes state-of-the-art performance, while MLEvolve also outperforms AlphaEvolve on mathematical optimization tasks.The 12-hour budget is half the standard runtime, and the result demonstrates cross-domain generalization to specialized algorithm discovery.

2. Related Work

Prior MLE agents mainly search for code-based solutions, progressing from greedy search to structured MCTS variants, while related graph and memory methods address narrower settings. MLEvolve builds on these lines by targeting open-ended search with evolving cross-branch information flow and experience reuse.

  • MLE coding agents: MLE coding agents frame the task as searching for an optimal code-based solution and are commonly evaluated on MLE-Bench.AIDE uses greedy search, while later frameworks adopt more structured exploration, including MCTS-based approaches.
  • Graph-based search: Early MCGS methods merge identical states in defined state spaces, whereas LocAgent and CodexGraph use static graphs for retrieval or localization.These prior graph-based approaches do not evolve during search.
  • MLEvolve framework: MLEvolve combines Progressive MCGS, Retrospective Memory, and hierarchical planning with adaptive code generation as three framework components.The components support cross-branch information flow, knowledge and experience retrieval, and decoupled strategic planning and code implementation.
  • Memory mechanisms: Long-term memory methods let LLM agents accumulate and retrieve experiential records across extended horizons, with MLE systems exploring structured optimization directions and experience reuse.ROME stores successful trajectories as momentum memory and introduces “reasoning gradients” as structured optimization directions.

3. MLEvolve

MLEvolve is a self-evolving multi-agent framework for automated end-to-end machine-learning algorithm discovery. It combines graph-based progressive search, retrospective memory, and hierarchical adaptive code generation to support information reuse and stable long-horizon optimization.

  • Framework Objective: MLEvolve automates the search, design, and optimization of end-to-end ML pipelines by treating each graph node as a complete candidate solution.Candidate solutions cover preprocessing, feature engineering, model training, and prediction, and are evaluated with task-specific metrics such as accuracy, AUC, or loss.
  • Progressive MCGS: Progressive MCGS addresses branch isolation and inefficient fixed search by combining graph-based information flow with a schedule that shifts from exploration toward exploitation.Reference edges connect nodes across branches or non-adjacent levels, while entropy-inspired selection gradually concentrates computation on promising branches.
  • Progressive MCGS: Graph-based expansion supports intra-branch reflection, cross-branch reference, and multi-branch aggregation, enabling reuse and synthesis of information from multiple candidate pathways.These operations reference prior attempts, strong solutions from other branches, or complementary insights across trajectories.
  • Retrospective Memory: Retrospective Memory retrieves relevant historical experience before each planning decision through a static domain knowledge base and a dynamic global memory.The static knowledge base supports cold-start initialization, while the dynamic memory accumulates runtime experience.
  • Hierarchical Planning with Adaptive Code Generation: Hierarchical generation separates strategic planning from code implementation and adaptively selects code-generation modes according to the current search state.This pipeline is designed to address the lack of hierarchical control in one-shot code generation.

4. Experiments

MLEvolve is evaluated on MLE-Bench and mathematical optimization tasks, achieving strong performance under a 12-hour budget and across domains. Ablations and search-dynamics analyses further support the contributions of its components and progressive exploration-to-exploitation schedule.

  • Evaluation setup: The evaluation covers 75 Kaggle tasks from MLE-Bench and 15 open-ended mathematical optimization tasks from AlphaEvolve.MLE-Bench spans low, medium, and high complexity levels, while the mathematical tasks assess cross-domain generalization.
  • MLE-Bench results: 65.3% average medal rate and 34.7% gold medal rate make MLEvolve the best overall method on MLE-Bench under a 12-hour budget.It also records 80.3%, 64.0%, and 46.7% medal rates on low-, medium-, and high-complexity tasks, respectively.
  • MLE-Bench results: 100.0% valid submission rate and 76.0% above-median rate further distinguish MLEvolve on the full MLE-Bench evaluation.These results are reported over 75 tasks with results averaged across three seeds.
  • Ablation study: Removing any one core component reduces performance on MLE-Bench Lite, with Progressive MCGS producing the largest drops in medal rate and beat ratio.The ablation uses 22 tasks and keeps all other components unchanged.
  • Search dynamics: The effective branch count decreases from 4.8 to 2.8 during search, whereas Vanilla MCTS remains near 4.3, validating progressive exploration-to-exploitation.The effective count is measured as exp(H(π_t)) over the empirical distribution of selected branches.
  • Search dynamics: 98.2% final beat ratio is reached on representative tasks, while Vanilla MCTS ends at approximately 70% after plateauing earlier.MLEvolve improves rapidly early and continues gaining through the middle and late stages of the 12-hour search.

5. Conclusion · Appendix

MLEvolve is presented as a unified LLM-based self-evolving multi-agent framework for long-horizon machine learning engineering tasks. Its integrated components support adaptive search, sustained experience accumulation, flexible code generation, and state-of-the-art MLE-Bench performance.

  • 5. Conclusion: MLEvolve is an LLM-based self-evolving multi-agent framework for long-horizon MLE tasks.The framework is designed around a unified optimization process.
  • 5. Conclusion: Progressive MCGS enables adaptive search within MLEvolve’s unified optimization process.It is integrated with Retrospective Memory, Hierarchical Planning, and Adaptive Code Generation.
  • 5. Conclusion: Retrospective Memory enables sustained experience accumulation during long-horizon optimization.The memory component is part of MLEvolve’s integrated framework.
  • 5. Conclusion: Hierarchical Planning with Adaptive Code Generation enables flexible code generation.These components operate within the same unified optimization process as adaptive search and experience accumulation.
  • 5. Conclusion: 65.3% average medal rate was attained by MLEvolve on MLE-Bench under a 12-hour budget.The passage reports this as part of MLEvolve’s state-of-the-art experimental performance.
  • 5. Conclusion: The reported experiments show that MLEvolve achieves state-of-the-art performance on MLE-Bench.The result is reported under the stated 12-hour budget condition.

A. Agent Descriptions · B. Expansion Type Formulations

MLEvolve uses specialized agents for distinct search phases and a unified, reference-set-based expansion rule with four formulations. These expansions range from parent-only generation to intra-branch, cross-branch, and multi-branch information reuse.

  • A. Agent Descriptions: MLEvolve organizes search around specialized agents tailored to specific phases or operator types.The framework explicitly presents a team-based agent design.
  • A. Agent Descriptions: The Draft Agent generates initial root-node candidates and can retrieve priors from the cold-start knowledge base.The Improve Agent refines runnable solutions using planner guidance and Diff mode, while the Debug Agent repairs execution failures from error traces.
  • B. Expansion Type Formulations: The unified expansion rule is parameterized by a reference set R, whose instantiation defines four expansion types.The appendix specifies the reference set for each formulation.
  • B. Expansion Type Formulations: Primary expansion uses R = ∅, generating each new node solely from its parent as the baseline for graph-based variants.It corresponds to operators such as Draft, Improve, and Debug.
  • B. Expansion Type Formulations: Intra-branch evolution references the nearest k ancestor nodes within the current branch to form a local trajectory.The parent–child relation remains a primary edge, while reference edges capture information flow from intra-branch history; selection and backpropagation use only tree edges.
  • B. Expansion Type Formulations: Cross-branch reference selects top-performing nodes across evaluated branches when the current branch stagnates, enabling reuse of effective designs.Reference edges provide external guidance for improving the current solution.
  • B. Expansion Type Formulations: Multi-branch aggregation responds to global stagnation by creating a new root-level branch from top trajectories across multiple branches.Unlike single-branch refinement, it pools information and opens an independent exploration trajectory.

C. MLE-Bench Benchmark and Evaluation Metrics · C.1. MLE-Bench

MLEvolve is evaluated on MLE-Bench, an OpenAI benchmark for autonomous machine learning engineering comprising 75 curated Kaggle competitions. The benchmark supports comparison with human competitors through reconstructed datasets, local grading, human baselines, and medal-based evaluation.

  • C. MLE-Bench Benchmark and Evaluation Metrics: MLEvolve is evaluated on MLE-Bench, introduced by OpenAI to assess autonomous machine learning engineering.
  • C.1. MLE-Bench: MLE-Bench contains 75 carefully curated Kaggle competitions spanning natural language processing, computer vision, signal processing, and tabular data analysis.
  • C.1. MLE-Bench: The 75 competitions were manually screened by ML engineers from 586 candidates to represent authentic and challenging machine learning engineering work.
  • C.1. MLE-Bench: The benchmark includes competitions with varying complexity, including 22 low-complexity tasks solvable by experienced engineers in under 2 h.
  • C.1. MLE-Bench: Each competition provides the original problem description, datasets with reconstructed train-test splits, local grading code, and human baseline performance from Kaggle leaderboards.
  • C.1. MLE-Bench: These resources enable direct comparison between AI agents and human competitors while maintaining evaluation integrity.
  • C. MLE-Bench Benchmark and Evaluation Metrics: Medal achievement rates are the primary metric, with bronze, silver, and gold thresholds determined relative to human participants’ performance.

C.2. Evaluation Metrics · D. Hyperparameters

The evaluation uses outcome-based metrics covering medals, validity, and performance relative to human competitors, while MLEvolve’s key hyperparameters are fixed across all 75 MLE-Bench tasks unless stated otherwise.

  • C.2. Evaluation Metrics: All evaluation thresholds and percentile data are officially provided by Kaggle and MLE-Bench.
  • C.2. Evaluation Metrics: Medal Rate (All, in %) measures the percentage of tasks earning any gold, silver, or bronze medal.It is also stratified by task complexity: Low, Medium, or High.
  • C.2. Evaluation Metrics: Gold Medal Rate (Gold, in %) measures the percentage of tasks earning a gold medal.
  • C.2. Evaluation Metrics: Valid Submission Rate (Valid, in %) measures the percentage of tasks producing submissions that pass format and correctness checks.
  • C.2. Evaluation Metrics: Above Median Rate (Med+, in %) measures the percentage of tasks outperforming half of the human competitors.
  • C.2. Evaluation Metrics: Beat Ratio (in %) is the average percentage of human competitors surpassed by the agent’s submission.
  • D. Hyperparameters: Table 4 lists the key hyperparameters used in all MLEvolve experiments.Their values remain fixed across all 75 MLE-Bench tasks unless otherwise stated.

E. Detailed Component Analysis

Detailed ablations on nine MLE-Bench tasks show that intra-branch evolution is the most critical Progressive MCGS mechanism, while both memory sources materially contribute to performance. Cross-branch reference and Elite-Guided exploitation affect complementary aspects of solution quality, with Elite-Guided exploitation improving leaderboard ranking.

  • Experimental setup: The analysis disables one Progressive MCGS or Retrospective Memory mechanism at a time on a 9-task MLE-Bench subset, keeping all others unchanged.This isolates the contribution of individual mechanisms beyond the component-level ablation in §4.3.
  • Progressive MCGS: Removing cross-branch reference and Elite-Guided exploitation causes milder medal-rate decreases but affects different performance aspects.Cross-branch reference provides external guidance, while Elite-Guided exploitation mainly refines already competitive solutions.
  • Retrospective Memory: 44.44%: removing either the Knowledge Base or Global Memory reduces the medal rate, demonstrating that both experience sources contribute to performance.Removing Global Memory produces a lower beat ratio than removing the Knowledge Base.
  • Progressive MCGS: Elite-Guided exploitation yields the lowest beat ratio after removal, indicating its primary benefit is further improving leaderboard ranking.It refines solutions that are already competitive toward higher-performing ones.

F. Detailed Results with Different LLMs … G.2. Cross-branch Reference

MLEvolve remains competitive across four LLM backbones without a single model dominating all representative MLE-Bench tasks. Case studies show how intra-branch evolution and cross-branch references address stagnation through architectural changes and transferred loss designs.

  • F. Detailed Results with Different LLMs: Across 8 representative MLE-Bench tasks spanning Image, NLP, and Audio, Gemini-3.1-Pro-preview, GPT-5.5, DeepSeek-v4-Pro, and Kimi-K2.6 each show distinct strengths.All four backbones are evaluated under the same MLEvolve pipeline.
  • F. Detailed Results with Different LLMs: No single LLM backbone dominates all tasks, while all four produce competitive results under the shared MLEvolve pipeline.The reported comparison supports backbone robustness across tasks and domains.
  • G. Case Study: The case studies present representative examples of three graph-based expansion operators drawn from actual MLE-Bench runs.These examples illustrate how the framework operates during search.
  • G.1. Intra-branch Evolution: After six draft-and-improvement steps plateaued on aptos2019-blindness-detection, the Evolution Agent identified an architectural bottleneck and proposed fusing DINOv3 with ResNet50.Earlier EMA, Mixup, and cross-validation attempts had failed to improve the score.
  • G.1. Intra-branch Evolution: The coder implemented the proposed DINOv3–ResNet50 architectural change using diff-mode edits.This followed the Evolution Agent’s diagnosis of the local trajectory’s bottleneck.
  • G.2. Cross-branch Reference: On mlsp-2013-birds, the Fusion Agent addressed stagnation under symmetric Focal Loss by identifying Asymmetric Loss from a strong solution in another branch.The example demonstrates cross-branch transfer of an alternative loss design.
  • G.2. Cross-branch Reference: The coder replaced FocalLoss with AsymmetricLoss through diff-mode edits.This implementation completed the cross-branch reference case.

G.3. Multi-branch Aggregation

After detecting global stagnation, MLEvolve’s Aggregation Agent combines successful components from multiple branches to construct a new branch starting point. Figure 8 demonstrates this process on mlsp-2013-birds.

  • G.3. Multi-branch Aggregation: After global stagnation is detected, the Aggregation Agent synthesizes successful components from multiple branches into a new branch starting point.The example combines EfficientNet-B1 with GeM pooling, a bandpass filter, Multi-Label Focal Loss, and 5-fold cross-validation.
  • G.3. Multi-branch Aggregation: The aggregation case uses EfficientNet-B1 with GeM pooling, a bandpass filter, Multi-Label Focal Loss, and 5-fold cross-validation.These components are drawn from successful branches on the mlsp-2013-birds task.
  • G.3. Multi-branch Aggregation: Figure 8 illustrates multi-branch aggregation on the mlsp-2013-birds task.The figure shows how the synthesized components define the new branch’s starting point.
Loading 2606.06473v1…