PaperlayerView exact revision on arXiv

Source-linked AI summary

ASI-Evolve: AI Accelerates AI

Weixian Xu, Tiantian Mi, Yixiu Liu, Yang Nan, Zhimeng Zhou, Lyumanshan Ye, Lin Zhang, Yu Qiao, Pengfei Liu

arXiv:2603.29640v1cs.AI

TL;DR

Whether agents can handle costly, long-horizon, weakly supervised AI research remains unclear. ASI-Evolve applies a learn–design–experiment–analyze loop and demonstrates AI-driven discovery across data, architectures, and learning algorithms.

  • Problem

    AI-development research cycles across data, architectures, and learning algorithms face limited human exploration capacity, substantial manual effort, and difficulty preserving insights across iterations.

    Source: §1 · p. 2

  • Method

    ASI-EVOLVE closes a learn–design–experiment–analyze loop by grounding exploration in accumulated research literature and using an analyzer to distill experimental outputs into decision-oriented reports.

    Source: §1 · p. 2, §3.3 · p. 5

  • Results

    ASI-EVOLVE demonstrated AI-driven discovery across model architecture, training data, and training algorithms; its top model gained +0.97 points over DeltaNet.

    Source: §1 · p. 2, §6 · p. 14

  • Takeaways & Limitations

    ASI-EVOLVE's discovered model designs also showed use outside AI development in a drug-target interaction experiment.

    Source: §6 · p. 14

    The full AI-development stack, including infrastructure, remains an area for future exploration.

    Source: §6 · p. 14

Abstract

from arXiv · show

Can AI accelerate the development of AI itself? While recent agentic systems have shown strong performance on well-scoped tasks with rapid feedback, it remains unclear whether they can tackle the costly, long-horizon, and weakly supervised research loops that drive real AI progress. We present ASI-Evolve, an agentic framework for AI-for-AI research that closes this loop through a learn-design-experiment-analyze cycle. ASI-Evolve augments standard evolutionary agents with two key components: a cognition base that injects accumulated human priors into each round of exploration, and a dedicated analyzer that distills complex experimental outcomes into reusable insights for future iterations. To our knowledge, ASI-Evolve is the first unified framework to demonstrate AI-driven discovery across three central components of AI development: data, architectures, and learning algorithms. In neural architecture design, it discovered 105 SOTA linear attention architectures, with the best discovered model surpassing DeltaNet by +0.97 points, nearly 3x the gain of recent human-designed improvements. In pretraining data curation, the evolved pipeline improves average benchmark performance by +3.96 points, with gains exceeding 18 points on MMLU. In reinforcement learning algorithm design, discovered algorithms outperform GRPO by up to +12.5 points on AMC32, +11.67 points on AIME24, and +5.04 points on OlympiadBench. We further provide initial evidence that this AI-for-AI paradigm can transfer beyond the AI stack through experiments in mathematics and biomedicine. Together, these results suggest that ASI-Evolve represents a promising step toward enabling AI to accelerate AI across the foundational stages of development, offering early evidence for the feasibility of closed-loop AI research.

1. Introduction

ASI-EVOLVE addresses the difficulty of automating costly, long-horizon AI research by grounding iterative experiments in accumulated knowledge and turning their outcomes into reusable insights. The paper reports AI-driven discovery across data, architectures, and learning algorithms, with additional experiments in circle packing and biomedicine.

1 Introduction

  • AI research loops remain difficult to automate because they require costly experiments, complex code changes, interpretation of multidimensional outcomes, and coherent exploration over many rounds.The introduction identifies human limits on parallel hypothesis exploration, manual experimental work, and systematic preservation of insights as bottlenecks.Source: §1 · p. 2
  • ASI-EVOLVE uses a learn–design–experiment–analyze cycle, with a cognition base to ground exploration and an analyzer to write actionable experimental insights back to its experience database.The framework is designed for long-horizon research where feedback is expensive, indirect, noisy, and difficult to interpret.Source: §1 · p. 2
  • 105 discovered architectures surpass DeltaNet in ASI-EVOLVE’s unified AI-driven discovery across architectures, data curation, and learning algorithms.The top architecture gained +0.97 points; evolved data curation improved average benchmark performance by 3.96 points and MMLU by more than 18 points; RL algorithms beat GRPO by up to +12.5 points on AMC32, +11.67 on AIME24, and +5.04 on OlympiadBench.Source: §1 · p. 2
  • On circle packing, ASI-EVOLVE reaches SOTA-level results in as few as 17 rounds, outperforming prior frameworks including OpenEvolve and GEPA.In drug-target interaction prediction, its evolved architecture improves cold-start generalization AUROC by 6.94 points.Source: §1 · p. 2

2. Preliminary

The section introduces Scientific Task Length (Ltask) to characterize scientific research by execution cost, search-space complexity, and feedback complexity. It positions ASI-EVOLVE for large-scale scientific exploration, where costly trials and rich feedback require cognition-guided search and dedicated analysis.

2. Preliminary

  • The section reports initial evidence that ASI-EVOLVE’s AI-for-AI paradigm can generalize beyond AI tasks to broader scientific applications.The claim is presented as initial evidence, not as a comprehensive demonstration across scientific fields.Source: §2 · p. 3

2 Preliminary

  • Ltask represents task complexity as ⟨Cexec, Sspace, Dfeedback⟩, capturing per-trial execution burden, solution-space openness, and the difficulty of extracting actionable insights from feedback.The framework distinguishes multidimensional experimental signals from simple scalar evaluation.Source: §2 · p. 3
  • Scientific question answering has low Ltask across all three dimensions because it requires little experimental execution, simple evaluation, and no iterative experimental loop.The passage characterizes idea-generation systems, automated surveys, and scientific QA benchmarks as examples of this class.Source: §2 · p. 3
  • Structured task execution adds experiments but remains bounded by predefined objectives, constrained exploration, and feedback that does not require deep synthesis for scientific progress.These systems prioritize completing tasks with clear success criteria rather than advancing scientific understanding.Source: §2 · p. 3
  • Lightweight evolutionary discovery raises execution cost and search-space complexity, but localized code changes and immediate feedback keep feedback complexity low.The passage cites discoveries in mathematics, optimization, activation functions, and systems as examples.Source: §2 · p. 3
  • Large-scale AI research spans architecture design, data curation, and training algorithms, with costly trials, open-ended search, and feedback across benchmarks, training dynamics, and efficiency.Validating one candidate can require tens to hundreds of GPU hours and modifications to large, interdependent codebases.Source: §2 · p. 4
  • ASI-EVOLVE stores and retrieves accumulated cognition and analysis so each iteration can use prior experience to guide where to search next.Its pipeline retrieves relevant cognition, generates and evaluates a candidate, then stores an analysis report for future rounds.Source: §2 · p. 4, Figure 2 · p. 4

3 ASI-Evolve

ASI-Evolve runs an iterative research loop that learns from prior experiments and retrieved human knowledge, generates and evaluates candidate programs, and analyzes outcomes into reusable insights. Its components combine compact analytical reports, cognition-based cold-start guidance, and persistent task-specific memory with configurable sampling strategies.

3 ASI-Evolve

  • ASI-Evolve iterates through learning, candidate design, experimentation, and analysis, using each experiment’s evaluation signals to inform later rounds.The pipeline stores experimental outcomes and reusable lessons for subsequent iterations.Source: §3 · p. 4

3.1 Researcher

  • The Researcher samples prior database nodes and retrieves semantically similar cognition items to guide its next candidate program.Retrieved items are selected using the sampled nodes’ analyses or motivations as search queries.Source: §3.1 · p. 4

3.2 Engineer

  • The Engineer evaluates generated programs for structured metrics and a primary fitness score, filtering flawed candidates early and optionally adding an LLM judge.Candidate programs can be generated in full or edited through localized diffs, which can help with larger codebases.Source: §3.2 · p. 5

3.3 Analyzer

  • The Analyzer examines full experimental outputs, including raw logs and detailed metrics, and stores compact, decision-oriented reports for later retrieval.This lets it conduct causal analysis while keeping subsequent-round context manageable.Source: §3.3 · p. 5

3.4 Cognition

  • The cognition base injects retrieved human heuristics, pitfalls, and design principles into each round, improving cold-start climb speed and iteration efficiency without limiting long-term exploration.Cognition entries are retrieved by embedding-based semantic search using sampled database nodes as queries.Source: §3.4 · p. 5

3.5 Database

  • The database preserves motivations, programs, evaluation results, analyses, and metadata; its UCB1, random, greedy, and MAP-Elites sampling policies produce distinct evolution trajectories.Historical nodes become the dominant information source as evolution progresses, while sampling choice affects sustained improvement.Source: §3.5 · p. 5

4 Main Tasks

ASI-Evolve is evaluated on architecture design, pretraining data curation, and RL algorithm design—tasks characterized by limited task-specific priors, long iteration cycles, complex implementation, and difficult-to-interpret feedback. The results cover AI research from model structure through data and training.

4.1 Scenario 1: Model Architecture Design

  • Adaptive, multi-scale routing—not fixed structural choices—was the shared design theme among the five top architectures.Their approaches included hierarchical allocation across local, contextual, and delta-rule paths; content-aware routing; independent sigmoid gates; learnable floors that protect critical paths; and token-level balanced sparse gating with entropy penalties.Source: §4.1 · p. 6
  • 57.28% average accuracy on development benchmarks and 45.40% on generalization benchmarks were the best discovered results, versus DeltaNet’s 55.76% and 44.74%, respectively.Across 1773 exploration rounds, 105 architectures exceeded DeltaNet in verification; the best model achieved nearly 3× the gain of Mamba2’s +0.34 points over DeltaNet (Dao and Gu, 2024).Source: §4.1 · p. 6, Table 1 · p. 7
  • 51.7% of all discovered architectures derived from the cognition base, while 44.8% of SOTA architectures came from accumulated experience; the authors identify a hardware-efficiency limitation.The provenance shares shift from 51.7% cognition-base, 38.2% experience, and 10.1% novel across all 1773 architectures to 44.8% experience and 6.6% novelty among SOTA architectures. Because the system designs attention mechanisms rather than low-level kernels, it cannot directly produce hardware-optimized CUDA kernels, and judge scores cannot guarantee wall-clock efficiency after optimization.Source: §4.1 · p. 7

4.2 Scenario 2: Pretraining Data Curation

  • 44.13 average score: Nemotron-CCASI+ surpassed raw data by 3.96 points across 18 benchmarks under the reported 3B-parameter, 500B-token training setup.Gains included MMLU +18.64 points, CSQA +18.80 points, and MedQA +13.48 points; the paper also reports superiority to DCLM, FineWeb-Edu, and Ultra-FineWeb under identical training budgets.Source: Table 2 · p. 8, §4.2 · p. 8
  • Discovered curation strategies converged on targeted cleaning, format normalization, and domain-aware preservation, with iterative diagnostic feedback supporting refinement.Concrete thresholds, targeted deletion, and explicit preservation rules characterized effective strategies; the 2.93-point gap between optimized and suboptimal strategies illustrates the value of refinement over one-shot generation.Source: §4.2 · p. 8, §4.3 · p. 9

4.3 Scenario 3: Reinforcement Learning Algorithm Design

  • +12.5 points on AMC32 (67.5 →80.0), +11.67 points on AIME24 (20.00 →31.67), and +5.04 points on OlympiadBench (45.92 →50.96) were the best evolved variants’ improvements over GRPO.Three algorithms showed statistically significant improvements across all tested domains after verification at 14B parameters; 10 algorithms had outperformed GRPO during exploration.Source: Table 3 · p. 9, §4.3 · p. 9
  • Two representative RL algorithms use distinct mechanisms: pairwise asymmetric optimization and a budget-constrained dynamic update radius.Algorithm A uses tanh-normalized pairwise reward differences, advantage-dependent asymmetric clipping, and high-impact gradient dropout. Algorithm B uses percentile-based advantage normalization and a global update budget that bounds total policy-update magnitude.Source: §4.3 · p. 9

5. Empirical Analysis · 5.1 Benchmarking ASI-Evolve on Circle Packing · 5.2 Comparison Experiments

On circle packing, ASI-Evolve reaches a score of 2.63597 in 17 steps and a best score of 2.635983; in framework comparisons, it is the only system reported to reliably reach SOTA-level performance. Tests with GPT-5-mini and Qwen3-32B show similar convergence ranges and mid-to-late-stage improvement trends.

5. Empirical Analysis

  • The analysis frames ASI-Evolve’s RL solutions as using rigorous mathematical derivation to address stability and variance challenges, paralleling human-designed advances.Source: §5 · p. 10

5 Empirical Analysis

  • The empirical analysis includes controlled ablations and examines whether evolved designs apply beyond AI/ML, including an architecture evaluated in a biomedical setting.The section introduces circle packing as a direct-comparison benchmark before these analyses.Source: §5 · p. 10

5.1 Benchmarking ASI-Evolve on Circle Packing

  • The circle-packing benchmark uses 26 circles in a 1 × 1 square and measures the sum of their radii, providing an aligned comparison task with low verification cost.The task still requires non-trivial algorithm design and iterative refinement.Source: §5.1 · p. 10
  • 2.63597 in 17 steps is the fastest reported circle-packing result, and ASI-Evolve’s best score of 2.635983 is comparable to other frameworks’ top results.The task packs 26 circles in a 1 × 1 square to maximize the sum of their radii.Source: Table 4 · p. 10, §5.1 · p. 10

5.2.1 Framework Comparison

  • ASI-Evolve exits cold start ahead of OpenEvolve and GEPA, improves steadily, and is the only framework reported to reliably reach SOTA-level performance.OpenEvolve shows high run-to-run variance and limited improvement, while GEPA converges around 2.630.Source: §5.2.1 · p. 10

5.2 Comparison Experiments

  • Analyzer and Cognition components provide structured feedback and domain priors that support faster convergence and higher final scores across diverse evolutionary frameworks.These advantages align with the framework-comparison findings and ablation results.Source: §5.2.1 · p. 10

5.2.2 Model Comparison

  • GPT-5-mini and Qwen3-32B converge to similar ranges with consistent mid-to-late-stage improvement, though their early-stage cadence differs.The reported trends indicate that ASI-Evolve’s evolution capability is not tied to one model family.Source: §5.2.2 · p. 10

5.2 Comparison Experiments · 5.3 Ablation Study: Validating Component Effectiveness

The experiments compare ASI-Evolve across frameworks and base models, then examine parent-sampling strategies and ablated variants. Sampling results indicate that cognition-guided UCB1 can accelerate discovery relative to MAP-Elites, while random sampling’s early advantage fades.

5.2 Comparison Experiments

  • ASI-Evolve’s base-model comparison shows a mid-stage jump before converging to a comparable level, supporting compatibility and robustness across models.Figure 3 also plots ASI-Evolve against GEPA and OpenEvolve, and compares GPT-5-mini with Qwen3-32B; shaded regions show variability across repeated runs.Source: §5.2 · p. 11, Figure 3 · p. 11

5.2.3 Algorithm Comparison

  • The sampling ablation compares MAP-Elites, UCB1, and uniform random parent selection, which differ in how they balance quality, diversity, and exploration.MAP-Elites preserves behavioral niches; UCB1 combines estimated value with an exploration bonus, while random sampling selects uniformly without regard to score or diversity.Source: §5.2.3 · p. 11
  • Random sampling starts above MAP-Elites but gradually slows and falls behind both MAP-Elites and UCB1 in later stages.The passage attributes its early lead to unrestricted exploitation of informative parent nodes, and its later decline to lacking a mechanism that balances exploration and exploitation.Source: §5.2.3 · p. 11, Figure 4 · p. 11
  • UCB1 reaches high-score regions faster than MAP-Elites and has lower run-to-run variance, which the authors attribute to cognition making additional diversity less valuable.With a well-initialized cognition repository and structured Analyzer feedback, UCB1’s value-guided selection can concentrate on productive design patterns and accelerate convergence.Source: §5.2.3 · p. 11, Figure 4 · p. 11
  • 2.63597: UCB1 with GPT-5-mini found a circle-packing solution matching SOTA in 17 steps, while MAP-Elites with the same model took 79 steps to reach 2.63597.The authors present this as an example of cognition-guided, exploitation-oriented sampling accelerating discovery.Source: §5.2.3 · p. 11

5.3 Ablation Study: Validating Component Effectiveness

  • The circle-packing ablation plots the full method against ablated variants, with shaded regions indicating variability across repeated runs.The supplied passage identifies the ablation figure’s scope but gives no variant-specific results or values.Source: Figure 5 · p. 12

5.3 Ablation Study: Validating Component Effectiveness

The ablations indicate that Cognition improves early exploration, while the Analyzer supports more sustained and reproducible gains; without either component, ASI-Evolve can still evolve through trial and error.

5.3.1 Ablation design

  • The controlled ablation compares the full method with versions lacking the Analyzer or Cognition, running each configuration three times.The evaluation examines overall improvement rate, convergence behavior, and cross-run gaps from evolution curves.Source: §5.3.1 · p. 12

5.3.2 Impact of removing Analyzer

  • Without the Analyzer, evolution plateaus after a high start and sustained improvement is slower and less reliable.The high initial scores are attributed to Cognition’s literature-derived priors; some runs still reach SOTA-level scores, aided by raw evaluation feedback and the relative simplicity of circle packing.Source: §5.3.2 · p. 12

5.3.3 Impact of removing Cognition

  • Removing Cognition delays productive exploration, but self-guided trial-and-error still enables sustained evolution.After a prolonged low-score period, the No Cognition curve improves and enters a higher-scoring regime; in novel domains without effective priors, the framework remains usable but requires longer exploration.Source: §5.3.3 · p. 13

5.4 Validating Real-World Applicability: Drug–Target Interaction Discovery

ASI-Evolve applied an evolved DrugBAN architecture to drug–target interaction prediction, showing benchmark gains that also extend to cold-start settings with unseen drugs and proteins.

Task Formulation and Methodology

  • Starting from DrugBAN, ASI-Evolve used cognition from approximately 80 papers to evaluate 100+ candidate architectures over 100+ rounds across four DTI datasets and four generalization settings.The Researcher proposed modifications, the Engineer trained candidates on BindingDB development data, and the Analyzer assessed AUROC, AUPRC, F1, and MCC while flagging failure modes.Source: §5.4 · p. 13

Results

  • On BindingDB development, AUROC rose +1.91 points (0.9415 →0.9606) and F1 +2.95 points (0.8689 →0.8984); BindingDB-Random gains were +1.05 AUROC and +1.39 F1.The architecture also achieved 0.9889 AUROC on Human and gains of +0.25 AUROC and +0.23 F1 on BioSNAP.Source: §5.4 · p. 13

Generalization Analysis

  • Cold-start gains reached +6.94 AUROC points for unseen drugs, +3.56 points for unseen proteins, and +4.36 points in doubly-cold-start evaluation.These gains exceed the in-distribution improvements, according to the paper.Source: §5.4 · p. 14, Table 6 · p. 14

Analysis

  • The best architecture, ban_sinkhorn_ds_marginal_topk_v6, combines Sinkhorn attention, domain-specific marginalization over drug substructures and protein domains, and learnable top-k sparse gating.The design aims to balance attention, model compositional interaction patterns, and reduce noise from irrelevant molecular features.Source: §5.4 · p. 14
  • As experiments accumulated, the system combined cross-paper ideas to derive Sinkhorn attention and used Analyzer-identified failure modes to guide mitigation with entropy regularization and orthogonality constraints.Reported bottlenecks included representational collapse, overfitting to binding sites, and attention saturation.Source: §5.4 · p. 14

Results

  • Against TransformerCPI, PSICHIC, ConPlex, and ColdStartCPI, the discovered architecture achieved competitive or superior benchmark performance.Those expert-designed methods use pre-trained molecular encoders, protein language models, or specialized graph convolutions.Source: §5.4 · p. 14

6 Conclusion

ASI-Evolve uses a closed learn–design–experiment–analyze loop to support autonomous research, with reported breakthroughs across model architecture, training data, and training algorithms. The conclusion also highlights real-world deployment beyond the AI pipeline and envisions AI taking on broader search and implementation work as scientists focus on defining problems.

6 Conclusion

  • With a structured cognition base and dedicated analyzer, ASI-Evolve achieves rapid cold-start, sustains continuous improvement, and reliably reaches SOTA-level results.The framework was evaluated through controlled comparisons against evolutionary baselines and systematic ablation studies.Source: §6 · p. 14
  • The closed research loop demonstrates breakthroughs across model architecture, training data, and training algorithms despite their implementation complexity, iteration cost, and indirect feedback.The conclusion presents these as central stages of AI development that AI can accelerate through self-improvement.Source: §6 · p. 14
  • The drug-target interaction experiment shows that AI-discovered model designs can be deployed in real-world tasks, indicating practical scientific value beyond the core AI pipeline.Source: §6 · p. 14
  • The authors envision expanding AI self-acceleration across architecture, data, algorithms, and infrastructure, with scientists defining problems while AI searches hypothesis spaces and handles implementation and iteration.They expect this shift to support the self-evolution of the entire AI field, beyond individual models.Source: §6 · pp. 14–15

A. Analysis Configuration Details SII-GAIR · A Analysis Configuration Details

This appendix records the experimental settings for circle-packing analysis, including model and framework comparisons, sampling and ablation protocols, and the shared cognition repository. It specifies the repository’s geometric knowledge and optimization methods used in the experiments.

A Analysis Configuration Details

  • The framework comparison aligned OpenEvolve-style prompting and compared GPT-5-mini with Qwen3-32B under the same system design.The shared system used max size 70, sample_n=3, a 300-second engineer timeout, four parallel workers, cognition retrieval top-k=5, and three runs per model. GPT-5-mini used temperature 0.7 and max tokens 32768; Qwen3-32B used temperature 0.6 and max tokens 65536, with top-p=0.95 and its specified additional decoding settings.Source: A Analysis Configuration Details · p. 18
  • The sampling comparison changed only the database algorithm, contrasting island-based MAP-Elites with UCB1 across three independent runs per strategy.The backbone was Qwen3-32B, and prompt template, cognition setting, full-file evolution mode, timeout, parallelism, and database size were held fixed; UCB1 used exploration constant c=1.414.Source: A Analysis Configuration Details · p. 18
  • The ablation study disabled either the Analyzer or cognition while keeping the rest of the Researcher–Engineer–Analyzer loop unchanged where applicable.Conditions used GPT-5-mini, three independent runs, and a shared setup with Judge disabled; curves report mean trends and variability across runs.Source: A Analysis Configuration Details · p. 18
  • Experiments using cognition shared a knowledge base initialized before evolution, containing 12 items across three categories.The categories include geometric priors and optimization methodology; the passage also identifies a third category without naming it in the supplied text.Source: A Analysis Configuration Details · p. 18
  • The geometric priors emphasize variable-radius arrangements, boundary and corner handling, and an n=26 target sum-of-radii of approximately 2.635.The supplied repository notes attribute that target to AlphaEvolve and identify variable radii and corner optimization as critical factors.Source: A Analysis Configuration Details · p. 18
  • The optimization methodology uses SLSQP to maximize summed radii under no-overlap and in-bounds constraints, warm-starting from high-scoring database nodes.The warm-start threshold is score ≥2.2; the supplied method also describes 3–5 independent starts from strong nodes and retaining the best optimized result.Source: A Analysis Configuration Details · p. 18

A. Analysis Configuration Details SII-GAIR

This section documents SII-GAIR’s optimization and implementation guidance, along with the settings used to align framework comparisons and configure GEPA. It emphasizes targeted refinement, constraint validity, and controls intended to reduce comparison confounds.

A. Analysis Configuration Details SII-GAIR

  • When local optimization plateaus, use scipy.optimize.differential_evolution for global refinement, optionally refining radii only before SLSQP polishing.The AlphaEvolve reference achieved 2.635 via constructor-plus-constrained-optimization, with good geometric initialization followed by strict-constraint numerical refinement.Source: A. Analysis Configuration Details… · p. 19
  • Keep construction and optimization separate, apply diff-style changes to a high-scoring code base, use epsilon 10^-8 in constraints, and verify no constraint is violated.If stuck at approximately 2.3–2.4, the checklist recommends changing the initial pattern, raising maxiter to 500–1000, using explicit inequality constraints, increasing multi-starts to 3–5, then running differential evolution and SLSQP polishing.Source: A. Analysis Configuration Details… · p. 19
  • ASI-Evolve and OpenEvolve-style runs used the same backbone model to focus comparison on framework design, while GEPA kept official settings except model parameters and scored invalid outcomes 0.The authors note this simplification may bias results relative to the best OpenEvolve configuration, but avoids confounding from auxiliary-model choice; GEPA’s other configuration and evaluation logic remained unchanged.Source: A. Analysis Configuration Details… · p. 19

Related papers

Loading 2603.29640v1…