PaperlayerView exact revision on arXiv

Source-linked AI summary

AutoSOTA: An End-to-End Automated Research System for State-of-the-Art AI Model Discovery

Yu Li, Chenyang Shao, Xinyang Liu, Ruotong Zhao, Peijie Liu, Hongyuan Su, Zhibin Chen, Qinglong Yang, Anjie Xu, Yi Fang, Qingbin Zeng, Tianxing Li, Jingbo Xu, Fengli Xu, Yong Li, Tie-Yan Liu

arXiv:2604.05550v2cs.CLcs.CE

TL;DR

AI research requires lengthy reproduction, debugging, and optimization cycles, while existing automation does not cover the full path from published paper to improved executable method. AutoSOTA addresses this gap with an eight-agent end-to-end system for grounding papers, running and repairing experiments, generating constrained improvements, and validating results. Across a filtered collection of papers, it discovered 105 new SOTA models and is presented as infrastructure for reducing repetitive experimental work.

  • Problem

    AI model improvement depends on prolonged reproduction, debugging, evaluation, and refinement, but existing systems do not automate the full path from top-tier paper to empirically superior executable repository.

  • Method

    AutoSOTA uses eight specialized agents across resource preparation, experiment evaluation, code optimization, and reflection and ideation, with supervision against invalid metric gains.

  • Results

    AutoSOTA discovered new SOTA models for 105 papers, averaging nearly 10% improvement over original main methods with about five hours of execution per paper.

  • Takeaways & Limitations

    The results suggest end-to-end research automation can support empirical advancement across heterogeneous AI areas while serving as infrastructure that absorbs repetitive experimental iteration.

  • Takeaways & Limitations

    The current evaluation and refinement loops emphasize result matching and may overlook structural and semantic fidelity, while 20 of 125 baselines experienced invalid optimizations.

Abstract

from arXiv · show

Artificial intelligence research increasingly depends on prolonged cycles of reproduction, debugging, and iterative refinement to achieve State-Of-The-Art (SOTA) performance, creating a growing need for systems that can accelerate the full pipeline of empirical model optimization. In this work, we introduce AutoSOTA, an end-to-end automated research system that advances the latest SOTA models published in top-tier AI papers to reproducible and empirically improved new SOTA models. We formulate this problem through three tightly coupled stages: resource preparation and goal setting; experiment evaluation; and reflection and ideation. To tackle this problem, AutoSOTA adopts a multi-agent architecture with eight specialized agents that collaboratively ground papers to code and dependencies, initialize and repair execution environments, track long-horizon experiments, generate and schedule optimization ideas, and supervise validity to avoid spurious gains. We evaluate AutoSOTA on recent research papers collected from eight top-tier AI conferences under filters for code availability and execution cost. Across these papers, AutoSOTA achieves strong end-to-end performance in both automated replication and subsequent optimization. Specifically, it successfully discovers 105 new SOTA models that surpass the original reported methods, averaging approximately five hours per paper. Case studies spanning LLM, NLP, computer vision, time series, and optimization further show that the system can move beyond routine hyperparameter tuning to identify architectural innovation, algorithmic redesigns, and workflow-level improvements. These results suggest that end-to-end research automation can serve not only as a performance optimizer, but also as a new form of research infrastructure that reduces repetitive experimental burden and helps redirect human attention toward higher-level scientific creativity.

1 Introduction

AutoSOTA targets the prolonged, artifact-heavy cycle required to turn published AI methods into reproducible and improved SOTA systems. Its multi-agent workflow supports replication, optimization, and broader research automation across heterogeneous AI areas.

  • Motivation: AutoSOTA formulates end-to-end discovery as translating top-tier AI papers into executable repositories that empirically surpass their original methods.The workflow addresses resource preparation, evaluation, and reflection-driven improvement rather than isolated code generation.
  • Approach: Eight specialized agents divide paper grounding, environment execution, experiment monitoring, optimization, scheduling, repair, and validity supervision.The architecture distributes long-horizon research work across complementary roles instead of relying on a monolithic routine.
  • Results: 105 papers yielded new SOTA models, with nearly 10% average improvement over original main methods and about five hours of execution per paper.The evaluation covered papers from eight top-tier AI conferences filtered for code availability, repository readiness, and tractable execution cost.
  • Results: Case studies across LLM, NLP, computer vision, time series, and optimization show interventions beyond routine hyperparameter tuning, including architectural, algorithmic, and workflow-level changes.These results support broad empirical advancement across heterogeneous AI research areas.
  • Significance: AutoSOTA is positioned as research infrastructure that absorbs repetitive experimental iteration while redirecting human effort toward deeper conceptual innovation.The proposed human–AI collaboration separates large-scale execution and reflection from higher-level scientific creativity.

2 Research Problem

The research problem is to transform an unstructured top-tier AI paper into an executable repository whose method improves on the paper’s reported performance. This requires coordinating resource grounding, faithful evaluation, code-level optimization, and reflection-driven scientific search under long-horizon uncertainty.

  • Research objective: AutoSOTA defines discovery as a function from an unstructured research paper to an executable repository containing an algorithmic variant that strictly outperforms the published method.The output is an advanced GitHub-style repository, distinguishing scientific discovery from ordinary code generation.
  • Resource Preparation and Goal Setting: Resource preparation grounds a paper in repositories, datasets, base models, and a precisely defined target metric for baseline replication.The main difficulty is fragmented resources and dense, multidimensional empirical reporting.
  • Experiment Evaluation: Experiment evaluation must synthesize a functional repository whose empirical evaluation closely approximates the target metric, eval(Rrep) ≈ g∗.This phase converts static research artifacts into a dynamic execution pipeline for later evolution.
  • Code Optimization: Code optimization translates abstract improvement ideas into executable candidate repositories while coordinating code variations and rapid evaluator feedback.The challenge is mapping structural or algorithmic concepts into syntactically correct, context-aware modifications.
  • Reflection & Ideation: Reflection and ideation searches for validated candidates satisfying eval(R∗) > eval(Rrep), favoring structural innovations over trivial hyperparameter tuning.The stage must balance exploration and exploitation while using historical feedback and costly experimental trials.
  • Cross-stage challenge: The four stages compound errors from knowledge extraction through software engineering, code mutation, and autonomous discovery, requiring fault tolerance under ambiguity.The problem therefore spans both scientific reasoning and robust long-horizon execution.

3 AutoSOTA Framework

AutoSOTA implements an eight-agent, fault-tolerant workflow that converts papers and distributed dependencies into execution-ready tasks, then iteratively evaluates and improves candidate repositories. Its modules separate acquisition, execution, repair, ideation, scheduling, and scientific validation while maintaining feedback across the pipeline.

  • Architecture Design: Eight bounded agents decompose resource acquisition, evaluation, code execution, reflection, scheduling, and scientific validation to avoid compounding errors in long-horizon research.The architecture separates physical operations from logical evaluation and hypothesis generation from validation.
  • Architecture Design: AgentResource grounds papers to repositories and dependencies, while AgentInit, AgentMonitor, and AgentFix support resilient baseline execution and repair.AgentFix uses cross-task failure memory for protocol-preserving engineering fixes.
  • Architecture Design: AgentIdeator constructs protocol-respecting hypotheses, AgentScheduler manages resources and experiment state, and AgentSupervisor blocks invalid shortcuts through a Red Line System.The loop routes newly generated ideas back into physical evaluation while preserving comparability with the original baseline.
  • Paper-to-Repository Grounding and Curation: AgentResource converts papers into machine-addressable units by normalizing metadata and identifying empirically validated methodological papers from eight top-tier conferences.The resulting filtering narrows heterogeneous literature into candidate optimization tasks.
  • Paper-to-Repository Grounding and Curation: Repository discovery extracts direct or indirect project links, selects relevant official repositories, removes duplicates, and shallow-clones normalized local copies.The structured output includes linked repositories and readiness signals for downstream processing.
  • External Resource Acquisition: External Resource Acquisition bridges repository code and execution readiness by identifying and downloading datasets, pretrained weights, and specialized checkpoints.These dependencies are often absent from repositories or referenced only implicitly.
  • External Resource Acquisition: The acquisition pipeline separates zero-download symbolic dependency discovery from validated physical downloading through a centralized Global Resource Registry.The registry records source URLs, resource types, estimated sizes, and total projected workload for scheduling.
  • External Resource Acquisition: Size gates and historical-state checks restrict downloads to admissible workloads and skip tasks already reaching terminal success or failure states.This reduces runaway downloads and redundant execution before assets are materialized.

3.3 AgentObjective: Automated Objective Rubric Construction

AgentObjective automates evaluation-rubric construction by decomposing paper-level replication goals into verifiable, evidence-grounded sub-tasks. Hierarchical context injection combines paper, visual, and code knowledge while current refinement remains chiefly results-centric.

  • Motivation: Manual expert involvement makes rubric construction difficult to scale for large-scale, closed-loop scientific optimization.
  • AgentObjective: AgentObjective automatically maps the macro research goal into a structured evaluation hierarchy with explicit constraints and evidentiary support from the paper and codebase.
  • Tree-Structured Recursive Decomposition: The tree-structured rubric recursively decomposes replication into high-level modules and specific, verifiable facets across the research lifecycle.
  • Tree-Structured Recursive Decomposition: Breadth-First Search expands the rubric from the complete-replication root into binary Pass/Fail sub-tasks, using weight-driven termination to regulate depth and efficiency.
  • Hierarchical Context Injection: The rubric’s knowledge pool integrates paper text, visual information from tables and figures, and engineering logic from the official codebase.
  • Hierarchical Context Injection: Hierarchical context delivery injects macro information at shallow layers and increasingly detailed technical or experimental information as decomposition proceeds.
  • Hierarchical Context Injection: This injection framework turns static paper and code knowledge into task-relevant validated support for a high-fidelity diagnostic rubric.
  • Limitations: Current evaluation and refinement emphasize Result Match values for the proposed and best baseline methods, leaving structural and semantic fidelity for future expansion.

3.5 AgentMonitor: Tracking the state of experiment and preventing deadlock

AgentMonitor supervises long-horizon replication by tracking execution state, intervening at dead ends, and managing budgets. Its external memory and retrieval mechanisms preserve coherent reasoning beyond a single context window.

  • Motivation: Long-horizon replication requires repeated command execution, output inspection, code editing, dependency installation, and recovery across partially observed failures.
  • Supervision: AgentMonitor observes execution traces and provides high-level interventions without replacing the main execution agent.
  • Supervision: Online trace interpretation detects failure patterns while persistent state tracking supplies cross-iteration context for long-horizon reasoning.
  • State Tracking: Phase-aware monitoring distinguishes setup, installation, launch, evaluation, reporting, and failure handling during streamed execution.
  • Intervention: Corrective guidance redirects the agent toward more promising execution strategies rather than prescribing brittle low-level patches.
  • Budget and Safety: The monitor enforces wall-clock, interaction-round, timeout, and process-cleanup controls as the system’s global budget and safety controller.
  • External Memory: Code, outputs, modification history, ideas, and research insights exceed a single context window, motivating structured external memory.
  • External Memory: Repository distillation plus on-demand command-line retrieval bounds context use while maintaining a coherent optimization view across long horizons.

3.6 AgentFix: Resolving Runtime Conflicts

AgentFix converts recurring runtime failures into a protocol-preserving repair process based on reusable skills and failure memory. AgentIdeator complements this reliability layer by generating admissible, constraint-aware optimization hypotheses.

  • AgentFix: Academic repositories exhibit recurring failures including dependency, CUDA, networking, path, checkpoint, and evaluation problems that generic reasoning must repeatedly repair.
  • AgentFix: AgentFix maps observed failures to validated repair actions using structured skills and reusable historical failure memory.
  • Repair Skills: Explicit repair skills address package-installation and network failures through predefined environment-aware procedures rather than improvised fixes.
  • Protocol Preservation: Repairs remain constrained by preservation of the original scientific protocol.
  • Failure Memory: Failure memory records normalized errors, attempted remedies, and outcomes to prevent cycling among equivalent unsuccessful fixes.
  • AgentFix: AgentFix serves as AutoSOTA’s operational reliability layer, improving recovery while preserving experimental comparability.
  • AgentIdeator: AgentIdeator constructs a structured, protocol-respecting search space of admissible improvement hypotheses before expensive optimization begins.
  • AgentIdeator: External research expertise broadens candidate generation beyond static repository inspection toward structural and domain-aware possibilities.

3.8 AgentScheduler: Lifecycle Management for Optimizing Research Codebases

AgentScheduler orchestrates each paper from initialization through iterative optimization, while version control, resource scheduling, persistence, and artifact verification support reliable unattended operation. Its Leap Path deliberately seeks structural improvements after repeated parameter tuning.

  • Lifecycle Management: AgentScheduler manages environment initialization, baseline measurement, code modification, evaluation, state persistence, and result export across the paper lifecycle.
  • Optimization Workflow: The workflow proceeds through initialization and baseline measurement, code understanding, idea-library construction, and iterative optimization.
  • Optimization Workflow: LLM proposals combine repository-wide code understanding with awareness of state-of-the-art research to identify implementable improvements.
  • Phase 0: Phase 0 containerizes the target paper, records the unmodified baseline, and measures reproducible metrics before optimization.
  • Phase 1: Phase 1 creates a repository cognitive map and records immutable experimental constraints for subsequent decisions.
  • Phase 2: Phase 2 organizes candidate ideas by type, priority, risk, and micro, meso, or macro granularity.
  • Phase 3: Each Phase 3 iteration reflects, selects an idea, snapshots code, implements and evaluates it, debugs failures, records results, and updates the library.
  • Optimization Strategy: After three consecutive parameter-only iterations, the Leap Path requires a structurally informed idea and grants additional exploration before rollback if needed.

3.9 AgentSupervisor: Preventing Invalid Optimization Behaviors

AgentSupervisor constrains long-horizon optimization with non-negotiable rules that preserve comparability, evaluation integrity, and legitimate improvement. Multi-layer audits identify paper-specific constraints and reject violating ideas before execution.

  • Constraint framework: The Red Line System defines six non-negotiable constraints governing valid optimization behavior.These constraints cover evaluation metrics, evaluation scripts, model outputs, metric trade-offs, dataset integrity, and dataset modification.
  • Evaluation integrity: Evaluation parameters, scripts, and aggregation procedures must remain unchanged, with optimization restricted upstream of the evaluation boundary.Examples include preserving recall@k, context windows, history windows, and run-aggregation strategies.
  • Data and output integrity: Model outputs must come from actual inference, while train/test splits and evaluation distributions must not be contaminated or altered.Fabricated predictions, test-data training, relabeling, filtering, and resampling are prohibited.
  • Metric trade-offs: The primary metric cannot improve through significant degradation of other reported metrics, and every iteration must report all metrics.This explicitly constrains unfair trade-offs between metric dimensions.
  • Multi-layer supervision: Paper-specific constraints are identified before idea generation and checked through audits before candidate ideas enter execution.The workflow includes a dedicated hard-constraints list, a post-idea Red Line Audit, and another check for dynamically generated Leap candidates.
  • Validity boundary: Optimization gains that violate these boundaries are considered invalid regardless of their magnitude.The AgentScheduler explores within the boundaries defined by AgentSupervisor, which operationalizes legitimate improvement.

4.1 Experiment Setup

The evaluation benchmark was built from recent empirical papers at eight major AI conferences and progressively filtered for reproducible, executable optimization. The final testbed contains 125 method-driven papers selected under artifact, environment, and computational constraints.

  • Benchmark construction: The benchmark began with accepted papers from eight premier AI conferences published within the past year.The conferences included NeurIPS, ICLR, ICML, CVPR, ICCV, ACL, NAACL, and AAAI.
  • Benchmark construction: The screening process excluded theoretical, review, and purely analytical work while retaining empirically validated methods with public code artifacts.This methodological and artifact-availability filtering targeted papers suitable for automated closed-loop optimization.
  • Computational screening: 745 papers entered preliminary empirical testing under strict environment and execution-cost constraints.Papers with insurmountable installation failures or excessive execution demands were filtered out.
  • Final testbed: 125 method-driven research papers remained as the final empirical testbed for replication and SOTA-surpassing optimization.AutoSOTA subsequently optimized 105 of these 125 baselines.

4.2 Main Result

AutoSOTA improves replicated research baselines across diverse domains, with many gains arising from substantive algorithmic innovation rather than only parameter tuning. The results also expose a measurable rate of invalid optimizations that supervision does not eliminate.

  • Result reporting: Table 1 reports AutoSOTA Improvement on each paper’s primary optimization metric after replication of the original main method.The table’s improvement values are tied to the scientifically significant core metric g* for each paper.
  • Cross-domain performance: AutoSOTA produced gains across multidisciplinary tasks ranging from medical time-series analysis to protein binding energy prediction.The reported examples support robustness across diverse scientific domains while maintaining computational efficiency.
  • Improvement types: 8.82% was the mean enhancement for Class A, compared with 5.67% for Class B.The reported comparison distinguishes substantive algorithmic innovation from hyperparameter-focused optimization.
  • Validity and limitations: 20 of 125 cases, or 16%, were classified as invalid optimizations after AgentSupervisor failed to detect or prevent potential violations.The remaining approximately 84% maintained methodological consistency and evaluation comparability under the supervision mechanism.

4.3 Case Study

AutoSOTA’s case studies show that its automated optimization can improve diverse research systems through bug fixes, graph redesign, data integration, architectural changes, algorithmic refinement, and hybrid neural-symbolic search. Across LLM, NLP, biology, vision, time series, and optimization examples, gains extend beyond routine hyperparameter tuning.

  • LLM: AutoSOTA improves LLM inference throughput from 674.58 to 765.27 tok/s, a 13.44% improvement over the baseline.It traced abnormal acceptance to incorrect reuse of tree_draft_ids, then added GPU-side checking and reduced CPU–GPU synchronization.
  • NLP: AutoSOTA raises correlation_care from 0.4577 to 0.4970, an 8.59% relative gain, by smoothing graph weights and introducing multi-scale propagation.The final propagation schedule uses six resolvent operators with α = [0.84, 0.82, 0.80, 0.68, 0.49, 0.25].
  • CV: AutoSOTA improves PLCC from 0.7803 to 0.8012, a 2.68% gain, by concatenating CLS, average-pooled, and max-pooled visual features.The three-way feature blending redesign addresses information loss in the original single pooling path without larger backbones or additional training data.
  • Time series: AutoSOTA reduces time-series MAE to 14.096, an approximate 19.0% improvement over the original paper’s baseline, through targeted algorithmic refinements.It discarded changes that improved NLL but not MAE, demonstrating self-correction under a 300-sampling-run compute boundary.
  • Optimization: AutoSOTA reduces the optimization problem’s optimality gap to 0.444%, a 51.4% relative improvement over baseline, by combining neural changes, inference scaling, and routing heuristics.Inference time increases from approximately 3.7 to 103 seconds, while the system also manages memory through batch-size reduction.

5 Related Works

Related work progresses from AutoML’s search over hyperparameters and architectures, to LLM-based code and algorithm discovery, and finally to systems targeting automated scientific research. These lines of work establish increasingly broad automation but differ in the scope of the workflow they address.

  • Automated machine learning: Traditional AutoML primarily searches hyperparameters and neural architectures, using methods such as reinforcement learning, evolutionary algorithms, and differentiable search.DARTS introduced differentiable search spaces to reduce computational cost.
  • LLM-based discovery: LLM-based systems extend automation to programmatic logic, repository navigation, software debugging, and scientific or algorithmic discovery.Examples include SWE-agent, AutoCoder, and FunSearch.
  • AI Scientists: AI Scientist systems aim to automate the broader research lifecycle from idea generation and experimentation to manuscript drafting.AI Scientist and AI Scientist v2 represent this trajectory toward fully automated discovery.

6 Conclusion

AutoSOTA presents an end-to-end multi-agent workflow that turns academic literature into reproducible, empirically improved implementations while enforcing safeguards against methodological shortcuts. The conclusion positions it as infrastructure for empirical research automation, while identifying broader modalities and closed-loop ideation as future directions.

  • Contribution: AutoSOTA coordinates eight specialized agents across resource preparation, experiment evaluation, code optimization, and reflection and ideation.AgentSupervisor’s Red Line System is intended to keep discovered innovations rigorous, reproducible, and free from methodological shortcuts.
  • Implications: AutoSOTA absorbs environment setup, baseline reproduction, and empirical tuning so researchers can focus on higher-level theoretical design, causal reasoning, and conceptual breakthroughs.The paper frames this as a blueprint for human-AI collaboration and research infrastructure.
  • Future work: Future work includes extending the objective rubric to hardware-in-the-loop and robotics settings and integrating the execution engine with autonomous macro-ideation frameworks.The proposed direction is a closed loop that invents, validates, and publishes new algorithmic paradigms.
Loading 2604.05550v2…