PaperlayerView exact revision on arXiv

Source-linked AI summary

Toward Autonomous Long-Horizon Engineering for ML Research

Guoxin Chen, Jie Chen, Lei Chen, Jiale Zhao, Fanzhe Meng, Wayne Xin Zhao, Ruihua Song, Cheng Chen, Ji-Rong Wen, Kai Jia

arXiv:2604.13018v2cs.CL

TL;DR

AI research automation still struggles to turn underspecified objectives into runnable, experimentally validated systems while preserving progress under delayed, confounded feedback. AiScientist couples thin hierarchical control with a File-as-Bus workspace that preserves decision-relevant artifacts across roles and invocations. Across PaperBench and MLE-Bench Lite, it improves over matched baselines, while ablations show substantial degradation when durable state is removed.

  • Problem

    Turning underspecified research objectives into runnable, experimentally validated ML systems while sustaining cumulative progress across heterogeneous stages remains a central bottleneck.

  • Method

    AiScientist uses thin hierarchical orchestration over a File-as-Bus workspace that externalizes decision-relevant state into durable project artifacts.

  • Results

    Across PaperBench and MLE-Bench Lite, AiScientist improves over strong matched baselines and exceeds a Codex/GPT-5.5 xhigh harness reference on key metrics.

  • Takeaways & Limitations

    The results suggest that long-horizon AI research automation is a systems problem of maintaining cumulative, inspectable project progress, not only stronger local reasoning.

Abstract

from arXiv · show

Agentic systems increasingly automate pieces of AI research. Yet turning underspecified research objectives into runnable, experimentally validated ML systems remains a central bottleneck. We study this operational setting as \emph{long-horizon ML research engineering}: converting a research specification into a runnable ML system through repeated implementation, experimentation, and refinement. The central challenge is to sustain cumulative project progress across heterogeneous stages under delayed, confounded feedback. We introduce AiScientist, a multi-agent system built around thin control over thick state: a lightweight hierarchical research team coordinates through a File-as-Bus workspace that preserves decision-relevant artifacts across roles and invocations. On PaperBench, AiScientist improves over the strongest matched baselines by 9.92 and 11.15 points with Gemini-3-Flash and GLM-5, respectively. On MLE-Bench Lite, it reaches 81.82 Any Medal\% under both backbones, improving over the strongest matched baselines by 4.55 and 16.67 points, and exceeding a Codex/GPT-5.5 xhigh frontier harness reference by 13.64 Any Medal points. Ablations and process analyses show that durable project state is central to later-round refinement: removing File-as-Bus lowers PaperBench score by 6.41 points and MLE-Bench Lite Any Medal\% by 31.82 points. These results suggest that long-horizon AI research is not only a problem of stronger local reasoning, but a systems problem of maintaining cumulative, inspectable project progress.

1. Introduction

Automating AI research remains bottlenecked by turning underspecified objectives into runnable, experimentally validated systems while preserving cumulative progress across delayed feedback. AiScientist addresses this systems challenge with thin orchestration over durable project state and improves performance across two benchmarks.

  • Motivation: Turning underspecified research objectives into runnable, experimentally validated ML systems remains a central bottleneck in AI research automation.The engineering loop includes interpretation, implementation, setup, experimentation, diagnosis, and refinement.
  • Motivation: Long-horizon ML research engineering requires role continuity and project continuity because agent invocations are transient while research progress must remain cumulative.Later specialists need prior assumptions, failures, unresolved questions, and shared cross-role evidence.
  • Approach: AiScientist uses thin control over thick state, pairing hierarchical specialist orchestration with a File-as-Bus workspace that externalizes decision-relevant state into durable artifacts.The Orchestrator maintains concise summaries, directives, and a compact workspace map while specialists handle research stages.
  • Results: 9.92 and 11.15 points are AiScientist’s improvements over strongest matched baselines on PaperBench with Gemini-3-Flash and GLM-5, respectively.The evaluation also includes MLE-Bench Lite and a frontier harness reference.
  • Results: 81.82 Any Medal% is achieved on MLE-Bench Lite under both backbones, improving over strongest matched baselines by 4.55 and 16.67 points.Removing File-as-Bus lowers PaperBench score by 6.41 points and MLE-Bench Lite Any Medal% by 31.82 points, especially in later-round refinement.

2. Task Formulation

Long-horizon ML research engineering is framed as producing a runnable ML submission from an underspecified research specification under environmental, resource, and time constraints. The task combines interpretation, system setup, and iterative reasoning from delayed, confounded experimental feedback.

  • Task definition: Long-horizon ML research engineering turns a research specification into a runnable ML system evaluated through fresh execution in a clean environment.Evaluation measures both executability and whether empirical behavior satisfies the target objective.
  • Task definition: The specification, environment, resource-access policy, and time budget define the conditions under which the agent must produce a submission.
  • Task dimensions: Underspecification requires recovering implicit, scattered, or omitted implementation decisions from incomplete specifications and permitted public resources.
  • Task dimensions: System setup requires configuring environments, acquiring permitted datasets and models, and integrating those resources into a runnable system.
  • Task dimensions: Delayed and often confounded feedback can reflect interpretation, implementation, data processing, or infrastructure, making progress dependent on continuity across heterogeneous stages.

3. AiScientist

AiScientist sustains long-horizon research progress by coupling a schema-governed File-as-Bus workspace with thin hierarchical control. Specialists publish durable evidence and updates while the Orchestrator routes work through compact summaries and workspace state, enabling iterative implementation and experimentation.

  • File-as-Bus Coordination: File-as-Bus is an artifact-mediated coordination protocol in which durable evidence, decisions, and execution state remain inspectable across invocations.The workspace serves as the system of record for project progress.
  • File-as-Bus Coordination: The File-as-Bus schema declares each artifact’s purpose, update mode, permitted writers, and readers, giving specialists stable contracts for coordination.
  • File-as-Bus Coordination: The workspace map combines durable workspace state with schema metadata to produce a compact index that directs agents to relevant artifacts.Agents inspect task-specific workspace contents on demand after starting from this compact view.
  • File-as-Bus Coordination: Append-only, versioned, and mutable update modes preserve chronological evidence, recoverable canonical state, and lightweight control information for different artifact lifecycles.
  • Continuity: Persistent role artifacts and explicit cross-role dependencies support continuity from paper comprehension through prioritization, implementation, and experimentation.
  • Hierarchical Research Team: The Orchestrator maintains compact control context, monitors stage progress, and selectively invokes specialists through the Agent-as-Tool abstraction.Specialists receive directives and workspace maps, then write concise summaries and detailed workspace updates.
  • Hierarchical Research Team: Specialists cover paper comprehension, prioritization, implementation, experimentation, and auxiliary support across the major stages of ML research engineering.
  • Adaptive Research-Engineering Loop: An adaptive loop alternates implementation and experimentation so recorded failures and successful changes become durable evidence for later refinement.This lets later invocations continue from prior evidence instead of rediscovering it under finite time budgets.

4. Experiments

AiScientist is evaluated on complementary long-horizon ML engineering benchmarks against matched autonomous systems and contextual frontier references. Results show stronger performance, continued late-budget improvement, and evidence that durable state and organized engineering effort support gains.

  • Main results: AiScientist improves over the strongest matched PaperBench baselines by 9.92 points with Gemini-3-Flash and 11.15 points with GLM-5.The best AiScientist backbone also exceeds the Codex/GPT-5.5 xhigh harness reference by 4.28 points on PaperBench average score.
  • Main results: 81.82 Any Medal% is achieved on MLE-Bench Lite under both backbones, improving over the strongest matched baselines by 4.55 and 16.67 points.AiScientist also exceeds the Codex/GPT-5.5 xhigh harness by 13.64 Any Medal points.
  • Long-horizon improvement dynamics: AiScientist starts slower than AIDE and Codex but continues improving after they largely plateau, with its advantage broadening across paired runs over the full budget.Many runs require late-budget refinement to reach their final best score.
  • Mechanism analysis: Removing File-as-Bus lowers PaperBench average score by 6.41 points and MLE-Bench Lite Any Medal by 31.82 points, while minimally runnable outcomes remain comparatively intact.The ablation indicates durable state continuity primarily supports later-round diagnosis and improvement rather than first-pass validity.
  • Mechanism analysis: Without File-as-Bus, hierarchical AiScientist still improves over a simpler agent baseline by 4.74 PaperBench points and by 22.73 Above Median and 9.09 Any Medal points on MLE-Bench Lite.This indicates gains are not reducible to interaction count alone.
  • Behavioral analysis: AiScientist concentrates more workflow steps in implementation, experimentation, and validation, while higher-scoring trajectories emphasize later validation and closure rather than simply more experimentation.Without durable state, higher-scoring trajectories require substantially more implementation and experimentation, consistent with less efficient evidence reuse.

5. Related Work

Recent systems automate distinct parts of AI research, from discovery and literature synthesis to engineering and reproduction. This work studies how to sustain coherent research-engineering progress over long horizons.

  • Recent AI research automation spans scientific discovery, research assistance, objective-driven ML engineering, and paper-to-code reproduction.

6. Conclusion

Long-horizon ML research engineering requires agents to turn underspecified objectives into runnable systems while sustaining progress under delayed, confounded feedback. AiScientist addresses this setting with durable cross-role state and shows improvements across two benchmarks, alongside degradation when that state is removed.

  • Long-horizon ML research engineering turns underspecified objectives into runnable systems under delayed, confounded experimental feedback.
  • AiScientist uses hierarchical orchestration over a File-as-Bus workspace to preserve decision-relevant artifacts across roles and invocations.
  • AiScientist improves over strong matched baselines across PaperBench and MLE-Bench Lite and exceeds a frontier Codex/GPT-5.5 harness reference on key metrics.
  • Recent work spans automated scientific discovery, objective-driven ML engineering, and paper-to-code reproduction, while AiScientist focuses on sustaining coherent progress over long horizons.

B. File-as-Bus Implementation Details

File-as-Bus uses explicit artifact schemas and role-scoped access to make the shared workspace a durable collaboration substrate. Different update modes preserve current state, chronological evidence, and runnable project files for downstream coordination.

  • Each File-as-Bus artifact declares its purpose, update mode, writer set, and reader set.The schema defines which evidence specialists consult, which artifacts they own, and which downstream roles consume their outputs.
  • Versioned artifacts keep a canonical current form while preserving prior revisions conceptually for analyses and plans.
  • Append-only logs preserve chronological implementation rationales, experiment traces, failures, and diagnoses without rewriting prior evidence.
  • Mutable runnable artifacts include source code, setup scripts, and execution entry points, with their decision history preserved in append-only logs.
  • Role-scoped access lets specialists write primarily to responsibility-specific artifacts while inspecting upstream evidence relevant to their decisions.Tier-2 subagents default to read-only access, and the workspace map is refreshed after each specialist invocation.

C. Evaluation and Metric Details

The evaluation uses task–seed trajectories and reports benchmark-specific protocols alongside anytime, uncertainty, and pairwise metrics. These views distinguish improvement magnitude from how broadly AiScientist leads matched references.

  • The supplementary material includes anonymized implementation, evaluation scripts, and experiment artifacts.
  • PaperBench evaluates 20 from-scratch paper-replication tasks with one H20 GPU and a 24-hour budget under the official full-evaluation protocol.
  • MLE-Bench Lite contains 22 competition-style ML tasks, with three runs or seeds and Any Medal% as the primary metric.
  • The anytime score curve plots mean best-so-far normalized validation scores across valid task–seed trajectories at each wall-clock time.Its shaded band is ±1 standard error of the mean across trajectories.
  • The pairwise lead-rate curve measures how broadly AiScientist leads matched references, assigning 1 for a lead, 0.5 for a tie, and 0 for a reference lead.The score curve measures improvement magnitude, whereas lead rate measures breadth across matched trajectories.

D. Baseline and Reference Details

The evaluation distinguishes controlled matched baselines from contextual references, enabling direct system comparisons while separately calibrating results against external systems and leaderboard outcomes.

  • Matched baselines are evaluated under the same controlled setup as AiScientist, supporting direct system-level comparisons.
  • Codex with GPT-5.5 xhigh is a frontier harness reference, not a matched baseline, because its harness and backbone differ from the controlled comparisons.
  • PaperBench uses BasicAgent and IterativeAgent as matched baselines under the official task suite and grading setup.
  • MLE-Bench Lite compares AiScientist against AIDE, LoongFlow, and ML-Master 2.0 in the controlled setup, while official leaderboard rows provide contextual ecosystem references.

E.1. Delegation Patterns on PaperBench

AiScientist regularly delegates specialist and subagent work across PaperBench tasks while the Orchestrator retains stage-level control through the thin-control design.

  • AiScientist regularly invokes specialist and subagent work across PaperBench tasks.
  • The delegation pattern is consistent with thin control: focused agents perform substantial local work while the Orchestrator maintains stage-level control.
  • Delegated outputs are folded back into the shared workspace for continued project progress.

E.2. Budget and Medal Outcomes on MLE-Bench Lite

The MLE-Bench Lite analyses examine how performance and medal outcomes develop over time and budget, highlighting the importance of later-budget refinement for stronger competitive results.

  • Later-budget refinement is important for converting runnable submissions into stronger competitive outcomes on MLE-Bench Lite.
  • The analysis complements score trajectories, pairwise lead rates, and time to final best score by focusing on budget-dependent medal outcomes.
  • Subagent delegation appears across most PaperBench tasks rather than being concentrated in a small number of tasks, indicating systematic hierarchical-team use.
  • The budget analysis summarizes how medal-level outcomes depend on the available budget under GLM-5.
Loading 2604.13018v2…