PaperlayerView exact revision on arXiv

Source-linked AI summary

AutoLab: Can Frontier Models Solve Long-Horizon Auto Research and Engineering Tasks?

Zhangchen Xu, Junda Chen, Yue Huang, Dongfu Jiang, Jiefeng Chen, Hang Hua, Zijian Wu, Zheyuan Liu, Zexue He, Lichi Li, Shizhe Diao, Jiaxin Pei, Jinsung Yoon, Hao Zhang, Mengdi Wang, Radha Poovendran, Misha Sra, Alex Pentland, Zichen Chen

arXiv:2606.05080v1cs.AIcs.LG

TL;DR

Existing evaluations provide limited evidence about whether agents can sustain iterative optimization over long horizons. AutoLab evaluates agents on 36 tasks across four domains, finding that persistent empirical iteration, rather than initial solution quality, is the dominant predictor of success.

  • Problem

    Existing evaluations focus largely on single-turn or short interactive tasks, while newer hour-long closed-loop optimization benchmarks remain limited in scale and generality.

    Source: §1 · p. 1

  • Method

    AutoLab evaluates agents on 36 executable tasks across four domains, each requiring iterative improvement of a correct but suboptimal baseline under a strict wall-clock budget.

    Source: §1 · p. 2

  • Results

    claude-opus-4.6 leads every sub-domain with Avg@3 of 0.68 versus 0.50 for the next-best model, and persistence predicts performance better than initial solution quality.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    The findings identify time awareness, persistence, and more effective harness design as priorities for future ultra-long-horizon agents.

    Source: §6 · p. 13

    AutoLab benchmarks measurable auto-research in executable system and machine-learning engineering workflows, not scientific discovery in its broadest sense.

    Source: Limitations and Broader Impact · p. 13

Abstract

from arXiv · show

Scientific and engineering progress is fundamentally a long-horizon iterative process: proposing changes, running experiments, measuring outcomes, and continuously refining artifacts. Yet existing benchmarks for frontier models primarily evaluate either single-turn responses or short-horizon agent trajectories, failing to capture the challenges of sustained iterative improvement over extended time horizons. To address this gap, we introduce AutoLab, a new benchmark for ultra long-horizon closed-loop optimization. AutoLab consists of 36 realistic, expert-curated tasks spanning four diverse domains: system optimization, puzzle & challenge, model development, and CUDA kernel optimization. Each task begins with a correct but deliberately suboptimal baseline and challenges agents to improve it within a strict wall-clock budget. Evaluating 17 state-of-the-art models reveals the dominant predictor of success is not the quality of an agent's initial attempt, but its persistence in repeatedly benchmarking, editing, and incorporating empirical feedback. While claude-opus-4.6 exhibits strong long-horizon optimization capabilities, most frontier models, including several proprietary ones, either terminate prematurely or exhaust their budgets with minimal progress. These results underscore the importance of time awareness and persistent iteration in autonomous agents. We open-source the full benchmark, evaluation harness, and task artifacts, to accelerate research toward truly capable long-horizon agents.

1 Introduction

AUTOLAB evaluates whether agents can sustain iterative optimization across long-horizon scientific and engineering tasks. Its evaluation finds that persistence in empirical refinement matters more than initial solution quality, while many models fail to manage their time budgets effectively.

1 Introduction

  • Long-horizon empirical progress requires repeated experimentation and refinement, but existing evaluations rarely test this regime and prior long-horizon benchmarks remain limited in scale and domain.Some empirical-optimization demonstrations also rely on heavily engineered, model-specific harnesses, tools, and search strategies, making the model’s contribution difficult to isolate.Source: §1 · pp. 1–2
  • AUTOLAB introduces 36 executable tasks across four domains, with suboptimal baselines to improve under strict wall-clock budgets and continuous scoring backed by hack-resistant checks.Its safeguards include sealed evaluators, correctness gates, immutable-file checks, and adversarial auditing.Source: §1 · pp. 2–3
  • claude-opus-4.6 led every sub-domain at Avg@3 0.68 versus 0.50 for the next-best model, in an evaluation using 2,544 wall-clock hours and 8.60 billion tokens.Figure 1 also shows claude-opus-4.6 leading the overall ranking.Source: §1 · p. 2, Figure 1 · p. 2
  • The authors evaluate 17 state-of-the-art models, including four proprietary frontier models, with a fixed standardized harness under identical experimental conditions.Source: §1 · p. 3
  • Trajectory analysis finds persistence in iterative refinement predicts final performance better than initial-solution quality, while poor time awareness leads agents to terminate early or exhaust their budgets.The analysis includes manual inspection of 302 zero-score rollouts.Source: §1 · pp. 2–3

2 The AUTOLAB Benchmark

AUTOLAB evaluates sustained empirical iteration on realistic research and engineering tasks, with continuous calibrated scoring across heterogeneous metrics and hack-resistant verification.

2.1 Task Formulation

  • Each task pairs an instruction and sandboxed, working-but-suboptimal codebase with local evaluation, a held-out verifier, human reference, and 2–12-hour budget for iterative editing and testing.Agents can edit, run, profile, and inspect intermediate outputs; the held-out verifier scores the final implementation relative to the reference.Source: §2.1 · pp. 3–4
  • AUTOLAB anchors continuous scores to baseline and reference performance, using log-stretch for performance optimization and linear interpolation for bounded quality metrics to reward partial progress.Scores are normalized to [0, 1]; for log-stretch tasks, the reference corresponds to 0.5, while linear scoring assigns the reference 1.0.Source: §2.1 · p. 4

2.2 Benchmark Construction

  • Tasks drawn from practitioners’ real workflows pass expert audits for validity, solvability, integrity, and measurement stability; sealed tests, correctness gates, adversarial probing, and file pinning deter reward hacking.At least two independent experts and a format-audit agent review each task; tasks failing any criterion are revised or rejected.Source: §2.2 · p. 5

2.3 Benchmark Composition

  • The final benchmark contains 36 tasks: 15 System Optimization, 10 Puzzle & Challenge, 7 Model Development, and 4 CUDA kernel optimization tasks.The tasks span low-level systems engineering, algorithmic challenges, the LLM development pipeline, and GPU kernel optimization.Source: §2.3 · pp. 5–6

3 Benchmark Results

On the 36-task AUTOLAB benchmark, claude-opus-4.6 leads the overall results, while models show distinct strengths across categories. The flash_attention case study illustrates how sustained, feedback-driven iteration can produce large improvements.

3.1 Experimental Setup

  • Each model–task pair receives three independent rollouts; Avg@3 captures typical performance, Best@3 the ceiling, and Dominance compares head-to-head wins using Avg@3.Dominance is bounded by 0 and 1, with 0.5 corresponding to average performance across models.Source: §3.1 · p. 6

3.2 Main Results

  • Across 36 tasks, claude-opus-4.6 leads with Avg@3 0.68 and Dominance 0.93, ahead of gemini-3.1-pro at Avg@3 0.50.Table 1 reports per-category Avg@3, Best@3, and Dominance; the prose identifies claude-opus-4.6 as the overall leader.Source: §3.2 · p. 6, Table 1 · p. 7
  • Among open-weight models, kimi-k2.6 (0.46), mimo-v2.5-pro (0.45), and glm-5 (0.43) cluster closely; deepseek-v4-flash (0.37) performs on par with much larger deepseek-v4-pro (0.38) overall.Deepseek-v4-flash is under 400B parameters and surpasses deepseek-v4-pro on CUDA kernel optimization and algorithmic puzzle tasks.Source: §3.2 · p. 7
  • claude-opus-4.6 leads all four categories; gemini-3.1-pro is strongest on puzzles but lags on CUDA and model development, while CUDA is a weakness for open-weight models.gemini-3.1-pro’s median rollout is 12 steps, versus 57 for claude-opus-4.6.Source: §3.2 · p. 7
  • On flash_attention, claude-opus-4.6 cuts runtime from about 750 ms to 18 ms in 44 feedback-driven iterations, achieving 42.4× speedup and beating the 100 ms reference.kimi-k2.6, gemini-3.1-pro, and glm-5 reach 50–80 ms but do not improve further.Source: §3.2 · p. 7, Figure 4 · p. 8
  • Prolonged per-step reasoning delays benchmarking and limits iterations; premature termination, discarding a promising result, or failing to self-verify also constrains progress.deepseek-v4-pro times out before exploiting promising directions; qwen-3.6-plus discards a strong intermediate result as illegal, and grok-4-20 runs the evaluation script once. The case study emphasizes sustained, measurement-driven iteration and time awareness.Source: §3.2 · p. 8

4 Analysis

Across AutoLab analyses, higher scores are associated with more iteration and runtime, while premature stopping, timeouts, and instruction violations limit agent performance. Harness choice also changes scores and compute costs, and additional analyses find that capability and reliability are distinct.

4.1 Cost Analysis

  • Higher average scores correlate with more agent steps and runtime; claude-opus-4.6 is a high-scoring, high-step outlier, while gpt-5.4 and grok-4-20 terminate early.The analysis also reports that higher performance generally incurs higher inference cost, while deepseek-v4-flash and mimo-v2.5-pro achieve competitive performance at substantially lower costs.Source: §4.1 · pp. 8–9, Figure 5 · p. 9

4.2 Failure Case Analysis

  • Manual inspection grouped all 302 zero-score rollouts across 11 models into four failure modes: timeout/context exhaustion, capability gap, instruction violation, and other.Timeouts include failures to submit within budget; capability gaps include incorrect or sub-threshold solutions, early give-ups, and missing required files; instruction violations break task constraints; other failures are upstream or sandbox issues.Source: §4.2 · p. 9
  • Agents show poor time awareness: some exhaust their budgets without submitting, while gpt-5.4 and grok-4-20 often submit after minimal exploration.An open-weight-specific pattern is extremely long reasoning with few actions: kimi-k2.6 timed out in all three toy_isa_opt trials at 2–11 steps, and deepseek-v4-pro submitted fewer than 10 actions in 9 of 12 CUDA trials; no closed-weight model showed this pattern.Source: §4.2 · p. 10
  • Instruction violations cluster in gemini-3.1-pro (5 cases) and glm-5 (4 cases), with ntt_butterfly_cuda accounting for half of all violations.The analysis concludes that robust instruction following remains challenging even for capable frontier models.Source: §4.2 · p. 10

4.3 Harness Ablation Analysis

  • Harness choice changes both cost and performance: for the same model, per-trial inference cost varies by more than 5×, and mini-swe-agent* boosts deepseek-v4-flash and mimo-v2.5 relative to terminus-2.The gains are +0.13 and +0.20, respectively, while gpt-5.4 declines by −0.03. Under pi-mono, gpt-5.4 scores 0.50, the highest in that harness, and the other three models score 0.21–0.27.Source: §4.3 · p. 11
  • The authors identify harness design as a way to close part of the performance gap between weaker and stronger base models without changing the models.They propose more iteration headroom for smaller models and tighter, high-quality patch loops for stronger instruction-followers.Source: §4.3 · pp. 11–12

4.4 More Analysis

  • Appendix analyses find modest, uneven generational gains across sub-domains and show that capability and reliability are correlated but distinct.claude-opus-4.6 performs strongly on both axes, while high variance among many lower-performing models makes single-trial evaluation unreliable and Best@3 can inflate their perceived strength.Source: §4.4 · p. 12

5 Related Work

Prior evaluations often stop at one-shot outputs or final states, while existing long-horizon benchmarks and agent systems leave sustained iteration, broad task coverage, or model-level comparison unmeasured.

Static and short-horizon agent benchmarks

  • Static and short-horizon suites score one-shot outputs or terminal states, but none makes sustained empirical iteration under quantitative metrics its primary evaluation target.This includes coding benchmarks and multi-step agent benchmarks that grade only a final artifact or terminal state.Source: §5 · p. 12

Long-horizon optimization and research-agent benchmarks

  • Long-horizon benchmarks cover ML, systems, kernels, and engineering, but typically narrow domain scope and score only final outputs; AutoLab spans four categories with calibrated, hack-resistant scoring.The four categories are system optimization, puzzles, model development, and CUDA kernels.Source: §5 · p. 12

Closed-loop agent frameworks and training environments

  • Closed-loop frameworks support iterative work, but leading optimization demonstrations couple models to bespoke systems; AutoLab fixes harness, interface, tasks, budgets, and scoring across models.This enables apples-to-apples comparison of long-horizon optimization capability under identical conditions.Source: §5 · p. 12

6 Conclusion

AUTOLAB evaluates frontier models on ultra long-horizon research and engineering tasks, finding that sustained iteration—not raw capability alone—is the dominant predictor of success. Claude-opus-4.6 leads through steady optimization, while most other frontier models terminate prematurely or exhaust their budgets without submitting.

6 Conclusion

  • AUTOLAB evaluates frontier models on research and engineering tasks requiring sustained iteration over hours rather than minutes.It combines ultra long-horizon tasks, continuous calibrated scoring, and strong anti-hacking safeguards.Source: §6 · p. 12
  • Claude-opus-4.6 achieves a commanding lead through long, steady optimization, while most other frontier models terminate prematurely or exhaust their budgets without submitting.The results point to persistence in evaluating, editing, and iterating over extended horizons as the dominant predictor of success.Source: §6 · p. 13
  • Future models and agents need greater time awareness, persistence, and more effective harness design.The conclusion identifies these as priorities for improving ultra long-horizon agent capabilities.Source: §6 · p. 13
  • The authors release the full benchmark, evaluation harness, and task artifacts to accelerate progress toward capable ultra long-horizon agents.Source: §6 · p. 13

Limitations and Broader Impact · A Task Specifications

AutoLab evaluates measurable system and machine-learning engineering workflows, not scientific discovery in its broadest sense, and pairs performance with trajectory and resource reporting for long-horizon evaluation.

Limitations and Broader Impact

A.1 Task Descriptions

This subsection inventories the benchmark’s 36 tasks, grouped by category, with implementation language, difficulty tier, and a short description of each optimization challenge. The tasks range from established algorithmic workloads to specialized engineering and research-style problems.

System Optimization

  • System-optimization tasks cover performance bottlenecks in cryptography, search, rendering, image processing, and spatial indexing, with challenges including SIMD, pipelining, cache locality, and index design.Examples include AES-128 CTR encryption, lexical tool routing, BVH ray tracing, Gaussian blur, and a Z-order range-count index.Source: §A.1 · pp. 17–18

Puzzle and Challenge

  • Puzzle-and-challenge tasks require constructing or improving compressors, access sequences, sorting networks, reversible circuits, and low-parameter models under correctness or performance constraints.The tasks include adaptive compression across varied sequence families, maximizing splay-tree rotations, reducing comparator or gate counts, and finding bit edits that degrade a frozen model.Source: §A.1 · pp. 18–19
  • Other puzzle tasks target instruction or cycle reduction through stack-machine code golf, latency-hiding ISA scheduling, and dependence-aware VLIW scheduling.These tasks require unrolling or interleaving operations while respecting instruction-set constraints, pipeline hazards, and operation dependencies.Source: §A.1 · p. 20

Model Development

  • Model-development tasks span data selection, fine-tuning, online serving, video prediction, OCR, and language-model pretraining, with constraints on compute, quality, retention, and generalization.Examples include selecting training data without access to the evaluation set, fitting FLUX.2 LoRA training within memory limits, and tuning serving-engine scheduling while keeping kernels and the model frozen.Source: §A.1 · pp. 20–21

CUDA kernel optimization

  • CUDA-kernel tasks address Huffman decoding, ICP correspondence, elliptic-curve multi-scalar multiplication, and number-theoretic transforms using GPU-specific cooperation, memory, and arithmetic strategies.The descriptions emphasize shared-memory lookup tables and warp cooperation, KD-tree pruning and reductions, Pippenger accumulation with Montgomery arithmetic, and tiled butterflies with specialized modular reduction.Source: §A.1 · pp. 21–22

A.2 Per-Task Scoring Anchors and Gates · B More on Experiments · B.1 More on Experimental Setups

The scoring appendix defines clipped task-level scoring anchors and feasibility gates, while the experimental setup appendix summarizes the organizations and API providers behind the evaluated models.

A.2 Per-Task Scoring Anchors and Gates

  • Both scoring schemes are clipped to [0, 1] and assign zero whenever an agent fails the correctness check.Per-task definitions specify the metric, optimization direction, baseline and reference anchors, and any feasibility gate for positive scores.Source: §A.2 · p. 22
  • Two Puzzle & Challenge parameter-count tasks use degenerate linear scoring equivalent to an implicit zero-parameter reference anchor.The listed reference values for smallest_game_player and safety_router document strong solutions rather than scoring anchors.Source: §A.2 · p. 22
  • resnet_bit_flip awards no positive reward unless the corrupted model’s accuracy falls below 12%.This task-specific feasibility gate applies before scoring.Source: §A.2 · p. 22
  • All system-optimization and CUDA tasks use log-stretch scoring on speed-up and require beating the baseline.Source: §A.2 · p. 22

B.1 More on Experimental Setups

  • Table 5 summarizes the developing organizations and API providers for every model evaluated on AUTOLAB.Source: §B.1 · p. 22

B.2 Detailed Experimental Results · C More on Analysis

The detailed results report per-task average and best scores across three trials for 11 frontier models, alongside task-specific scoring setups across four domains. The supplied passages describe the tables’ scope and metrics but do not provide cell values for model comparisons.

B.2 Detailed Experimental Results

  • Tables 6 and 7 report per-task Avg@3 and Best@3 results across three independent trials for each of 11 frontier models.The tables group tasks by sub-domain; Table 6 marks per-row best and runner-up, and Table 7 uses the same column order as Table 6.Source: §B.2 · p. 22, Table 6 · p. 25, Table 7 · p. 26
  • System-optimization (15 tasks) and CUDA (4 tasks) use log-stretch scoring with a must-beat-baseline improvement gate.mB and mR are empirically measured baseline and reference runtimes on standardized hardware and sandbox environments, in seconds for system tasks and milliseconds for CUDA tasks.Source: Table 2 · p. 23
  • Puzzle-and-challenge covers 10 tasks scored with anchored linear scoring unless otherwise noted, using mB and mR for baseline and reference targets.Source: Table 3 · p. 23
  • Model development covers 7 tasks, all using anchored linear scoring.Source: Table 4 · p. 24
  • For flux2_klein_lora, the baseline anchor is the measured no-LoRA quality score of approximately 0.49, while 0.0 is the OOM-crash floor.Source: Table 4 · p. 24
  • For llm_online_serving, the metric is a 50/50 throughput/latency composite that equals 1.0 at baseline by construction.Source: Table 4 · p. 24

C.1 Model Generations

Holding the harness at terminus-2, the comparison examines four old-to-new model-generation pairs. Three show modest gains, while Qwen 3.6 Plus regresses and newer models do not improve uniformly across categories.

C.1 Model Generations

  • Qwen 3.6 Plus drops 0.09 on Avg@3 and 0.12 on Best@3, retaining a 0.88 Model Development score while collapsing to near zero in three other categories.The categories with collapsed performance are CUDA, Puzzle & Challenge, and System Optimization.Source: §C.1 · p. 27
  • Provider-level generation gains are not uniform across sub-domains: MiniMax M2.7 improves overall but trails the median on CUDA, while MiMo v2.5 Pro gains on Model Development but loses ground on CUDA.Source: §C.1 · p. 27
  • Three of four generation pairs show modest gains, led by MiMo, followed by MiniMax and Kimi; Qwen 3.6 Plus regresses relative to 3.5 Plus.The comparison covers Qwen 3.5 Plus to 3.6 Plus, MiMo v2 Pro to v2.5 Pro, MiniMax M2.5 to M2.7, and Kimi K2.5 to K2.6 across all 36 tasks.Source: Figure 9 · p. 27, §C.1 · p. 27

C.2 Stability Analysis

Across-trial stability is distinct from raw capability, and reporting only mean performance masks meaningful variance. The analysis finds that single-trial and Best@3 evaluations can misrepresent high-variance models, supporting the use of Avg@3 or more trials.

C.2 Stability Analysis

  • Four complementary metrics quantify stability across three independent trials: mean per-task standard deviation ¯σ, mean per-task range ¯R, CV, and normalized dispersion ¯σ/Best@3.CV is ¯σ/Avg@3; all four metrics show consistent trends.Source: §C.2 · p. 27
  • ¯σ = 0.099 for claude-opus-4.6, which is both the highest-scoring and most stable model, with less than half the dispersion of other top-5 models.Stability and capability are correlated but not identical: three of the four weakest models have CV ≥0.43, while gemini-3.1-pro and grok-4-20 are more stable than peers with similar Avg@3 and kimi-k2.6 is among the noisiest models.Source: §C.2 · p. 27, Table 8 · p. 28
  • 0.28–0.34 is the mean across-trial range for models with CV ≥0.40, making single-trial rankings highly unreliable and supporting Avg@3 or more trials.These ranges are on a [0, 1] scale, so a single rollout can land roughly one-third of the full score range apart.Source: §C.2 · pp. 27–28
  • Pearson r = 0.84 links increasing ¯σ with a widening Best@3–Avg@3 gap, so Best@3 inflates noisy models’ apparent capability more.The paper therefore reports both metrics in its main leaderboard.Source: §C.2 · p. 28

Related papers

Loading 2606.05080v1…