PaperlayerView exact revision on arXiv

Source-linked AI summary

AutoLab: Can Frontier Models Solve Long-Horizon Auto Research and Engineering Tasks?

Zhangchen Xu, Junda Chen, Yue Huang, Dongfu Jiang, Jiefeng Chen, Hang Hua, Zijian Wu, Zheyuan Liu, Zexue He, Lichi Li, Shizhe Diao, Jiaxin Pei, Jinsung Yoon, Hao Zhang, Mengdi Wang, Radha Poovendran, Misha Sra, Alex Pentland, Zichen Chen

arXiv:2606.05080v1cs.AIcs.LG

TL;DR

Existing evaluations largely miss whether frontier models can sustain iterative optimization over long horizons. AutoLab addresses this with a 36-task benchmark and finds that persistent empirical iteration predicts success, with claude-opus-4.6 achieving an Avg@3 of 0.68 versus 0.50 for the next-best model.

  • Problem

    Existing evaluations largely overlook sustained, iterative optimization over long horizons, remaining limited in scale and generality.

  • Method

    AutoLab evaluates agents on 36 executable tasks across four domains, each requiring iterative improvement of a suboptimal baseline within a strict wall-clock budget.

  • Results

    Avg@3 was 0.68 for claude-opus-4.6 versus 0.50 for the next-best model, while persistence in benchmarking, editing, and incorporating feedback predicted stronger outcomes.

  • Takeaways & Limitations

    Persistence, time awareness, and empirical search are central priorities for future autonomous research agents.

  • Takeaways & Limitations

    AutoLab measures executable system and machine-learning engineering workflows rather than scientific discovery in its broadest sense.

Abstract

from arXiv · show

Scientific and engineering progress is fundamentally a long-horizon iterative process: proposing changes, running experiments, measuring outcomes, and continuously refining artifacts. Yet existing benchmarks for frontier models primarily evaluate either single-turn responses or short-horizon agent trajectories, failing to capture the challenges of sustained iterative improvement over extended time horizons. To address this gap, we introduce AutoLab, a new benchmark for ultra long-horizon closed-loop optimization. AutoLab consists of 36 realistic, expert-curated tasks spanning four diverse domains: system optimization, puzzle & challenge, model development, and CUDA kernel optimization. Each task begins with a correct but deliberately suboptimal baseline and challenges agents to improve it within a strict wall-clock budget. Evaluating 17 state-of-the-art models reveals the dominant predictor of success is not the quality of an agent's initial attempt, but its persistence in repeatedly benchmarking, editing, and incorporating empirical feedback. While claude-opus-4.6 exhibits strong long-horizon optimization capabilities, most frontier models, including several proprietary ones, either terminate prematurely or exhaust their budgets with minimal progress. These results underscore the importance of time awareness and persistent iteration in autonomous agents. We open-source the full benchmark, evaluation harness, and task artifacts, to accelerate research toward truly capable long-horizon agents.

1 Introduction

AUTOLAB addresses the gap in evaluating ultra long-horizon, closed-loop optimization by providing diverse tasks with deliberately suboptimal baselines under strict time limits. Its evaluation shows that persistence, time awareness, and empirical iteration matter more than initial solution quality, with claude-opus-4.6 leading across domains.

  • Motivation and gap: Frontier-agent tasks increasingly require iterative artifact inspection, experimentation, measurement, and refinement over hours rather than one-shot answers.Existing static benchmarks test one-shot coding, while newer agentic benchmarks generally cover only short interactive trajectories.
  • Benchmark: AUTOLAB addresses narrow existing benchmarks and model-specific harnesses by evaluating ultra long-horizon optimization across diverse domains with a standardized harness.The evaluation covers 17 state-of-the-art models, including four proprietary frontier models, under identical experimental conditions.
  • Benchmark: AUTOLAB provides 36 executable tasks across system optimization, puzzle & challenge, model development, and CUDA kernel optimization.Each task starts from a correct but deliberately suboptimal baseline and imposes a strict wall-clock budget for iterative improvement.
  • Results: 2,544 wall-clock hours and 8.60 billion tokens powered the main evaluation, where claude-opus-4.6 achieved an Avg@3 of 0.68 versus 0.50 for the next-best model.claude-opus-4.6 led every sub-domain, while some strong models terminated after minimal exploration or exhausted their budgets.
  • Results: Trajectory analysis, including 302 zero-score rollouts, identifies premature termination versus budget exhaustion as a major limitation and persistence as the dominant predictor of final performance.The findings emphasize time awareness, persistent iterative refinement, and empirical search for autonomous research agents.

2 The AUTOLAB Benchmark

AUTOLAB benchmarks frontier models on ultra long-horizon research and engineering optimization through interactive, budgeted experimentation. It combines realistic tasks, held-out verification, anchored continuous scoring, and diverse coverage across four domains.

  • Task Design: Each AUTOLAB task combines an instruction, containerized CPU or single-GPU environment, verifier, hidden reference solution, and wall-clock budget.The environment contains a correct but unoptimized baseline and a local evaluation script; the reference anchors scoring but is never exposed to agents.
  • Task Design: Agents iteratively edit, execute, profile, evaluate, inspect outputs, and refine implementations before final held-out verification within the allotted budget.Budgets range from 2 hours for the smallest puzzle tasks to 12 hours for end-to-end LLM development tasks.
  • Baselines and References: Baselines are correct but suboptimal, while references must deliver substantial gains, creating measurable headroom for agent improvement.Reference solutions typically improve system-optimization metrics by at least an order of magnitude and provide a clear statistical gain on model-development tasks.
  • Scoring: AUTOLAB uses baseline- and reference-anchored continuous scores normalized to [0, 1], rewarding partial progress across heterogeneous metrics rather than only pass/fail outcomes.The log-stretch scheme gives s = 0 at the baseline and s = 0.5 at the reference, while linear scoring gives s = 0 at the baseline and s = 1.0 at the reference.
  • Task Collection: The final benchmark contains 36 tasks spanning Model Development (7), System Optimization (15), Puzzle & Challenge (10), and CUDA (4).Tasks were contributed by senior researchers and engineers based on real problems, prioritizing realism and diversity across research and engineering workflows.

3 Benchmark Results

claude-opus-4.6 leads AUTOLAB by a substantial margin, with distinct strengths across categories and especially strong CUDA performance. Other models often underperform because of premature termination or excessive reasoning that limits iterative benchmarking and editing.

  • Overall Performance: claude-opus-4.6 leads with Avg@3 0.68 and Dominance 0.93, while second-place gemini-3.1-pro records Avg@3 0.50.The gap indicates clear separation among frontier models on long-horizon iterative improvement tasks.
  • Overall Performance: Among open-weight models, kimi-k2.6, mimo-v2.5-pro, and glm-5 form a tight cluster at 0.46, 0.45, and 0.43, respectively.mimo-v2.5 and deepseek-v4-flash remain competitive despite having under 400B parameters; deepseek-v4-flash scores 0.37 versus deepseek-v4-pro’s 0.38.
  • Performance by Category: claude-opus-4.6 leads all four categories, with its largest advantage on CUDA tasks, where most other models score near zero.gemini-3.1-pro performs best on puzzle tasks but lags significantly on CUDA and model development tasks.
  • Case Study: Flash Attention Optimization: On flash_attention, claude-opus-4.6 reduces runtime from approximately 750 ms to 18 ms through 44 feedback-driven iterations over roughly 40 minutes.All models begin from the same baseline runtime, but their optimization trajectories diverge sharply.
  • Case Study: Flash Attention Optimization: Reasoning-heavy models spend most of their budget thinking rather than executing commands, delaying their first benchmark and limiting edit-and-rerun iterations.deepseek-v4-pro times out before fully exploiting promising directions, so its final submission is not its best trajectory result.

4 Analysis

AutoLab performance rises with agent steps and runtime, but higher scores generally cost more, while some open-weight models achieve competitive performance at lower inference cost. Zero-score rollouts reveal failures in time awareness, capability, instruction following, and harness sensitivity, showing that harness design materially affects both performance and spending.

  • Resource utilization: Higher average overall scores correlate positively with agent steps and wall-clock runtime, while inference costs generally increase with performance.claude-opus-4.6 is a prominent outlier requiring substantially more steps; the analysis also identifies competitive lower-cost performance from deepseek-v4-flash and mimo-v2.5-pro.
  • Failure analysis: 302 zero-score rollouts across 11 models were manually classified into four mutually exclusive failure modes: timeout/context exhaustion, capability gaps, instruction violations, and other upstream issues.Capability gaps include incorrect or sub-threshold outputs, early give-ups, and missing required files; instruction violations receive zero regardless of solution correctness.
  • Failure analysis: Models show poor time awareness, either terminating too early or exhausting the budget without submitting, with deepseek-v4-pro, hunyuan-3-preview, and qwen-3.6-plus frequently timeout-dominated.Open-weight models also exhibit unusually long reasoning chains: all kimi-k2.6 toy_isa_opt trials timed out after 2–11 steps, while 9 of 12 deepseek-v4-pro CUDA trials submitted fewer than 10 actions.
  • Failure analysis: Instruction violations remain concentrated in gemini-3.1-pro (5 cases) and glm-5 (4 cases), with ntt_butterfly_cuda accounting for half of all violations.The finding indicates that robust instruction following remains challenging even for capable closed-source models.
  • Harness effects: Harness choice shifts mean scores by up to ∆= 0.43 and produces non-transitive model rankings, demonstrating that harness-induced variance can rival model-induced variance.For kimi-k2.6, the mean score changes from 0.21 under pi-mono to 0.64 under mini-swe-agent*.
  • Harness effects: Per-trial inference cost varies by more than 5× across harnesses for the same model, as kimi-k2.6 costs $0.40 under pi-mono versus $2.05 under mini-swe-agent*.Different harnesses encourage substantially different iteration efforts before termination, making harness selection also a spending choice.
  • Harness effects: The analyses identify harness design as a research direction because tuned harnesses could narrow performance gaps by providing iteration headroom or tighter patch loops without changing base models.The proposed tuning differs by model need: more iteration headroom for smaller models and tighter, high-quality patch loops for stronger instruction-followers.

5 Related Work

Prior benchmarks largely evaluate single-turn or short-horizon performance, while newer work studies multi-hour optimization, research workflows, and closed-loop training environments. AutoLab is positioned within this shift toward iterative agent evaluation.

  • Static and short-horizon agent benchmarks: Most public frontier-model benchmarks remain single-turn or terminal-state, scoring one-shot generation or one-edit-one-submission despite harder variants.Examples include HumanEval, LiveCodeBench, BigCodeBench, SWE-bench, MMLU-Pro, and LiveBench.
  • Long-horizon optimization and research-agent benchmarks: A growing benchmark family evaluates agents on realistic multi-hour ML and engineering workflows across research pipelines, systems, and kernels.Named examples include MLE-Bench, RE-Bench, PaperBench, PostTrainBench, AIRS-Bench, KernelBench, FrontierCS, and Frontier-Eng.
  • Closed-loop agent frameworks and training environments: Closed-loop frameworks support iterative software engineering and open-ended scientific work, while Gym-style environments enable iterative training and evaluation.Examples include SWE-agent, OpenHands, Aider, The AI Scientist, SWE-Gym, R2E-Gym, and MLGym.

6 Conclusion

AUTOLAB evaluates frontier models on ultra long-horizon research and engineering tasks requiring sustained iteration over hours rather than minutes. Its results show that persistence in evaluating, editing, and iterating predicts success more strongly than raw capability alone, with claude-opus-4.6 leading while most other frontier models terminate prematurely or exhaust their budgets without submitting.

  • Benchmark contribution: AUTOLAB evaluates frontier models on ultra long-horizon research and engineering tasks requiring sustained iteration over hours rather than minutes.The benchmark enforces ultra long-horizon tasks, continuous calibrated scoring, and strong anti-hacking safeguards.
  • Main finding: Persistent evaluation, editing, and iteration over extended horizons are the dominant predictors of success, while raw capability alone is insufficient.This finding follows from AUTOLAB’s emphasis on sustained iteration and empirical progress across long trajectories.
  • Model comparison: claude-opus-4.6 achieves a commanding lead through long, steady optimization trajectories.The result demonstrates the value of persistence on ultra long-horizon tasks.
  • Model comparison: Most other frontier models, including several proprietary models, terminate prematurely or exhaust their budgets without submitting.Their failure modes contrast with the long, steady optimization trajectories associated with claude-opus-4.6.

Limitations and Broader Impact · A Task Specifications

AutoLab benchmarks measurable auto-research in executable system and machine-learning engineering workflows, not scientific discovery broadly. Its long-horizon evaluation jointly considers trajectories, resource consumption, and final performance because execution depends on multi-hour runs, APIs, GPUs, and the surrounding stack.

  • Limitations and Broader Impact: AutoLab targets executable system and machine-learning engineering workflows.It should be understood as a benchmark for measurable auto-research rather than broad scientific discovery.
  • Limitations and Broader Impact: The benchmark does not measure scientific discovery in its broadest sense.Its scope is limited to measurable auto-research in executable engineering workflows.
  • Limitations and Broader Impact: Long-horizon evaluation inherently depends on multi-hour execution.This dependence distinguishes the benchmark from evaluations that can be completed in shorter interactions.
  • Limitations and Broader Impact: API interactions are part of the execution requirements for long-horizon evaluation.The surrounding execution stack also affects how these evaluations operate.
  • Limitations and Broader Impact: GPU workloads are another dependency of the evaluation setting.Together with execution time and APIs, they make the surrounding stack relevant to benchmark interpretation.
  • Limitations and Broader Impact: AutoLab reports trajectory analysis, resource consumption, and final performance jointly.The benchmark therefore treats these dimensions as connected aspects of long-horizon evaluation rather than isolating a single score.

A.1 Task Descriptions

AutoLab lists 36 expert-curated tasks across four categories, specifying each task’s implementation language, difficulty tier, and optimization challenge. The tasks span systems, puzzles, model development, and CUDA kernel optimization, with examples requiring algorithmic, hardware-aware, or constraint-driven improvements.

  • Task Descriptions: 36 tasks are grouped into four domains, with each entry identifying its implementation language, difficulty tier, and short optimization description.Tier 1 denotes textbook-classic optimization with well-known techniques, while tier 2 denotes bespoke, domain-specific, or research-style tasks.
  • System Optimization: System tasks impose concrete workloads and implementation constraints, such as encrypting 256 MiB with AES-NI, routing under MRR@10 ≥0.82 and Recall@10 ≥0.94, or processing 638×638 rays.Several tasks require single-threaded execution, standard-library-only implementations, SIMD, pipelining, inverted indexes, or cache-friendly data structures.
  • Puzzle and Challenge: Puzzle and challenge tasks use hidden sequence families, adversarial access patterns, combinatorial verification, reversible circuits, and constrained bit edits to define optimization objectives.Examples include adaptive compression, splay-tree rotation maximization, sorting-network synthesis, Fredkin-gate reduction, and MiniResNet bit-flip attacks.
  • Puzzle and Challenge: Several puzzle tasks provide explicit baselines and references, including 5.0 bpb versus 3.8 bpb for compression, 80 versus 60 sorting-network comparators, and 128 versus 88 reversible gates.The MiniResNet task targets fewer than 12% CIFAR-10 accuracy using a smallest-set bit-edit objective, with 95-flip baseline and 40-flip reference values.

A.2 Per-Task Scoring Anchors and Gates · B More on Experiments · B.1 More on Experimental Setups

AutoLab scores task improvements with clipped anchored-linear or log-stretch schemes, while correctness and task-specific feasibility gates determine whether positive scores are possible. The experimental setup also documents the organizations and API providers associated with evaluated models.

  • A.2 Per-Task Scoring Anchors and Gates: Both scoring schemes are clipped to [0, 1] and saturate to 0 when the agent fails the correctness check.The schemes are anchored linear and log-stretch.
  • A.2 Per-Task Scoring Anchors and Gates: Each task specifies a metric, optimization direction, baseline anchor mB, reference anchor mR, and any feasibility gate required for positive scoring.Directions are marked lower-is-better (↓) or higher-is-better (↑).
  • A.2 Per-Task Scoring Anchors and Gates: Two Puzzle & Challenge parameter-count tasks, smallest_game_player and safety_router, use the degenerate linear form s(x) = clip.The supplied passage truncates the displayed formula after “clip.”
  • A.2 Per-Task Scoring Anchors and Gates: For smallest_game_player and safety_router, the degenerate form is equivalent to anchored linear scoring with an implicit reference anchor of mR = 0.Here, mR = 0 denotes zero parameters.
  • A.2 Per-Task Scoring Anchors and Gates: The mR values listed for smallest_game_player and safety_router represent documented strong solutions rather than scoring anchors.This distinction applies to the two parameter-count tasks described in the preceding passage.
  • A.2 Per-Task Scoring Anchors and Gates: resnet_bit_flip awards positive reward only when the corrupted model’s accuracy falls below 12%.This is an additional feasibility gate for that task.
  • A.2 Per-Task Scoring Anchors and Gates: All system-optimization and CUDA tasks use log-stretch scoring on a spee.The supplied passage truncates the scoring description after “spee.”
  • B.1 More on Experimental Setups: Table 5 summarizes the developing organizations and API providers for all models evaluated on AUTOLAB.This information appears in the experimental setup material.

B.2 Detailed Experimental Results · C More on Analysis

The detailed results report per-task average and best scores across frontier models, organized by sub-domain and evaluated with task-specific scoring rules. Tables define the benchmark task coverage, scoring anchors, runtime references, and Avg@3 and Best@3 reporting conventions.

  • B.2 Detailed Experimental Results: 11 frontier models are compared using per-task average and best scores across three independent trials, grouped by sub-domain.The tables’ diverging green-to-pink shading is centered at 0.5 to show individual task strengths and weaknesses.
  • B.2 Detailed Experimental Results: 36 tasks span system optimization, CUDA, puzzle-and-challenge, and model-development settings with domain-specific scoring rules and improvement gates.System optimization covers 15 tasks and CUDA covers 4, using log-stretch scoring with a “must beat baseline” gate; puzzle-and-challenge covers 10 tasks, while model development covers 7.
  • B.2 Detailed Experimental Results: mB and mR are empirically measured baseline and reference runtimes on standardized hardware and sandbox environments.For system-optimization tasks, runtimes are measured in seconds; for CUDA tasks, they are measured in milliseconds.
  • B.2 Detailed Experimental Results: Puzzle-and-challenge tasks use anchored linear scoring unless otherwise noted, with mB and mR denoting baseline and reference targets.The task set contains 10 puzzle-and-challenge tasks.
  • B.2 Detailed Experimental Results: Model-development tasks use anchored linear scoring, with task-specific baseline conventions for flux2_klein_lora and llm_online_serving.For flux2_klein_lora, the no-LoRA anchor is approximately 0.49 while 0.0 is the OOM-crash floor; llm_online_serving uses a 50/50 throughput/latency composite equal to 1.0 at baseline.
  • B.2 Detailed Experimental Results: AUTOLAB’s evaluated models are listed with their developing organization and API provider.Table 5 provides the model, organization, and provider information.
  • B.2 Detailed Experimental Results: Avg@3 results report per-task averages, with the per-row best bolded and the runner-up underlined.Table 6 presents these per-task Avg@3 results.
  • B.2 Detailed Experimental Results: Best@3 results report per-task best scores using the same column order as the Avg@3 table.Table 7 bolds the per-row best and underlines the runner-up, with columns ordered by overall Avg@3.

C.1 Model Generations

Holding the harness at terminus-2, the study compares four within-provider model-generation pairs across all 36 tasks. Three newer variants improve modestly, while Qwen 3.6 Plus uniquely regresses, especially outside Model Development.

  • Generation comparisons: Four old-to-new pairs are compared at terminus-2: Qwen 3.5 Plus→3.6 Plus, MiMo v2 Pro→v2.5 Pro, MiniMax M2.5→M2.7, and Kimi K2.5→K2.6.Figure 9 reports per-provider generation deltas across all 36 AUTOLAB tasks.
  • Generation comparisons: Three of four pairs show modest gains, with MiMo improving most, followed by MiniMax and Kimi.The newer variants outperform their older counterparts for these three providers.
  • Generation comparisons: 0.09 and 0.12 are Qwen 3.6 Plus’s regressions on Avg@3 and Best@3, respectively, making it the only generation to decline.Its Model Development score remains 0.88, while CUDA, Puzzle & Challenge, and System Optimization collapse to or near zero.

C.2 Stability Analysis

Across-trial stability captures reliability beyond average capability, using four complementary dispersion metrics computed from three independent trials. Results show that high variance makes single-trial and Best@3 evaluations unreliable, while claude-opus-4.6 is both highly capable and stable.

  • C.2 Stability Analysis: Stability is distinct from raw capability because identical Avg@3 scores can conceal substantially different across-trial reliability.The analysis illustrates this with one model scoring 0.85, 0.20, and 0.20 versus another consistently scoring 0.65.
  • C.2 Stability Analysis: Four metrics quantify stability across three independent trials: mean per-task standard deviation ¯σ, mean per-task range ¯R, coefficient of variation CV, and normalized dispersion ¯σ/Best@3.CV is defined as ¯σ/Avg@3; lower values indicate higher stability.
  • C.2 Stability Analysis: claude-opus-4.6 is both the highest-scoring and most stable model, with ¯σ = 0.099, while three of four weakest models have CV ≥0.43.Stability and capability are correlated but not identical; gemini-3.1-pro and grok-4-20 are notably more stable than similarly scoring peers.
  • C.2 Stability Analysis: For models with CV ≥0.40, the mean across-trial range reaches 0.28–0.34 on a [0, 1] scale, making single-shot rankings highly unreliable.The analysis recommends Avg@3, or more trials, as the primary metric for these high-variance models.
  • C.2 Stability Analysis: The gap between Best@3 and Avg@3 widens with increasing ¯σ, with Pearson r = 0.84, so Best@3 over-credits noisy models.Accordingly, the main leaderboard reports both metrics.
Loading 2606.05080v1…