PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Final Scores: A Systematic Evaluation of Agents for Long-Horizon AI Research and Development

Yiwei Li, Wanli Yang, Hexiang Tan, Xiangzhou Huang, Zhengyu Chen, Ziran Li, Borun Chen, Shanglin Lei, Huaisheng Zhu, Hao Tian, Fei Sun, Xunliang Cai, Jingang Wang

arXiv:2608.13417v1cs.AI

TL;DR

Final scores alone provide limited evidence about how long-horizon agents gain and sustain progress or reuse experience. The paper evaluates seven frontier models on 36 tasks with rule-based process metrics and controlled experience comparisons. It finds partial research-loop automation: agents can produce practical solutions, but reliability varies, novelty is rare, experience transfer is unstable, and harnesses mainly improve reliable realization.

  • Problem

    Existing evaluations primarily use a single final score, which misses where progress is gained or lost and whether accumulated experience improves later decisions.

  • Method

    The study evaluates seven frontier models on 36 long-horizon tasks using rule-based metrics for Solution Framing, Execution, and Feedback Control plus controlled intra- and inter-task experience comparisons.

  • Results

    Agents achieve partial research-loop automation: they can identify practical approaches and implement them, but performance varies across runs, process bottlenecks differ, and genuine innovation remains rare.

  • Takeaways & Limitations

    Observed capability depends on model behavior, accumulated experience, and harness design, motivating targeted training, inference-time selection, memory, and system-design improvements.

  • Takeaways & Limitations

    The process metrics are reproducible proxies that cannot capture unrealized idea quality or latent reasoning, and Feedback Control is weakly evidenced without regressions.

Abstract

from arXiv · show

Autonomous agents are increasingly capable of improving models, systems, and other technical artifacts through long-horizon experimentation. To understand the current state of this capability, however, evaluation must go beyond final scores, which neither reveal where progress is gained or lost nor indicate whether accumulated experience improves later decisions. We therefore present a systematic evaluation of seven frontier models on 36 long-horizon tasks based on a new framework that uses rule-based metrics to characterize within-run behavior through Solution Framing, Execution, and Feedback Control and controlled comparisons to assess experience reuse within and across tasks. The results show that current agents operate more like engineering optimizers than fully autonomous researchers: they can formulate and implement practical solutions, but their performance varies substantially across runs, their strongest solutions mainly adapt or combine established techniques, and genuine methodological novelty remains rare. Detailed analysis reveals that observed performance is shaped by multiple factors, including distinct process bottlenecks behind similar final outcomes, experience reuse that can help or mislead subsequent decisions, and harness designs that affect performance stability. These findings suggest concrete directions for improving model training, inference-time strategies, experience management, and harness design.

1 Introduction

Long-horizon automated research is becoming measurable through agents that iteratively propose, test, and refine technical artifacts, but final-score evaluation misses how progress is produced or sustained. These gaps motivate finer-grained evaluation of research behavior and experience use.

  • Frontier language models increasingly conduct long-horizon automated research by proposing changes, running experiments, interpreting feedback, and refining executable artifacts.
  • Single final scores fail to reveal where progress is gained or lost within an agent’s experimentation-and-refinement loop.
  • Final scores also cannot show whether agents translate ideas into reliable implementations or use feedback to preserve progress and recover from failures.

C3 Feedback Control

The evaluation treats automated research as a process of framing solutions, executing changes, and controlling feedback, while also testing how experience affects later decisions. Results indicate that agents can produce practical solutions, but reliability, bottlenecks, novelty, experience reuse, and harness design materially shape outcomes.

  • C3 Feedback Control: The process view separates Solution Framing, Execution, and Feedback Control, while the experience view tests accumulated experience in intra- and inter-task settings.
  • C3 Feedback Control: Rule-based metrics diagnose these capabilities from verifier outcomes and recorded trajectory signals rather than LLM judgments.
  • C3 Feedback Control: Current agents resemble engineering optimizers more than fully autonomous researchers: they formulate practical directions and implement working solutions, but success varies across runs.
  • C3 Feedback Control: The gap between strongest and weakest models is 0.237 on avg@3 but 0.122 on best@3, indicating substantially different consistency across repeated runs.
  • C3 Feedback Control: Similar final scores can conceal different process bottlenecks, while only three of 252 best-seed solutions qualify as novel approaches under the review protocol.
  • C3 Feedback Control: Experience can improve or degrade later performance, whereas native harnesses mainly improve run-to-run stability without materially changing best@3 or model ordering.
  • C3 Feedback Control: Observed performance reflects interactions among the model, accumulated experience, and the surrounding system rather than a single static capability.

2 Evaluation Setting and Outcome-Level Landscape

The study compares seven frontier models on 36 expert-curated tasks using a shared harness, repeated rollouts, outcome metrics, and resource measurements. Results show a clear overall hierarchy, substantial category-specific differences, and a trade-off between performance and inference cost.

  • 2 Evaluation Setting and Outcome-Level Landscape: The evaluation covers 36 expert-curated tasks across Model Development, System Optimization, Puzzle & Challenge, and CUDA workload families.The four families contain 7, 15, 10, and 4 tasks, respectively.
  • 2 Evaluation Setting and Outcome-Level Landscape: Seven frontier models are compared primarily with Claude Code as a shared harness, holding the tool interface and iteration policy fixed.
  • 2 Evaluation Setting and Outcome-Level Landscape: Each model-task pair receives three independent rollouts, with avg@3 and best@3 reporting typical and best-observed performance.The full evaluation comprises 756 rollouts, with task budgets of 2–12 hours.
  • 2 Evaluation Setting and Outcome-Level Landscape: Opus-4.7 ranks first overall at 0.739 avg@3 and 0.790 best@3, while GPT-5.5, GLM-5.2, and Gemini-3.1-Pro form a compact second tier.
  • 2 Evaluation Setting and Outcome-Level Landscape: The highest-to-lowest model gap is 0.237 on avg@3 versus 0.122 on best@3, showing that lower-ranked models can reach competitive solutions less consistently.
  • 2 Evaluation Setting and Outcome-Level Landscape: CUDA is the most separating and lower-scoring category, with gaps of 0.403 on avg@3 and 0.414 on best@3, whereas Puzzle & Challenge has the smallest gaps.
  • 2 Evaluation Setting and Outcome-Level Landscape: Opus-4.7 achieves 0.790 best@3 at $89.9 per task, while GPT-5.5 and GLM-5.2 provide close alternatives at 0.772 and 0.757 for $16.5 and $33.0 per task.

3 Process-Level Evaluation

The paper evaluates long-horizon research as a process, decomposing it into Solution Framing, Execution, and Feedback Control with deterministic trajectory-based metrics. Results show that similar outcomes can arise from different bottlenecks across models and task categories.

  • Process Evaluation Design: The evaluation decomposes automated R&D into Solution Framing, Execution, and Feedback Control, measured deterministically from verifier outcomes and trajectory signals.These dimensions assess pursued directions, executable delivery, and preservation or recovery after regressions.
  • Process Capability Results: Execution is broadly reliable, while Solution Framing and Feedback Control vary more across models.Opus-4.7 leads outcome at 0.739, C1 at 0.612, and C2 at 0.967; C1 ranges from 0.473 to 0.612, while C3 ranges from 0.772 to 0.928.
  • Process Capability Results: Similar outcomes can conceal sharply different process profiles: GPT-5.5 and Gemini-3.1-Pro score 0.663 and 0.652 overall, but differ in C2 and C3.GPT-5.5 reaches 0.958 on C2 and 0.858 on C3, whereas Gemini-3.1-Pro reaches 0.889 on C2 and 0.920 on C3.
  • Process Capability Results: Task categories expose distinct bottlenecks: CUDA is weakest in C1 and C2, while Model Development is strongest in C2 but weakest in C3.CUDA records C1 at 0.370 and C2 at 0.850; Model Development records C2 at 0.985 and C3 at 0.743.
  • Behavioral Diagnostics: Opus reaches the highest observed score at 0.757, while Gemini-3.1-Pro combines 83.7% early capture with only 16.5% later headroom capture.These diagnostics distinguish absolute solution quality, initial direction strength, and subsequent progress.
  • Behavioral Diagnostics: Kimi-K2.7-Code and LongCat-2.0 have nearly identical C2 scores of 0.880 and 0.888 despite different build activity and error rates.LongCat performs 4.66 builds per round with errors in 17.1% of rounds, versus 2.70 builds and 8.5% errors for Kimi.

4 Learning from Experience

The paper tests whether agents can reuse experience within the same task and transfer lessons across tasks using controlled comparisons. Experience usually helps, but its benefits vary widely and can sometimes mislead agents or alter model rankings.

  • Intra-Task Self-Improvement: Intra-task self-improvement compares the next solution with and without accumulated experience after branching from the same intermediate trajectory.The design re-initializes the agent and erases its history and persisted notes in the without-experience condition.
  • Inter-Task Experience Reuse: Inter-task reuse compares baseline and lesson-augmented target runs while holding the model, harness, environment, resource limits, and workspace isolation fixed.Each model extracts lessons from a solved source task and applies only those lessons to a held-out target task.
  • Intra-Task Experience Reuse: Intra-task experience generally improves the next commit across models, with Kimi-K2.7-Code the sole exception at −0.0127.Kimi still benefits on more tasks than it is harmed, 17 versus 10, across the 32 retained tasks.
  • Intra-Task Experience Reuse: Models differ widely in reliance on intra-task experience: Opus-4.7 gains +0.0362, while LongCat-2.0 gains +0.1454.The paper links the smaller Opus gain to strong Solution Framing and describes LongCat as depending more on accumulated exploration.
  • Intra-Task Experience Reuse: Retained experience can improve commits by avoiding dead ends, reusing tuned configurations, and carrying forward hard-won implementations, but it can also backfire.The paper reports positive and negative trajectory case studies for these effects.
  • Inter-Task Experience Reuse: Initial performance does not reliably predict improvement through experience: DeepSeek-V4-Pro gains +0.093 on avg@3, while Gemini-3.1-Pro declines by −0.017.GPT-5.5 gains more on avg@3 than best@3, whereas GLM-5.2 and Opus-4.7 gain more on best@3.
  • Inter-Task Experience Reuse: Explicitly extracted, self-generated lessons outperform raw source-workspace access for the tested models, while overall transfer gains remain unstable.The results motivate improving lesson extraction, selection, adaptation, application, and revision across longer workflows.

5 The Role of the Agent Harness

Harness choice has little effect on best observed scores but substantially affects run-to-run stability. Harness evolution can improve performance on related tasks, while specialized controls help preserve strong states during long research loops.

  • Harness comparison: Best@3 differs by at most 0.035 across harnesses, while avg@3 improves by up to 0.055 for Kimi-K2.7-Code with the native harness.Model rankings remain preserved, and stability gains are especially pronounced for Kimi.
  • Auto Harness: A four-round evolved harness raises avg@3 by +0.12 on seed tasks, with smaller gains on same-model and cross-model System Optimization tasks.The gains are +0.06 on remaining same-model tasks and +0.03 for GPT-5.5.
  • Auto Harness: The evolved harness shows no clear gain on unrelated task families, indicating limited transfer from narrowly selected System Optimization seeds.The authors suggest broader and more diverse seed tasks would likely be needed.
  • Harness functions: General-purpose harnesses support long-horizon work through failure recovery, context management, and explicit research-loop management.Across 756 Claude Code trajectories, TaskCreate and TaskUpdate were invoked 2,711 and 4,632 times, respectively.
  • Auto Harness: Auto Harness controls protect verified improvements by isolating risky changes, restoring unsuccessful experiments, and requiring the agent to restore its best state before finishing.These controls strengthen version management within the research loop and help prevent late changes from replacing stronger results.

6 Solution Novelty Analysis

Agents most often improve artifacts by stacking established techniques rather than developing genuinely novel methods. Validated novelty is rare, and departures from standard solutions more often exploit evaluation shortcuts than produce new approaches.

  • Motivation: Because high task scores do not reveal whether agents discovered new ideas or assembled existing techniques, novelty analysis adds a methodological-quality dimension to outcome evaluation.This concern motivates examining solution composition and manual validation alongside performance.
  • Novelty prevalence: Composition-stacking is the largest solution category for every model, accounting for 111 of 252 solutions (44.0%).This category layers multiple established algorithmic and engineering optimizations onto a standard approach.
  • Novelty prevalence: Only three solutions (1.2%) retain the novel label after manual review, compared with 16 solutions (6.3%) exploiting evaluation-specific shortcuts.Shortcut solutions are more than five times as common as validated novel solutions, with GPT-5.5 accounting for eight shortcut cases.
  • Novelty characteristics: The three validated novel approaches come from GLM-5.2, Kimi-K2.7-Code, and LongCat-2.0 rather than concentrating in the highest-performing models.The results associate novelty with task-specific reframing rather than new technical primitives.

7 Discussion

The discussion attributes agent performance to distinct weaknesses in training, inference, memory, harness control, and evaluation objectives. It therefore argues for targeted interventions rather than a single optimization strategy.

  • Implications: Different failure patterns require corresponding changes to model training, inference-time strategies, long-horizon system design, or evaluation objectives.The paper explicitly rejects a single strategy for addressing current-agent limitations.
  • Training: Solution Framing and Feedback Control vary more across models than Execution, suggesting greater scope for model-specific improvements in direction selection and feedback use.Generic code-execution training alone is unlikely to be the primary field-wide opportunity because Execution is already strong and clustered.
  • Inference-time strategies: Diverse rollouts and verifier-guided compute allocation could improve reliability when models can reach competitive solutions but fail to reproduce them consistently.The discussion contrasts average and best-run performance as evidence for targeting consistency at inference time.
  • Experience management: Memory systems must selectively retrieve, validate, revise, and remove experience because accumulated information can preserve discoveries, propagate misleading conclusions, or anchor local optima.Simply storing more context is insufficient for reliable experience reuse.
  • Harness design: Native harnesses improve run-to-run stability without substantially raising the performance ceiling through error recovery, task management, and best-state protection.Automated harness optimization provides additional headroom by targeting distinct research bottlenecks.
  • Objectives and benchmarks: When rewards measure task performance but not methodological quality, stronger optimization may reinforce shortcut-seeking rather than improve research quality.The paper calls for tasks and feedback that reward novelty, validity, and generality as well as task performance.

8 Related Work

Related work spans benchmarks for executable research, process-level evaluation, experience reuse, and agent-harness design. Together, these lines of research motivate evaluating both terminal outcomes and the mechanisms shaping long-horizon behavior.

  • Benchmarks for autonomous research agents: Autonomous research benchmarks evaluate agents that modify code, run experiments, and improve machine-learning systems under realistic constraints.Examples include MLAgentBench, MLE-bench, RE-Bench, PostTrainBench, MLS-Bench, and AutoLab.
  • Process-level evaluation: Process-level evaluation measures intermediate behavior such as progress toward subgoals, tool correctness and ordering, exploration reach, and execution accuracy.These studies move beyond evaluating only terminal success metrics.
  • Experience reuse and self-improving agents: Experience-reuse research studies reflections, reusable trajectory insights, and sequential task streams as mechanisms for improving later agent decisions.Representative systems include Reflexion, ExpeL, LifelongAgentBench, and SEA-Eval.
  • Impact of agent harnesses: Agent-harness research shows that tool use, context construction, execution, and feedback scaffolds can materially affect performance alongside the underlying model.SWE-agent, the Holistic Agent Leaderboard, and Harness-Bench examine these model–scaffold interactions.

9 Conclusion

The evaluation places current long-horizon research agents at partial research-loop automation: they can produce practical technical solutions, but reliable reproduction and genuine innovation remain limited.

  • Agents can identify practical approaches, implement them, and sometimes reach competitive solutions within long-horizon research loops.
  • Strong behavior is not reproduced consistently across runs, while genuine innovation remains rare.
  • Process bottlenecks vary across models and workloads, and experience transfer remains unstable.
  • Harnesses mainly improve the reliable realization of existing capability.
  • These distinctions support targeted improvements in training, inference-time selection, selective memory, harness design, and stronger tasks or verifiers.

Limitations

The paper’s limitations concern the scope of its process metrics, controlled experience interventions, benchmark and harness settings, and resource-cost assumptions.

  • C1–C3 are reproducible trajectory-based proxies, not exhaustive measures of general research ability.They cannot capture unrealized idea quality or latent reasoning, and C3 may overstate feedback control when few regressions occur.
  • Self-improvement estimates depend on the chosen erasure points, source–target pairs, and representation of transferred experience.Other memory systems or task sequences may yield different estimates of experience-driven improvement.
  • Conclusions depend on AutoLab’s task distribution, budgets, verifiers, execution environment, and tested harness configurations.Other research domains or system configurations may change absolute scores and some relative rankings.
  • Estimated inference cost depends on provider pricing, token accounting, serving configurations, budgets, and infrastructure conditions.Cost results are most reliable for comparisons under the controlled setup, not as universal deployment prices.

A Outcome Performance by Category

Performance varies substantially by task category and by whether results measure average or best repeated-run performance. Resource use and metric construction further shape how these outcomes should be compared.

  • Model Development: Opus leads Model Development on avg@3 (0.785) and best@3 (0.833), while Kimi has the largest avg–best gap (0.240).Gemini’s best@3 is 0.819 and Kimi’s is 0.806, both above GLM’s 0.749 and GPT’s 0.738.
  • System Optimization: Opus leads System Optimization on avg@3 (0.675), while Opus, GPT, and GLM have nearly identical best@3 scores of 0.705, 0.703, and 0.700.
  • Puzzle & Challenge: Puzzle & Challenge has the smallest separation, with highest-to-lowest gaps of 0.150 on avg@3 and 0.074 on best@3.GLM leads both metrics, indicating relatively widespread strong puzzle-solving performance among the evaluated models.
  • CUDA: CUDA has the largest separation: 0.403 on avg@3 and 0.414 on best@3, with Opus leading avg@3 (0.617) and GPT leading best@3 (0.722).The contrast distinguishes consistent strength from occasional peak performance in low-level GPU optimization.
  • Resource use: Gemini-3.1-Pro and GPT-5.5 use the least wall-clock time overall at 66 and 70 minutes per task, while GPT uses 3.2 million tokens and GLM uses 29.4 million.Model Development is more time-consuming for every model than the other workload families.
  • Resource use: Figure 12 compares best@3 with total estimated cost, mean wall-clock time, and mean interaction steps across seven models.Figures 13 and 14 report category-level and overall wall-clock time and token consumption, averaging three independent rollouts per model–task pair.
  • Process metrics: C2 evaluates executable, correct delivery while discounting successful checkpoints for code-related build failures and excluding environment failures.Failed delivery receives no credit; optimization-only tasks require successful execution, while correctness-gated tasks additionally require correctness=True.
  • Process metrics: C3 measures preservation of successful discoveries and recovery from regressions, including bounded penalties for additional self-evaluated attempts.

H.1 Intra-Task Analysis: When Experience Helps or Hurts

Retained experience can improve or degrade subsequent solutions, depending on whether it preserves useful discoveries or anchors the agent to a bad state. Across tasks, transfer also depends on how experience is represented, which model produced it, and the harness used during execution.

  • Intra-Task Experience: Retained experience helps by avoiding known dead ends, preserving tuned configurations, and reusing hard-won implementations.These benefits save the agent from rediscovering approaches, hyperparameters, or low-level code within the remaining budget.
  • Intra-Task Experience: Memory can hurt when it preserves a premature conclusion or anchors the agent to a local optimum.On msm_pippenger, retaining an earlier negative judgment kept DeepSeek on a weaker approach, while erasing memory enabled reconsideration.
  • Inter-Task Experience: Extracted lessons outperform raw workspaces across all three models, averaging +0.035 versus −0.007 under avg@3 and +0.042 versus −0.009 under best@3.The results indicate that lesson extraction improves transfer by filtering noise and surfacing transferable knowledge.
  • Inter-Task Experience: Self-generated lessons outperform cross-model lessons, showing that effective reuse depends on compatibility between experience and the applying model.Replacing LongCat’s own lessons with GLM’s lowers its gain, while replacing GLM’s lessons with LongCat’s turns GLM’s positive transfer negative.
  • Harness Effects: Harness effects vary jointly with model and workload, so no harness dominates across models and categories.The findings support a fixed strong harness for controlled comparisons but task-aware harness selection in deployment.

J Examples and Evaluation Prompts

The appendix makes the evaluation concrete through a complete task instruction, trajectory-derived lessons, and a conservative rubric for judging whether solutions are novel. These materials specify task constraints, iterative record-keeping, diagnostic lessons, and examples of standard techniques that do not count as novelty.

  • J Examples and Evaluation Prompts: The appendix provides complete task instructions, trajectory-derived lessons, and the rubric and demonstrations used for solution-novelty assessment.The materials are intended to make the experimental interface and evaluation procedure concrete.
  • J.1 Task Instruction Example: The grpo_multisource task fine-tunes Qwen2.5-VL-7B with GRPO to maximize MathVista visual-math accuracy using specified editable files, datasets, and local evaluation.The instruction restricts edits, forbids network access, requires a single GPU, and imposes an eight-hour budget.
  • J.1 Task Instruction Example: The task requires full-budget iteration, prompt reporting of unrecoverable environment issues, and git commits that preserve each runnable round.The instructions prohibit rewriting prior history and require the final artifact to remain in the designated output directory.
  • J.2 Example of Trajectory-Derived Lessons: DeepSeek-V4-Pro’s transferred lessons increased avg@3 by +0.26 and best@3 by +0.66 on held-out llm_online_serving relative to lesson-free rollouts.The lessons warn against trusting noisy evaluations, stale caches, delayed ceiling detection, and omitting data augmentation or re-formatting.
  • J.3 Solution Novelty Classification Rubric: The novelty rubric treats textbook techniques, optimized standard implementations, stacked known techniques, standard hyperparameters, expected data engineering, and evaluation hacking as non-novel.Novelty is reserved for approaches such as cross-domain transfer, problem reformulation, or custom task-specific algorithms.
  • J.3 Solution Novelty Classification Rubric: Judges assign one primary category based only on the final solution, compare against the task’s standard technique, and resolve uncertainty conservatively toward a trivial-form category.The rubric states that novel-approach is the only category carrying novelty and that evaluation-hacking never qualifies.
  • J.3 Solution Novelty Classification Rubric: The grpo_multisource example is classified as training-signal/data-eng because its multi-source data, reward engineering, and tuning follow the task’s standard playbook.The resnet_bit_flip example is classified as search-hardcode because gradient-saliency bit-flip search plus hardcoding is standard for that task.
Loading 2608.13417v1…