PaperlayerView exact revision on arXiv

Source-linked AI summary

AutoSaddler: Automatic Harness Optimization with Durable Updates from Agent Execution Traces

Sungho Park, Wonjoong Kim, Rongyuan Tan, Jue Zhang, Wook-Shin Han, Pengfei Gao, Chanyoung Park, Yongqiang Yao, Rao Fu, Elsie Nallipogu, Qingwei Lin, Saravan Rajmohan, Dongmei Zhang

arXiv:2608.23041v1cs.AIcs.CLcs.LGcs.MAcs.SE

TL;DR

Long-horizon LLM agents remain unreliable, while manually designing external harnesses across prompts, tools, and control logic is costly. AutoSaddler treats harness improvement as offline learning from execution failures, using diagnosis, structured patches, and generalization-aware selection. Across three benchmarks, it improves over corresponding base harnesses by 9.0, 9.6, and 10.0 percentage points.

  • Problem

    Long-horizon agent reliability remains challenging, and manually optimizing harness prompts, tools, and control logic is expensive.

  • Method

    AutoSaddler formulates harness optimization as offline learning over execution traces and combines failure diagnosis, structured patching, and generalization-aware update selection.

  • Results

    AutoSaddler improves over corresponding base harnesses by 9.0, 9.6, and 10.0 percentage points on GAIA2, SWE-Bench Pro, and Terminal-Bench 2.0, respectively.

  • Takeaways & Limitations

    The results support deep debugging, targeted modifications, and generalization-aware selection as ingredients for durable harness updates.

  • Takeaways & Limitations

    The approach assumes gold answers and a task-level success metric, which may be unavailable or prohibitively expensive in real-world deployment.

Abstract

from arXiv · show

LLM agents remain unreliable on long-horizon tasks, where small local failures can compound over extended interactions and lead to overall task failure. Although external harnesses can substantially improve robustness, harness design remains a manual and expensive process that requires searching over a large space of prompts, tool configurations, and control logic. We propose AutoSaddler, an automatic harness optimization framework that formulates harness improvement as an offline learning problem and iteratively updates the harness using failure signals from mini-batches. AutoSaddler combines failure-trace diagnosis, structured patch generation that treats the harness as code, and validation-based update selection. Experiments on GAIA2, SWE-Bench Pro, and Terminal-Bench 2.0 show that AutoSaddler substantially improves agent performance over the corresponding base harnesses, achieving gains of 9.0, 9.6, and 10.0 percentage points, respectively. Ablation studies further suggest that effective harness optimization benefits from three ingredients: deep debugging rather than shallow reflection, targeted modifications rather than unconstrained editing, and generalization-aware selection rather than trajectory-specific repair. Together, these results suggest that automatic harness optimization is a promising path toward more performant and reliable agent systems.

1 Introduction

LLM agents remain unreliable on long-horizon tasks, motivating external harnesses, but manually optimizing prompts, tools, and control logic is costly. AutoSaddler addresses this challenge by iteratively diagnosing failures, generating structured harness patches, and selecting updates that generalize.

  • Long-horizon agent reliability is difficult because success requires sustained competence across many consecutive decisions.
  • Manual harness tuning is costly because it requires searching broad design spaces, evaluating lengthy trajectories, and analyzing failures.
  • AutoSaddler formulates harness optimization as offline learning from failure signals in mini-batches.
  • AutoSaddler combines failure-trace diagnosis, structured patch generation, and validation-based selection to produce harness updates.
  • 9.0, 9.6, and 10.0 percentage points are the reported gains over corresponding base harnesses on GAIA2, SWE-Bench Pro, and Terminal-Bench 2.0.The evaluation covers three challenging agent benchmarks.

2 Related Work

Prior work optimizes prompts, agent experiences, workflows, and external harnesses, while AutoSaddler focuses on offline, generalization-oriented optimization across prompts, tools, and runtime control logic. It also targets mini-batch and long-trace diagnosis rather than only single-trace or task-specific repair.

  • Auto Prompt Optimization: Prompt optimization spans gradient-free, textual-gradient, evolutionary, planning-based, programming, autodiff-style, and Bayesian approaches.
  • Agent Systems and Harness Optimization: Harness optimization is broader than prompt optimization because its search space includes prompts, tools, and runtime control logic.
  • Self-Evolving Agents and Experience-Based Improvement: Experience-based and self-evolving agents use failures to build libraries, tools, memory, skills, or knowledge bases, typically through online continual learning.
  • Self-Evolving Agents and Experience-Based Improvement: AutoSaddler instead studies offline harness optimization for generalization across environments without requiring self-referential task and meta-agent dynamics.
  • Agent Trace Failure Diagnosis and Repair: Prior trace-diagnosis methods often target single traces and assume context-window fit, whereas AutoSaddler addresses very long traces and mini-batch diagnosis.
  • LLM-Driven Evolutionary Search: AutoSaddler uses LLM-based candidate selection and patch recombination from exploration history, while leaving specialized evolutionary algorithms for future work.

3 Preliminaries

The paper models agent execution as stochastic behavior induced by a harness and optimizes harness parameters under a rollout budget. Its scope covers prompts, tools, and middleware, with performance estimated from repeated task executions and candidates selected by development-set scores.

  • Agentic Task and Execution Trace: Agentic tasks require multi-step reasoning, repeated tool use, and environment interaction, with stateless and independent task instances.
  • Agentic Task and Execution Trace: A harness parameter θ induces a stochastic execution process producing an execution trace τ and final output ŷ for task x.Stochasticity includes LLM sampling, tool-use decisions, and environment interactions.
  • Optimization Space: Agent Harness: The optimization space contains prompt instructions, tool interfaces, and middleware runtime control logic, while excluding memory and skill curation.The exclusion follows the assumption that tasks are largely stateless and independent.
  • Objective: Budget-Constrained Optimization: The objective maximizes expected task performance over the target distribution using a task-level metric such as accuracy or pass/fail success.
  • Objective: Budget-Constrained Optimization: Given rollout budget K, AutoSaddler searches evaluated candidate harnesses and returns the one with the highest development-set empirical score.

4 Proposed Method: AutoSaddler

AutoSaddler treats harness optimization as offline mini-batch learning, iteratively diagnosing traces, generating structured patches, verifying updates, and selecting candidates using development-set generalization. Its EvoDAG records explored harnesses and lessons to guide subsequent evolution.

  • Overview: AutoSaddler formulates harness optimization as an offline learning problem using mini-batches during development.The task set is split into training, development, and test sets, with rollout-based evaluation performed during training.
  • Diagnosis–Patch–Verification: Each iteration evaluates the current harness, diagnoses failed traces, generates a structured patch, and verifies the patched harness on the same mini-batch.Diagnosis and patch generation use execution traces and expose only harness-functional source files, organized into Prompt, Tool, and Middleware categories.
  • Generalization-Aware Selection: Accepted updates are evaluated on the development set, and the final harness is the candidate with the highest empirical development-set score.The selected harness is evaluated once on the held-out test set and is not modified using test feedback.
  • Patch Space: Capability patches alter executable code or orchestration, while Steering patches modify textual prompts, tool descriptions, or reminders without changing executable code.AutoSaddler schedules these patch groups in phases, beginning with Capability Patches and then transitioning to Steering Patches.
  • Reflection and Evolution: Reflection compares pre- and post-patch outcomes, stores lessons and performance signals in the EvoDAG, and enables evolution by recombining components from previously explored harnesses.The EvoDAG is a directed acyclic graph whose nodes represent harnesses and whose edges represent harness diffs.
  • Design Rationale: AutoSaddler is designed around in-depth diagnosis, structured intervention, and generalization-aware selection rather than shallow reflection or unconstrained editing.These requirements motivate grounding updates in execution traces and harness code, restricting the patch space, and testing whether improvements generalize.

5 Experiments

Experiments across GAIA2, SWE-Bench Pro, and Terminal-Bench 2.0 show that AutoSaddler improves base-harness performance, remains robust under distribution changes, and benefits from diagnosis, structured intervention, and generalization-aware selection.

  • Main Results: AutoSaddler improves Pass@1 over the corresponding base harness by +9.0 pp on GAIA2, +8.4 pp on SWE-Bench Pro, and +10.0 pp on Terminal-Bench 2.0.The corresponding results are 53.0% → 62.0%, 37.3% → 46.9%, and 40.0% → 50.0%, respectively.
  • Main Results: AutoSaddler outperforms the strongest automated baseline by +7.4 pp on GAIA2, +6.2 pp on SWE-Bench Pro, and +4.4 pp on Terminal-Bench 2.0.On TB2, it also exceeds the manually expert-tuned Terminus KIRA harness by +2.5 pp.
  • Robustness: AutoSaddler remains effective under optimization reruns, training-distribution shifts, and a change from Opus 4.6 to Haiku 4.5.An independent GAIA2 run reaches 58.6% Pass@1, a different training universe yields a +5.9 pp gain, and cross-model transfer yields a +5.6 pp gain over the base harness.
  • Optimization Efficiency: AutoSaddler reaches higher development accuracy with fewer task-agent rollouts than automated baselines.On GAIA2, it reaches 72.3% with approximately 1,000 rollouts, while GEPA and Meta-Harness saturate at 64.6% and 61.5% after approximately 2,800 executions; its best dev-set score uses 147 learning rollouts, about 10× fewer than Meta-Harness.
  • Ablations: Removing in-depth diagnosis reduces GAIA2 Pass@1 from 62.0 to 57.8 and produces fewer accepted patches.The diagnosis–patch session uses, on average, 6.2 additional tool calls and 5.8 additional file accesses per optimization step; accepted patches reach 13 versus 5 by Iteration 25.
  • Ablations: Structured intervention raises GAIA2 Pass@1 from 56.9% to 62.0% relative to its ablation by diversifying exploration toward higher-value patch types.Without structured intervention, Steering patches comprise 91.5% of generated patches, while New Tool, Loop Change, and Infra Change patches account for only 4% versus over 25% with AutoSaddler.

6 Conclusion

AutoSaddler treats harness improvement as offline learning over execution traces, using diagnosis, targeted patching, validation, and symbolic evolution to produce durable updates. Its design emphasizes capability exploration followed by behavioral refinement, with validation before lessons enter optimizer memory.

  • Framework: AutoSaddler formulates harness improvement as offline learning over execution traces and combines diagnosis, structured patching, and generalization-aware selection.The harness spans prompts, tools, middleware, and control logic.
  • Optimization view: The mini-batch analogy is structural rather than literal because harness updates are semantic interventions selected through diagnosis, verification, validation, reflection, and evolution.Unlike neural parameters, harness changes are not direct numeric optimizer steps.
  • Diagnosis and patching: Failed rollouts require root-cause hypotheses and targeted interventions because failures may originate in prompts, tools, middleware, or the agent loop.Same-batch verification tests whether the inferred textual gradient is plausible before commitment.
  • Generalization and memory: Validation precedes symbolic optimizer updates because misleading lessons stored in EvoDAG can influence future harness evolution.Reflection distills before/after evidence, while EvoDAG retains patches, outcomes, and reusable lessons.
  • Patch scheduling: AutoSaddler uses a two-phase schedule: capability patches first address tooling and control-flow gaps, then steering patches refine behavior after capabilities stabilize.The transition point k may be specified by iteration count or training epochs, and verification remains necessary because patch effects vary.

B Additional Details on Experiment Setup

The experiments use benchmark-specific infrastructure and disjoint task-group splits to evaluate harness generalization under distribution shift. Baseline adaptations and budget choices account for differences in harness structure and optimization regimes.

  • EvoDAG implementation: The evo-dag CLI exposes structured summaries of patch history, lessons, scenarios, and code diffs to make accumulated EvoDAG state accessible during optimization.The interface groups commands into scoped read operations and per-session node or scenario updates.
  • Baseline adaptations: The Meta-Harness baseline is adapted to repository-level optimization because GAIA2 harnesses can span an entire repository rather than one Python script.This enables patching across the full repository instead of imposing the original single-file setting.
  • Evaluation protocols: AutoSaddler is evaluated with official benchmark infrastructure to keep reported results comparable to prior leaderboard numbers.GAIA2 uses ARE and a judge model, SWE-Bench Pro uses Docker and regression tests, and Terminal-Bench 2.0 uses isolated containers and task-specific scripts.
  • Data splits: Train, development, and test sets are separated by task groups for GAIA2 Universes and SWE-Bench Pro repositories, while Terminal-Bench 2.0 uses a uniform random partition.The grouped splits place unseen personas or repositories in testing; Terminal-Bench 2.0 has 89 heterogeneous tasks and no natural grouping axis.
  • Optimization budgets: AutoSaddler and GEPA use two epochs on GAIA2 and SWE-Bench Pro and four on Terminal-Bench 2.0, whose smaller training set motivates the larger budget.Meta-Harness uses full-batch optimization, so epoch counts are not directly comparable to mini-batch methods.

C Robustness to Optimization Stochasticity

Independent optimization runs indicate that AutoSaddler’s GAIA2 gains are not specific to one favorable search trajectory, although the experiment provides only limited statistical characterization.

  • Results: 58.6% Pass@1 on GAIA2 Universe 22 was achieved by AutoSaddler’s second run, 2.1 points below the first run and 7.1 points above the default agent.The second run also exceeded independently rerun GEPA and Meta-Harness harnesses by 8.0 and 7.4 points, respectively.
  • Interpretation: Two independent optimization runs produced consistent AutoSaddler gains, suggesting improvement is not specific to a single favorable search trajectory.Each resulting harness was evaluated on the held-out Universe 22 using repeated executions.
  • Scope: The stochasticity analysis is limited because only two optimization trajectories were evaluated.Table 7 reports Pass@1 as mean ± standard deviation over three executions.

D Robustness to Training-Distribution Shift

AutoSaddler retains strong GAIA2 performance when optimization uses a different training universe and when the resulting harness is deployed with a weaker task-agent backbone. These tests probe sensitivity to training distribution and model transfer.

  • Training-distribution shift: Optimization on different training universes produced effective harness updates rather than relying only on properties specific to Universe 29.The Universe 24-trained harness was evaluated three times on held-out Universe 22 under the unchanged protocol.
  • Training-distribution shift: 57.4% Pass@1 was achieved by a harness optimized on Universe 24, improving over the default agent by 5.9 percentage points.The result remained within 1.2 points of the independent AutoSaddler run trained on Universe 29.
  • Cross-model transferability: AutoSaddler was also evaluated for cross-model transfer using Claude Haiku 4.5 as the task-agent backbone with harnesses optimized by Claude Opus 4.6.The supplied passage introduces an overall improvement but does not provide its complete numerical value.

F Fine-Grained Ablations of Structured Intervention and Generalization-Aware Selection

Fine-grained ablations show that both mechanisms within structured intervention and generalization-aware selection contribute to final harness performance.

  • Structured Intervention: 5.9 percentage points: removing only Phased Patch Scheduling reduces Pass@1 from 60.7% to 54.8% on GAIA2 Universe 22.Removing the patch taxonomy as well lowers performance to 53.3%, adding a further 1.5-point reduction.
  • Generalization-Aware Selection: Development-set filtering reduces Pass@1 from 60.7% to 50.0%, making it the larger effect in the generalization-aware selection comparison.The ablation is evaluated on GAIA2 Universe 22 with three repeated test executions.
  • Generalization-Aware Selection: 5.1 percentage points: removing Reflection with EvoDAG after development-set filtering further reduces Pass@1 from 50.0% to 44.9%.Together, the results indicate that both selection mechanisms contribute to final harness performance.

G Patch Durability Analysis

Patch durability depends more clearly on intervention mechanism than on broad harness component: capability patches preserve comparable repairs while causing fewer regressions than steering patches.

  • Component-Level Analysis: 57–59%: Prompt, Tool, and Middleware patches have similar fix rates, but Tool patches regress at 19% versus 14% for Prompt and Middleware.The broad component categories mix distinct intervention mechanisms, limiting their separation.
  • Mechanism-Level Analysis: 58% versus 55%: Steering Patches have a slightly higher fix rate than Capability Patches, but regressions are 17% versus 8%.Capability Patches therefore achieve comparable local repair rates while reducing regressions by 9 percentage points.
  • Mechanism-Level Analysis: Capability-level interventions tend to produce more durable updates, whereas textual steering is more susceptible to spillover outside the motivating scenarios.This comparison uses the higher-level Capability–Steering taxonomy to separate intervention mechanisms.

I End-to-End Optimization Cost Characterization

AutoSaddler’s optimization efficiency is assessed through both learning progress and optimizer-side and task-agent computation, whose costs differ substantially across methods.

  • Learning and Evaluation Efficiency: 73.7% dev accuracy: AutoSaddler reaches this Terminal-Bench 2.0 result after 31 task executions and 12 leveraged traces.This surpasses Meta-Harness at 63.2% by 10.5 percentage points and GEPA at 57.9% by 15.8 percentage points.
  • Learning and Evaluation Efficiency: Over 8× fewer traces: AutoSaddler reaches its best Terminal-Bench 2.0 performance with 12 leveraged traces versus 98 for Meta-Harness.The comparison begins from a common 52.6% starting point.
  • Optimizer-Side Cost: $14.56 per generated patch: AutoSaddler incurs higher optimizer-side cost than Meta-Harness, while optimizing broader harness components.GEPA costs $5.50 per patch but searches only over the system prompt.
  • Task-Agent Evaluation Cost: 20.2 LLM calls and 203.9 seconds: a single GAIA2 task-agent rollout averages these costs, alongside 550,988 input and 7,380 output tokens.Harness evaluation is considerably more expensive than the optimizer-side operations.
  • Evaluation Protocol: AutoSaddler evaluates six training scenarios per iteration and invokes 65-scenario development evaluation only for patches that improve the training mini-batch.Meta-Harness evaluates all 140 GAIA2 training and development scenarios for every candidate update.
  • Search Trajectory: The EvoDAG search progresses through foundation, rapid improvement, selective merging and repair, and consolidation phases.A regression to 33.8% at Iter20 was followed by recovery through the evolving search trajectory.

K Additional Analysis for RQ1

Additional analyses connect in-depth diagnosis with greater investigative effort, more accepted patches, and more accurate root-cause identification across diverse task failures.

  • Measurement: Tool and file-access counts are measured consistently across AutoSaddler and its ablation, with file accesses counted through dedicated tools and Bash commands.The accounting excludes tool calls issued inside sub-agents because they are not recorded in the main trace.
  • Search Trajectory: 72.3%: the EvoDAG reaches a global dev-set peak at Iter27 after escaping the Iter20 regression and local optima.The trajectory is organized into foundation, rapid improvement, selective merging, and consolidation phases.
  • Diagnosis Overhead: 6.2 additional tool calls and 5.8 additional file accesses per step: AutoSaddler investigates more than the ablated variant without in-depth diagnosis.The averages are 69.7 tool calls and 45.5 file accesses for AutoSaddler, versus 63.5 and 39.7 for the ablation.
  • Qualitative Case Studies: Deep debugging identifies the true root cause in an ambiguous calendar task, unlike shallow diagnosis.The case contrasts thorough investigation with a diagnosis that attributed the failure to misunderstanding the instruction.
  • Qualitative Case Studies: Filesystem verification reveals an unverified parent directory as the cause of the file-management failure, rather than a relative-path error.The agent had created benchmarks/sims3 at the filesystem root instead of locating benchmarks under Documents.
  • Qualitative Case Studies: Full ride-history inspection finds the missed second matching candidate and supports a targeted search_ride_history fix.The tool enables filtered search by date, start location, and end location, surfacing conflicting rides in one call.
  • Accepted Patches: 20 versus 15 accepted patches: AutoSaddler accumulates more accepted patches than the w/o In-depth Diagnosis ablation.The gap emerges by Iteration 25 and persists through Epoch 2.

L Additional Analysis for RQ2

AutoSaddler’s optimization trajectory alternates capability expansion with behavioral refinement, while trace-based diagnosis and mandatory causal checks aim to produce durable, generalizing updates.

  • Phased Patch Scheduling: During Exploration, AutoSaddler prioritizes Capability patches that address structural harness limitations beyond prompt editing.Examples include new tools, implementation fixes, and agent-loop logic changes.
  • Phased Patch Scheduling: During Refinement, the search shifts to Steering patches that tune prompts, tool descriptions, and PreToolUse hooks.A constructive response template replaced a prohibitive hook and recovered over-corrected cases.
  • Optimization Trajectory: 72.3% peak dev-set accuracy followed the constructive hook replacement, contributing to a cumulative 24.6 pp improvement.The reported trajectory links capability expansion followed by behavioral refinement with this cumulative gain.
  • Generalization-Aware Selection: Reflection compares patch contents with dev-set score changes to distinguish generalizing patches from patches that overfit identical mini-batch outcomes.The analysis examines lessons extracted across iterations and uses failure lineages to develop generalization principles.
  • Instructional Procedure: The procedure requires history analysis, root-cause diagnosis, phase-appropriate patching, and post-patch verification before completing an optimization session.State changes must be checked against traces and diffs to separate causal fixes or regressions from stochastic artifacts.
  • Limitations and Future Work: AutoSaddler’s current scope assumes supervised task data with gold answers and task-level success metrics, while stateful settings remain future work.The authors identify unsupervised or weakly supervised optimization, memory integration, and broader model coverage as directions for extension.
Loading 2608.23041v1…