PaperlayerView exact revision on arXiv

Source-linked AI summary

Adaptive Auto-Harness: Sustained Self-Improvement for Agentic System Deployment on Open-Ended Task Streams

Zewen Liu, Zhan Shi, Yisi Sang, Bing He, Minhua Lin, Tianxin Wei, Dakuo Wang, Benoit Dumoulin, Wei Jin, Hanqing Lu

arXiv:2606.01770v2cs.LGcs.AI

TL;DR

Open-ended, heterogeneous, and shifting task streams challenge auto-harness systems evaluated mainly on fixed benchmarks. Adaptive Auto-Harness combines sustained evolution, harness-tree routing, and human steering, with variants jointly leading all reported metrics and reaching 97.9% PolyBench Coverage without intervention.

  • Problem

    Auto-harness systems lack evaluation and design guidance for unbounded streams with growing histories, heterogeneous tasks, and distribution shifts.

  • Method

    Adaptive Auto-Harness decomposes regret into evolution and adaptation losses, then uses multi-agent evolution, solve-time harness-tree routing, and human steering.

  • Results

    97.9% PolyBench Coverage is achieved without human intervention, while the three variants jointly lead all reported metrics across the evaluated streams.

  • Takeaways & Limitations

    Sustained construction and task-wise adaptation address complementary deployment challenges, with routing helping when harness-branch signals are reliable.

  • Takeaways & Limitations

    Evaluation covers three benchmark streams, and evolution and adaptation losses are diagnosed analytically rather than directly estimated as oracle losses.

Abstract

from arXiv · show

Auto-harness systems such as A-Evolve, GEPA, and Meta-Harness improve LLM agents by optimizing prompts, skills, tools, memories, and supporting infrastructure from execution feedback, but they are typically evaluated on fixed offline benchmarks. Real deployments instead present open-ended task streams: histories grow without a fixed endpoint, heterogeneous tasks require different harnesses, and problem distributions shift over time. These challenges make a single repeatedly and densely updated harness brittle, causing performance degradation as accuracy peaks early and then declines. This motivates sustained harness construction with task-wise adaptation. We introduce Adaptive Auto-Harness, a framework and system for such streams. The framework decomposes the gap to an oracle harness into evolution loss and adaptation loss. The system addresses these losses with a stateful multi-agent evolver, a harness tree with solve-time routing, and human-steering hooks for cases where history lacks the needed signal. Across prediction-market, security-competition, and event-forecasting streams, Adaptive Auto-Harness outperforms five existing auto-harness baselines and ablations attribute gains to better construction, routing, or targeted human steering. Code is available in \href{https://github.com/A-EVO-Lab/a-evolve/tree/release/adaptive-auto-harness}{Link}.

1 Introduction

Open-ended task streams expose limits of static auto-harness evaluation because histories grow, tasks vary, and distributions shift. Adaptive Auto-Harness addresses these pressures through sustained harness construction, solve-time task adaptation, and targeted human steering.

  • Motivation: Open-ended streams require harnesses to improve continuously as histories grow, task types vary, and distributions shift over time.Static offline benchmarks do not capture this deployment pressure.
  • Framework: The framework decomposes the gap to an optimal harness into evolution loss and adaptation loss.This decomposition motivates separating sustained construction from per-task harness fitting.
  • System: Adaptive Auto-Harness uses a stateful multi-agent evolver, a harness-tree router for solve-time adaptation, and structurally triggered human-in-the-loop hooks.The hooks target cases where historical experience is insufficient.
  • Validation: The system is evaluated on streaming prediction markets, security challenges, and event forecasting, with diagnosis through baselines, component ablations, and human-in-the-loop slice analyses.These experiments examine both aggregate performance and the proposed deployment gaps.

2 Related Work

Related work spans continual learning and distribution-shift adaptation for task streams, mixture-of-experts routing for heterogeneous inputs, and LLM-agent systems that evolve their harnesses from execution feedback.

  • Continual Learning in Task Streams: Continual learning studies sequential task learning while retaining earlier capabilities, alongside methods addressing distribution shift through domain and test-time adaptation.
  • Continual Learning in Task Streams: Mixture-of-experts methods route heterogeneous inputs to specialised components, covering a complementary part of the task-stream setting.
  • Self-improving and self-evolving agents: A-Evolve, GEPA, and Meta-Harness are LLM-agent precedents that update prompts, skills, memory, tools, or archives directly from execution feedback.
  • Self-improving and self-evolving agents: A-Evolve uses a linear-chain evolver over batch trajectories, while GEPA performs reflective Pareto prompt evolution using textual rather than scalar feedback.

3 Method

Adaptive Auto-Harness models open-ended harness construction as regret minimization against an oracle and decomposes regret into evolution and adaptation losses. It addresses these losses with a stateful multi-agent evolver, a harness tree with solve-time routing, and human steering when history lacks needed signal.

  • Problem formulation: The evolver constructs a bounded harness C = φ(H_t) from prior experience, including prompts, skills, memory, and tools under capacity budget K.A solver then acts using the harness-conditioned policy π(a | x_t, C_t).
  • Loss decomposition: Regret is decomposed into evolution loss L_evo and adaptation loss L_adapt relative to an oracle harness optimized for each incoming task.L_evo reflects the evolver class’s capability gap, whereas L_adapt reflects committing to one harness before observing the task.
  • Evolution loss: A stateful four-phase multi-agent evolver, temporal-reveal feedback, and persistent cross-cycle memory expand evolver capability for unbounded streams.Temporal-reveal gates expose evaluation signals only after task resolution, avoiding future-information leakage.
  • Adaptation loss: The system stores specialized harnesses in a tree and uses agentic routing at solve time to select branches for heterogeneous recurring task regimes.Branches isolate regime-specific prompts, skills, and tools, reducing cross-contamination while keeping solve-time adaptation lightweight.
  • Human-in-the-loop channel: When history lacks required signal, a human-in-the-loop channel augments it through task-board steering and interactive research assistance.These hooks provide domain guidance, source access, or real-time intervention for barriers such as authentication walls.

4 Experiments

Experiments evaluate Adaptive Auto-Harness on three temporally ordered open-ended task streams against auto-harness baselines, no-evolution controls, and a human-designed system. Results show gains from benchmark-specific construction, task-wise routing, stateful evolution, and targeted human steering.

  • Experimental setup: Experiments use PolyBench, CTF-Dojo, and FutureX, with Claude Sonnet 4.6 as solver and Claude Opus 4.6 as temperature-zero evolver.Baselines include A-Evolve, GEPA, Meta-Harness, Continual Harness, SkillOS, no-evolution controls, and OctoTools.
  • Main comparison: A-Evolve leads CTF-Dojo and FutureX pass rates, while Meta-Harness leads PolyBench Coverage, Accuracy, and Return but underperforms no-evolution Sonnet on FutureX.A-Evolve reaches 45.2% CTF-Dojo and 47.5% FutureX; Meta-Harness reaches 55.3% Coverage, 50.8% Accuracy, and +320% Return.
  • Main comparison: The Multi-agent variant leads FutureX at 49.5% and reaches 50.2% CTF-Dojo Pass, while the Adaptive variant leads PolyBench Return at +352%.These results associate multi-agent evolution with source and tooling improvements and Adaptive routing with specialized market strategies.
  • Bottleneck analysis: Benchmark bottlenecks differ: PolyBench requires consensus-aware confidence calibration, FutureX requires source acquisition, and CTF-Dojo requires payload-handling infrastructure.FutureX pass rate rises from 34.0% to 47.6% to 57.1%; CTF-Dojo performance declines with file size, while the multi-agent variant retains roughly a 9-point margin.
  • Evolution ablations: The full four-role stateful evolver improves over a single-agent evolver on PolyBench, CTF-Dojo, and FutureX: 20.3 →44.3 CWR, 38% →43%, and 38% →44%.The ablations remove temporal-reveal feedback or cross-cycle memory, and the system uses resolved outcomes as later evolution signal.
  • Human steering: Human steering helps when it injects missing external source knowledge, using research-phase credentials or task-board source directions rather than generic advice.On 100 FutureX tasks, lift is 0 on broad polymarket questions and rises to +5 on some slices.

5 Conclusions

Adaptive Auto-Harness addresses unbounded arrivals, heterogeneous tasks, and non-stationarity by combining sustained harness construction with solve-time task adaptation. Its evolution-loss and adaptation-loss decomposition explains why repeatedly updating a single harness is insufficient.

  • 5 Conclusions: Adaptive Auto-Harness targets unbounded task arrival, heterogeneous tasks, and non-stationarity in open-ended deployment streams.These challenges motivate sustained harness construction paired with solve-time task adaptation.
  • 5 Conclusions: The framework decomposes the gap to an oracle harness into evolution loss and adaptation loss.This decomposition clarifies the distinct needs to build missing capabilities from stream evidence and select the appropriate specialized branch.
  • 5 Conclusions: A single repeatedly updated harness is insufficient because open-ended streams require both capability construction and specialized-branch selection.Adaptive Auto-Harness combines sustained construction with task-wise adaptation to address these requirements.

6 Limitations

The evaluation covers three open-ended task streams spanning unboundedness, task heterogeneity, and distributional non-stationarity, but broader deployment streams remain to be tested. The paper also identifies evolution loss and adaptation loss as diagnostic losses.

  • Benchmark coverage: The evaluation uses prediction markets, cybersecurity challenges, and event forecasting as open-ended task streams.These domains represent unbounded streams, heterogeneous tasks, and distributional non-stationarity.
  • Benchmark coverage: The framing should be tested on additional deployment streams expanded further across space and time to better mimic real-world deployment.
  • Diagnostic losses: The paper defines evolution loss Levo and adaptation loss Ladapt as diagnostic losses.

7 Ethics Statement

The system uses public benchmark tasks without private user data, isolates CTF-Dojo from real systems, and limits human steering to operational guidance rather than answer exposure or solver-branch selection.

  • The evaluation uses public research benchmark tasks and introduces no private user data.
  • CTF-Dojo runs only inside isolated benchmark containers and does not target real systems.
  • Human steering is limited to source guidance, task-board edits, and credential decisions, excluding answer labeling, answer exposure, and solver-branch selection.

8 AI Usage Statement … A.3 FutureX

The paper reports AI-assisted writing, debugging, and analysis, then details chronological evaluation across three open-ended benchmarks spanning prediction markets, security competitions, and event forecasting. The benchmark protocols address temporal information constraints, duplicate-log handling, and domain-specific evaluation conditions.

  • 8 AI Usage Statement: AI assistants refined the paper’s writing and accelerated debugging and analysis during implementation and evaluation.
  • A Benchmark and Evaluation Details: All three benchmarks evaluate tasks chronologically, restricting solvers and delaying evolver outcome labels until information becomes available.The analysis pipeline loads only the first record for each instance_id, so retries and duplicate logs do not change reported metrics.
  • A.1 PolyBench: PolyBench contains 5,075 Polymarket-derived tasks spanning politics, sports, finance, crypto, and entertainment from Feb 6–22, 2026.Market labels remain hidden from evolution until their corresponding resolution times.
  • A.1 PolyBench: Return = Coverage · CWR, combining market coverage with dollar-weighted profitability to complement Accuracy’s equal-trade treatment.Coverage discounts high CWR achieved on only a thin slice of the stream, while Accuracy is blind to stake sizing.
  • A.2 CTF-Dojo: CTF-Dojo provides 261 chronologically ordered security challenges from 2011 to 2024 across exploitation, web security, cryptography, reverse engineering, and forensics.Each challenge runs in a constrained per-task Docker sandbox, and submitted flags are verified by SHA-256 hash comparison.
  • A.3 FutureX: FutureX contains 503 event-forecasting questions over 82 days from Jan–Apr 2026, covering finance, technology, geopolitics, and entertainment in English and Chinese.The zh-finance slice requires source discovery beyond default English-only retrieval, while changing web pages and search indices create leakage risks.

B Implementation and Reproducibility Details

The paper specifies a reproducible execution protocol using chronological provider-hosted LLM runs, released artifacts, and benchmark-specific task and evolution counts. It also prevents temporal leakage and separates evolution workspaces from solve-time routing, with replay-based diagnostic controls and logged human interventions.

  • Execution protocol: 5,075/261/503 solve trajectories and 51/14/26 evolution cycles were run for PolyBench/CTF-Dojo/FutureX, respectively.Runs use provider-hosted LLM APIs with native tool calling and benchmark task order; reported numbers come from results.jsonl files.
  • Hyperparameters: All runs use T = 0 for both solver and evolver to attribute gains to the algorithm rather than sampling noise.Table 3 documents the benchmark-specific hyperparameters and the Expected-Gain-from-Learning trigger governing whether a cycle runs.
  • Temporal reveal: Evolution receives trajectory data immediately but outcome feedback only after resolution, so unresolved tasks remain unlabeled history rather than leaked supervision.Solver calls are filtered against each task’s release timestamp.
  • Workspace artifacts: The evolver maintains task, research, verification, testing, and architecture artifacts separately from the solver workspace, while routing inspects only branch metadata.The evolver can update harness files during evolution; the solve-time router selects a branch for each incoming task.
  • Branch replay and human steering: Every evolved-harness branch is replayed on a curated subset to construct Oracle and Worst diagnostic bounds, while the deployed router sees neither labels nor branch outcomes.Human-steering events are author-provided interventions logged with their triggering phase, requested external signal, and work.

C Benchmark Non-Stationarity Diagnostics · D Further Experiments

The diagnostics show that the benchmark streams are temporally non-stationary rather than static IID pools, making harnesses fitted to earlier tasks vulnerable to later mismatch. Further experiments indicate that evolver strength, construction budget, and specialized domain workspaces materially affect performance.

  • C Benchmark Non-Stationarity Diagnostics: Benchmark diagnostics show temporal shifts that can mismatch a harness fitted to earlier observations with later tasks.Figures 12–14 are descriptive diagnostics, not evaluation metrics, and use task or benchmark properties, with some panels incorporating outcomes or related signals.
  • C Benchmark Non-Stationarity Diagnostics: Tradeable markets fall from 97% early to 31% late, while near-even markets rise from 18% to 35%, requiring adaptation in calibration, abstention, and evidence gathering.Markets with maximum price above 0.95 also decline from 44% to 29%, and the marketprice correctness proxy changes from 84% early to 77% late.
  • D Further Experiments: Supplemental diagnostics examine evolver capability and construction budget, as well as cross-domain workspace dilution.These experiments clarify where the main gains come from without duplicating the primary RQ analyses.
  • D Further Experiments: Figure 10 shows that higher construction budget helps Haiku and Sonnet but gives little additional gain once Opus already reaches high performance.The experiment is conducted on CTF-Dojo and compares evolver capability across models and budgets.
  • D Further Experiments: Combining all evolved workspaces sharply reduces PolyBench CWR, while a PolyBench-evolved workspace reaches the highest CWR.The result supports specialized harness branches rather than a single dense harness.
  • D Further Experiments: On CTF-Dojo, stronger evolvers achieve higher pass rates, while additional construction budget mainly helps weaker evolvers and saturates for the strongest model.Figure 10 varies both the evolver model and construction budget; higher budget helps Haiku and Sonnet but adds little once Opus reaches high performance.
  • D Further Experiments: On PolyBench, the PolyBench-specific workspace performs best, whereas the all-evolved workspace loses 57 points of CWR.Figure 11 reports that combining all evolved workspaces sharply reduces CWR, supporting specialized harness branches rather than one dense harness.

E Per-Domain and Per-Category Breakdowns … H Human-in-the-Loop Event Log

The appendix shows that performance varies substantially by category, language, domain, and market conditions, motivating specialized harnesses and routing. It also reports multi-agent evolution dynamics, solve-time branch selection, and targeted human intervention for source-acquisition and task-board bottlenecks.

  • E Per-Domain and Per-Category Breakdowns: CTF-Dojo gains are largest on web and crypto, while binary/pwn remains hardest at 14.8% despite evolution and routing.The binary/pwn bottleneck is consistent with sandbox payload handling.
  • E Per-Domain and Per-Category Breakdowns: FutureX benefits most from evolution on English-language slices, but zh-finance remains constrained when source discovery is blocked by a search wall.Domains are inferred by keyword matching, with unmatched questions grouped into an “other” bucket.
  • E Per-Domain and Per-Category Breakdowns: PolyBench’s sports category dominates portfolio return because liquid sports markets carry most dollar-weighted profit, whereas politics has near-zero return despite high accuracy.Categories are inferred from keywords in the trajectory prompt’s Event description.
  • F Multi-Agent Evolution Dynamics: Table 9 contrasts No-evo, Single-agent, and Multi-agent systems on the full stream while adding the multi-agent run’s peak cycle and total evolution cycles.The full-stream comparison complements the subset-based Figure 6 analysis rather than reproducing it.
  • G Routing Behaviour and Branch Performance: The router always selects a regime-specific branch on the navigation subsets and never invokes its permitted main fallback.Low realised pass rates can reflect genuinely hard tasks rather than failing branches, as indicated by corresponding low Oracle pass rates.
  • G Routing Behaviour and Branch Performance: Adapt closes a substantial fraction of the Oracle–Naive gap on CTF-Dojo and PolyBench but trails Naive slightly on FutureX because source acquisition dominates.Per-batch analysis reports the same Oracle/Adapt/Naive/Worst series on replayable batches.
  • H Human-in-the-Loop Event Log: In the 5-batch FutureX human-in-the-loop run, two P2 research events fire at cycle 1, while later P3 prompts skip unless the cheat-sheet contains a relevant intervention.The reported slice-level lifts across regimes 1–5 are 0, +5, +20, +15, and 0.

I Run-Detail Analysis · J System Prompts

The run-detail analysis characterizes solver effort across benchmarks, while the system prompts specify routing and a four-phase evolution process designed to detect and isolate non-transferable fixes. Together, these details show how Adaptive Auto-Harness manages heterogeneous task workloads and prevents shortcut artifacts from degrading later performance.

  • I Run-Detail Analysis: Solver effort is right-skewed on CTF-Dojo and FutureX, so the analysis reports both arithmetic means and medians for turns and elapsed seconds.A small number of long-running tasks pull the mean above the median, while wall-clock excludes orchestration overhead.
  • I Run-Detail Analysis: CTF-Dojo frequently saturates its solver budget, whereas evolved variants typically reach a flag or give up earlier.Sonnet’s mean of 89.4 turns reflects many tasks looping until the cap.
  • I Run-Detail Analysis: Human steering in the FutureX run used credential and task-board hooks, with two bootstrap P2 events and one substantive P3 event directing the evolver toward Western and Chinese specialty endpoints.The remaining P3 prompts returned skip under the cheat-sheet protocol.
  • I Run-Detail Analysis: PolyBench is dominated by direct decisions, with most tasks submitting immediately at turn count 1 because the prompt already provides the needed market context.A direct submission without other tool use counts as 1 turn.
  • J System Prompts: The router selects exactly one branch by matching task signals against branch-specific positive and negative routing criteria.A strong unique match receives confidence >= 0.7; ambiguous cases route to main with uncertainty reflected in confidence.
  • J System Prompts: The evolution cycle assigns distinct roles: analysts identify prioritized failure regimes, researchers test solutions, builders implement verified approaches, and verifiers test the resulting code.Research is explicitly upstream of building, and verification determines whether code goes live or returns to the builder for retry.
  • J System Prompts: Branching decisions keep domain-generalizable, non-degrading fixes on main while isolating regime-specific or cross-category harmful fixes on branches.The prompts discourage creating branches for fewer than 2 tasks or single-cycle observations and prioritize keeping main clean.
Loading 2606.01770v2…