PaperlayerView exact revision on arXiv

Source-linked AI summary

HarnessX: A Composable, Adaptive, and Evolvable Agent Harness Foundry

Tingyang Chen, Shuo Lu, Kang Zhao, Weicheng Meng, Hanlin Teng, Tianhao Li, Chao Li, Xule Liu, Jian Liang, Zhizhong Zhang, Yuan Xie, Heng Qu, Kun Shao, Jian Luan

arXiv:2606.14249v3cs.AI

TL;DR

Agent harnesses mediate model behavior, yet infrastructure for composing them as typed, substitutable components is lacking. HarnessX combines typed composition with trace-driven multi-agent evolution, yielding +14.5% average gains across five benchmarks, up to +44.0%.

  • Problem

    Agent harnesses mediate model behavior, yet existing infrastructure lacks a typed, substitutable layer for composing them.

    Source: §1 · p. 3, §3 · p. 5

  • Method

    HarnessX treats harnesses as typed, composable values and evolves them from execution traces with AEGIS, a multi-agent engine.

    Source: §1 · p. 3, §3 · p. 5

  • Results

    +14.5% average gain across five benchmarks, up to +44.0%, with gains largest where baselines are lowest.

    Source: §1 · p. 4, §8 · p. 23

  • Takeaways & Limitations

    These results suggest that harness composition and evolution from execution feedback are a complementary lever for agent progress, particularly for capability-limited agents.

    Source: §8 · p. 23

    Evaluation uses the same task set for evolution and measurement, leaving generalization to unseen tasks untested.

    Source: §7.7 · p. 23

Abstract

from arXiv · show

AI agent performance depends critically on the runtime harness, comprising the prompts, tools, memory, and control flow that mediate how a model observes, reasons, and acts. Yet today's harnesses remain largely hand-crafted and static: each new model or task still demands bespoke scaffolding, and the rich traces produced during execution are rarely distilled back into systematic improvement. We introduce HarnessX, a foundry for composable, adaptive, and evolvable agent harnesses. HarnessX assembles typed harness primitives via a substitution algebra, adapts them through AEGIS, a trace-driven multi-agent evolution engine grounded in an operational mirror between symbolic adaptation and reinforcement learning, and closes the harness-model loop by turning trajectories into both harness updates and model training signal. Across five benchmarks (ALFWorld, GAIA, WebShop, tau^3-Bench, and SWE-bench Verified), HarnessX yields an average gain of +14.5% (up to +44.0%), with gains largest where baselines are lowest. These results suggest that agent progress need not come from model scaling alone: composing and evolving runtime interfaces from execution feedback is an actionable and complementary lever. Project homepage: https://darwin-agent.github.io/HarnessX/.

1 Introduction

Agent performance depends on harness mediation as well as the underlying model, but harnesses are static, architecturally entangled, and disconnected from model training. HarnessX treats the harness as a composable, adaptable, and evolvable object, using execution traces to improve harnesses and models together.

Empirical Validation

  • Harness evolution averaged +14.5% absolute across 15 model–benchmark configurations, reached +44.0%, and improved 14 of 15 configurations.The evaluation spans five benchmarks, three task-agent families, and up to 15 evolution rounds.Source: §1 · p. 3
  • On ALFWorld, Qwen3.5-9B gained +44.0% versus Sonnet 4.6’s +11.2%; on heterogeneous GAIA, variant isolation restored non-degrading improvement of +13.6% over 15 rounds.The introduction attributes the larger ALFWorld gain for the weaker agent to evolved harnesses addressing behavioral gaps it cannot self-correct.Source: §1 · p. 3

Harness Composition

  • Harness composition uses typed processors attached to lifecycle hooks, a nine-dimensional taxonomy, and a substitution algebra for type-safe per-task insertion and removal.Making each behavioral change’s scope explicit is presented as a precondition for variant isolation during evolution.Source: §1 · p. 3

Harness Adaptation

  • AEGIS maps harness adaptation to reinforcement-learning constructs and uses a four-stage trace-driven pipeline with deterministic gating to address reward hacking, catastrophic forgetting, and under-exploration.Its stages are Digester, Planner, Evolver, and Critic; optional variant isolation is intended to prevent cross-task interference on heterogeneous benchmarks.Source: §1 · p. 3

Harness-Model Co-Evolution

  • Harness-model co-evolution interleaves harness evolution with model reinforcement learning over a shared replay buffer, and cross-harness GRPO yields an additional +4.7% over harness-only evolution.The model is trained to internalize strategies from successive harness versions.Source: §1 · pp. 3–4

2 Related Work

Prior harness work provides reusable primitives, orchestration patterns, and evolving-agent methods, but leaves gaps in full harness composition and systematic trace-driven improvement. HarnessX addresses these gaps through typed composition and trace-driven harness evolution.

2.1 Harness Engineering

  • Typed libraries, orchestration frameworks [28], and static products [34] leave cross-layer composition and in-loop harness improvement unresolved.Source: §2.1 · p. 4
  • Claude Code’s Dynamic Workflows generates task-specific scripts in-session, while HarnessX adds multi-round trace-driven adaptation, typed composition, observability, and shared-buffer harness–model co-training.Source: §2.1 · p. 4

2.2 Self-Evolving Agents

  • Prompt optimizers such as APE [52], OPRO [43], ProTeGi [29], DSPy [14], and MIPRO [27] make instructions learnable, but leave tools, memory, and control flow outside their optimization scope.Source: §2.2 · p. 4
  • Memento [50] reuses execution cases without fine-tuning, while MIA [30] couples compressed trajectory memory with an evolving parametric planner through a bidirectional experience loop.MIA demonstrates superiority across eleven benchmarks.Source: §2.2 · p. 4
  • Workflow-search methods GPTSwarm, ADAS [12], AFlow [48], A2Flow, AgentSwift [21], ResMAS [53], and EvoAgentX [37] optimize agent collaboration but leave component-level artifacts static.These works show that workflow structure is learnable and yields larger gains than prompt-only optimization.Source: §2.2 · p. 5
  • Harness-evolution systems including SICA [31], Darwin Gödel Machine [18], and AHE [22] establish harnesses as evolutionary targets, while heuristic-learning theory maps trace credit assignment, falsifiable changes, and proposal-critique to RL concepts.The cited harness-evolution designs lack a unifying theoretical framework connecting observed failure modes to principled defenses; HarnessX formalizes the RL-symbolic correspondence as an operational mirror.Source: §2.2 · p. 5

3 Harness Composition

HarnessX treats the harness as a typed, substitutable configuration and composes behavior from processors inserted at typed hook points. It defines the harness, processor interface, and nine behavioral dimensions as the substrate for harness evolution.

3 Harness Composition

  • HarnessX makes the harness a first-class value and composes it by inserting typed processors at hook points.The design addresses the absence of infrastructure for typed, substitutable harnesses; its definitions establish the edit surface for harness evolution.Source: §3 · p. 5

3.1 The Harness as a First-Class Object

  • An agent pairs a model configuration with an independently substitutable harness configuration, separating model roles and fallback policies from agent behavior.The harness configuration comprises hook-indexed processor lists and shared infrastructure slots; it can be serialized, compared, hashed, and substituted independently of model identity.Source: §3.1 · pp. 5–6

3.2 The Processor Abstraction

  • Processors consume and yield events through five outcomes, and same-type per-hook composition allows insertion or removal while preserving pipeline type correctness.Hook contracts are validated after each invocation; singleton groups prevent duplicate conflicts, while ordering hints and soft dependencies constrain composition.Source: §3.2 · p. 6

3.3 The Nine-Dimensional Taxonomy

  • HarnessX organizes behavior into nine dimensions, from model selection and context assembly to observability and the training bridge.The dimensions cover memory, tools, execution environment, evaluation and reward, and control and safety; AEGIS edits across them, while observability supplies traces and the training bridge supplies trajectory records for co-evolution.Source: §3.3 · p. 6

4 Harness Adaptation

HarnessX evolves typed harnesses through AEGIS, which treats harness configurations as symbolic states and edits as actions, using execution traces and verifier scores as feedback. Its architecture combines trace-driven adaptation with deterministic safeguards and can isolate conflicting behaviors in multiple harness variants.

4.1 The Operational Mirror

  • Harness edits are open-ended code-level artifacts generated by the meta-agent, while type constraints prune invalid compositions rather than relying on exhaustive search.Edits can modify one or more dimensions while preserving type contracts; the Planner proposes them from trace-grounded hypotheses.Source: §4.1 · p. 8
  • Harness evolution is a harness-level MDP: configurations are states, typed code edits are actions, and traces plus verifier scores provide feedback.The model stays fixed during evolution; an edit is committed only if it does not regress previously solved tasks, while the trace store grows whether the candidate is accepted or rejected.Source: §4.1 · p. 8

4.2 Pathologies in Symbolic Space

  • Symbolic harness evolution amplifies reward hacking [10], catastrophic forgetting [15], and under-exploration [16], which AEGIS addresses with the Critic, deterministic gate, and Planner, respectively.The risks arise because an evolver can target verification protocols and edits to shared components can have non-local effects; local, low-risk edits can also crowd out structural exploration.Source: §4.2 · pp. 8–9

4.3 AEGIS Architecture

  • AEGIS’s four stages—Digester, Planner, Evolver, and Critic—are selectively invoked by one meta-agent, while every surviving candidate must pass a deterministic gate.The Digester compresses traces into task-level evidence; the Planner considers incremental and structural edits; the Evolver generates typed candidates; and the Critic checks their support and risks. The gate checks manifests, normalization, applicable tests, and regressions before shipping.Source: Figure 2 · p. 7, §4.3 · pp. 9–10

4.4 The Adaptation Loop

  • Each adaptation round runs the current harness on a batch and commits a new harness only when a candidate clears all acceptance checks.The Digester, Planner, and Evolver may each short-circuit when their continuation condition is unmet; the Critic and deterministic gate are mandatory for candidates reaching them.Source: §4.4 · p. 11

4.5 Variant Isolation via Ensemble Routing

  • Variant isolation maintains up to K harnesses and routes tasks to the variant with the highest estimated success rate for their cluster.If an edit improves some tasks but regresses others, the system forks a variant rather than rejecting it; regression checks are scoped to tasks routed to the target variant.Source: §4.5 · p. 11

5 Harness-Model Co-Evolution

HarnessX co-evolves the harness and model because improving either alone can hit a ceiling set by the other. Its shared replay buffer lets both updates learn from the same trajectories, while cross-harness GRPO trains the model on strategies elicited by evolving harnesses.

5 Harness-Model Co-Evolution

  • Co-evolution addresses the scaffolding ceiling of harness-only improvement and the training-signal ceiling of model-only improvement by improving both components together.The harness shapes what the model perceives and can invoke; the model supplies reasoning and planning. A stronger apparatus cannot compensate for a weak core, or vice versa.Source: §5 · p. 11

5.1 The Co-evolution Iteration

  • Each co-evolution iteration stores verified traces in a shared FIFO buffer, then uses that buffer for both a discrete AEGIS harness edit and a GRPO model update.Both updates finish before the next rollout, and neither conditions on the other’s output within the iteration.Source: §5.1 · p. 12

5.2 Optimization Substrates

  • The harness makes discrete structural changes, while the model learns fine-grained behaviors such as tool use and termination that symbolic edits cannot capture.The harness supplies coarse-grained strategy architecture; the model learns to exploit it. In co-evolution, buffered trajectories also span model checkpoints, exposing harness evolution to behavioral variation.Source: §5.2 · p. 13

5.3 Model Training via Cross-Harness GRPO

  • Cross-harness GRPO groups trajectories by task across model and harness versions, using verifier-reward contrasts to train the model on strategies that succeeded.This supplies inter-strategy variation for advantage estimation; trajectories are evaluated under the harness context that produced them, so versions with incompatible action spaces can share a task group.Source: §5.3 · pp. 13–14

5.4 Off-Policy Training over a Mixed-Policy Buffer

  • GRPO adds no rollouts: each exploration round’s trajectories train the model by replay while also driving AEGIS harness evolution.The added cost is limited to a cached forward pass per trajectory and gradient steps; FIFO eviction also bounds model-version lag and harness staleness.Source: §5.4 · p. 14

6 Experiments

HarnessX experiments test harness evolution across five benchmarks and multiple task-agent families, finding broad success-rate gains alongside evidence that variant isolation and model co-evolution address limits of a single evolving harness.

6.1 Experimental Setup

  • Experiments span five benchmarks and three task-agent families, using up to T=15 evolution rounds with early stopping after P=3 rounds without a shipped edit.The benchmarks span retrieval, embodied planning, web interaction, multi-turn dialogue, and software engineering; task success is measured with benchmark-specific verifiers using pass@2. The meta-agent is Claude Opus 4.6 unless otherwise noted, and held-out task generalization is not evaluated.Source: Table 3 · p. 15, Benchmarks · p. 15

6.2 Main Results

  • Qwen3.5-9B gains +44.0% on ALFWorld (baseline 53.0%), +17.1% on GAIA (baseline 20.3%), and +18.2% on SWE-bench Verified (baseline 23.6%).On ALFWorld, GPT-5.4 gains +20.9% and Sonnet 4.6 gains +11.2%; the reported pattern is that gain magnitude tracks baseline performance rather than proximity to the meta-agent’s family. ALFWorld and SWE-bench converge rapidly when failures concentrate in few component types, while GAIA (Sonnet 4.6) requires 11 rounds across four types.Source: §6.2 · pp. 15, 17
  • AEGIS improves 14 of 15 model–benchmark configurations, averaging +14.5% and reaching gains up to +44.0%.Table 4 reports pass@2 success rates before and after harness evolution; gains are measured on the same task set used for evolution, not held-out tasks.Source: §6.2 · p. 15, Benchmarks · p. 15, Table 4 · p. 16

6.3 Evolution Strategy Comparison

  • Variant isolation lifts GAIA GPT-5.4 from ∆=0.0 to +13.6% (87.4%, non-degrading), whereas Global peaks at 73.8% at R4 before a −24.3% peak–final gap.Ensemble routing maintains up to K harness variants and evaluates edits per variant, preventing edits for one task cluster from regressing another; it peaks at R14 and consumes 107.8M versus 143.7M tokens under Global.Source: Table 5 · p. 17, §6.3 · pp. 17–18

6.4 Meta-Agent Effectiveness

  • The four-stage AEGIS pipeline has comparable accuracy to the single-agent evolver, whose 1.0% accuracy gap is within one standard error but which uses ∼14% more tokens.The Digester compresses ∼10M raw trace tokens to ∼10K structured summaries; the authors attribute the token difference to less-informed, more frequently rejected proposals without this stage.Source: Table 6 · p. 18, §6.4 · p. 18

6.5 Co-Evolution

  • Co-evolution raises peak success from 37.4% to 41.7% on GAIA (+4.3%) and from 49.0% to 54.0% on WebShop (+5.0%), averaging +4.7%.The end-of-run gaps persist: GAIA 36.4% →39.8% and WebShop 46.0% →50.0%; cross-harness GRPO lets the model internalize strategies introduced by successive harness versions.Source: Figure 5 · p. 19, §6.5 · p. 19

6.6 Failure Analysis

  • All three predicted pathologies appear: reward hacking is mitigated within two rounds, under-exploration is signaled by declining ship-prediction accuracy, and Telecom forgetting self-corrects by R9.In Telecom, five consecutive same-type edits preceded a sixth reminder that reduced compliance from 94.7% to 80.7% (−14.0%); per-edit gating missed sub-threshold coupling. Reward hacking involved verifier-format exploitation, while prompt-space exhaustion accompanied under-exploration.Source: §6.6 · pp. 19–20

7 Discussion

The discussion argues that compositional structure and rich execution traces enable more diagnosable harness evolution, while neither guarantees stability. It also describes cost and oversight tradeoffs and bounds the claims with untested generalization and methodological limitations.

7.1 Why Compositional Structure Matters for Evolution

  • 73.8% (R4) to 49.5%: Global’s GAIA peak-to-final gap is −24.3%, as pass@2 can hide sub-threshold regressions without variant isolation.Typed components make edit scope explicit, enabling variant isolation; they do not prevent bad edits or guarantee bounded behavioral effects, as accumulated same-type edits can couple across dialogue patterns.Source: §7.1 · p. 21

7.2 The Role of Trace Richness

  • Structured traces reveal pathologies that binary outcomes cannot, but accumulated sub-threshold regressions can evade detection until damage occurs.GAIA reward hacking required distinguishing format exploitation from genuine retrieval, while ALFWorld under-exploration required tracking edit-type distribution and ship-prediction accuracy; τ 3-Bench Telecom regressions evaded the seesaw constraint despite five prior trace rounds.Source: §7.2 · p. 21

7.3 Scope and Limits of the Operational Mirror

  • The RL–symbolic-space mirror is a design checklist, not a formal or predictive theory, and the reported improvement is not guaranteed to extend to longer horizons or inseparable task distributions.It does not predict which pathology will dominate or when; the three observed pathologies are not exhaustive, and distribution shift or reward sparsity may produce analogous failures.Source: §7.3 · pp. 21–22

7.4 Generalization Across Model Families

  • 53.0% → 97.0% (+44.0%): Opus 4.6’s harness evolution generalizes to the different-family, weaker Qwen3.5-9B task agent on ALFWorld.Across the three configurations, the meta-agent remained Opus 4.6; gains followed inverse baseline performance (Qwen > GPT > Sonnet), and weaker meta-agents were not evaluated.Source: §7.4 · p. 22
  • 86.4% vs. 87.4%: the single-agent evolver and four-stage AEGIS pipeline achieved comparable accuracy within sampling noise at n=103.With the same meta-agent model and infrastructure, the four-stage decomposition primarily provided ∼12% fewer tokens and auditability rather than measurable accuracy improvement.Source: §7.4 · p. 22

7.5 Cost-Performance Tradeoffs

  • 87.4% vs. 49.5% final and 107.8M vs. 143.7M tokens: variant isolation outperforms and uses fewer tokens than Global on GAIA.Isolation evaluates each edit on its target cluster; Global’s degradation also wastes tokens on candidates that fail gating. Global suffices on benchmarks that converge quickly within the run horizon. At deployment, GAIA’s upfront 107.8M tokens amortize within ∼1,300 invocations (∼83K tokens saved per invocation), while ALFWorld’s per-task cost rises ∼60% for an accuracy gain of +11.2%; GAIA per-task token use drops ∼25%.Source: §7.5 · p. 22

7.6 Ethical Considerations

  • Per-edit gating and human oversight support safer evolution, but the τ 3-Bench failure shows that sub-threshold coupling can accumulate despite apparent stability.HarnessX archives edits with manifests and rollback targets, rejects edits regressing any previously solved task under pass@2, and supports human approval above a configurable risk threshold; the automated experiments did not exercise that approval.Source: §7.6 · pp. 22–23

7.7 Limitations

  • The evaluation lacks held-out tasks and covers only discrete text actions, a closed-source meta-agent, joint harness–model control, and limited benchmark samples.All gains use the evolution task set, leaving same-distribution generalization untested; open-weight meta-agents and continuous actions are untested, joint control may require cross-team coordination, and SWE-bench Verified uses 55 tasks while τ 3-Bench covers three domains.Source: §7.7 · p. 23

8 Conclusion · Appendix

HarnessX treats the runtime harness as a composable interface between model and environment, combining typed primitives, trace-driven evolution, and model training in a unified improvement loop. Across five benchmarks and three model families, its results support runtime-interface improvement from execution feedback as a complementary lever to model scaling.

8 Conclusion

  • HarnessX treats the harness as a first-class interface between model and environment.Source: §8 · p. 23
  • The interface can be composed from typed primitives.Source: §8 · p. 23
  • Execution traces provide the basis for evolving the harness.Source: §8 · p. 23
  • Harness evolution is coupled with model training in a unified improvement loop.Source: §8 · p. 23
  • Up to +44.0% gains and an average +14.5% across 15 configurations were achieved across five benchmarks and three model families.The gains came through trace-driven evolution over a compositional substrate.Source: §8 · p. 23
  • Co-evolution added +4.7% beyond harness-only evolution on two benchmarks.Source: §8 · p. 23
  • Harness-level gains are largest for capability-limited agents, making runtime-interface improvement from execution feedback a complementary and actionable lever to model scaling.Source: §8 · p. 23

A Experimental Setup: Full Details

The appendix expands Section 6.1’s condensed setup with full benchmark descriptions, formal metric definitions, evolution protocol hyperparameters, and runtime infrastructure.

A Experimental Setup: Full Details

  • The appendix provides full benchmark descriptions, formal metric definitions, evolution protocol hyperparameters, and runtime infrastructure details.It expands the condensed setup in Section 6.1.Source: A Experimental Setup: Full Details · p. 28

A.1 Benchmarks … SWE-bench Verified.

The evaluation spans five benchmarks selected to cover harness-sensitive tasks from embodied planning to software engineering. Together, they test tool use, planning, shopping, policy adherence, and repository-level code editing.

A.1 Benchmarks

  • Five benchmarks span harness-sensitive failure modes, from short-horizon embodied planning to long-horizon software engineering.ALFWorld represents embodied planning, while SWE-bench Verified represents software engineering.Source: §A.1 · p. 28, ALFWorld · p. 28, SWE-bench Verified · p. 28
  • GAIA [25] tests open-ended tool-based reasoning through real-world questions requiring multiple actions, with performance measured by exact match.Tasks may require web search, file extraction, multimodal interpretation, and arithmetic.Source: §A.1 · p. 28

ALFWorld.

  • ALFWorld [35] tests multi-step planning and grounded search on 134 valid-unseen household tasks, measured by goal-completion rate.The text-based agent directs a simulated robot under a tight step budget across six task types.Source: ALFWorld · p. 28

WebShop.

  • WebShop [44] evaluates product search, browsing, selection, and purchase across 100 fixed-seed shopping sessions, scoring how well the selected item matches the request.Source: WebShop · p. 28

τ 3-Bench.

  • τ^3-Bench [45] tests multi-turn customer-service policy adherence across Retail, Airline, and Telecom, measuring rule compliance over full conversations.The harness must prevent agreement to disallowed actions across turns.Source: τ 3-Bench · p. 28

SWE-bench Verified.

  • SWE-bench Verified measures patch resolution on a sampled 55-task subset requiring repository edits that pass hidden tests.Tasks stress codebase navigation, fault localization, patch implementation, and regression avoidance.Source: SWE-bench Verified · p. 28

A.2 Evaluation-Set Design … D Additional Results

This appendix documents the evaluation sets and scoring, the evolution protocol and runtime infrastructure, and the prompts and manifests guiding adaptation and task agents. It also traces a GAIA harness intervention and describes the organization of additional benchmark analyses.

A.2 Evaluation-Set Design

  • Evaluation reuses fixed task sets each round: GAIA 103 tasks, ALFWorld 134, WebShop 100, full lists in three tau^3-Bench domains, and 55 SWE-bench Verified tasks.The repeated scoring means Appendix D curves compare rounds on the same tasks rather than moving samples.Source: §A.2 · p. 28

A.3 Metric Definitions

  • Evolution curves use pass@2, counting a task solved if either of two independent rollouts succeeds; infrastructure-failure rollouts count as failures.This reduces sensitivity to single-rollout stochasticity while retaining a strict task-level criterion.Source: §A.3 · p. 29

A.4 Evolution Protocol and Hyperparameters

  • Round 0 starts from a competent composed harness with benchmark-specific tools; Opus 4.6 is the meta-agent, while task agents vary across experiments.Per-task step limits depend on the benchmark; the cited passage identifies Sonnet 4.6, GPT-5.4, and Qwen3.5-9B as task-agent models.Source: §A.4 · p. 29

A.5 Runtime Infrastructure

  • Each rollout uses a fresh environment and records its full trajectory; task rollouts run at concurrency 10, and meta-agent roles at concurrency 4.Co-evolution training uses 8× H100 GPUs, batch size 256, and learning rate 1 × 10−6.Source: §A.5 · p. 29

B.1 Meta-Agent Prompts

  • The Planner synthesizes trace evidence for the Evolver, while the Critic audits candidates and requires regressions to be addressed or explicitly justified.For code candidates, the Evolver prompt requires actual verification; the Critic rejects a round if listed regressions are neither targeted nor justified as acceptable.Source: §B.1 · pp. 30–33

B.2 Round-0 Task-Agent Prompts

  • The Round-0 composition layer loads one system prompt per benchmark; the appendix shows ALFWorld as a representative default with one admissible action per call.The prompt instructs the agent to choose commands verbatim from the admissible list and only report completion after the environment’s done signal.Source: §B.2 · p. 34
  • Round-0 ALFWorld prompts prescribe ordered subgoals, systematic exploration, immediate object pickup, required transformations, direct delivery, progress tracking, and loop avoidance.They also instruct agents to trust the admissible action list and open closed containers before judging them empty.Source: §B.2 · p. 35

B.3 Change-Manifest Schema

  • Each Evolver candidate includes a change manifest linking its proposed edit to evidence, mechanism, expected effect, and attribution signal for falsification against the next round’s trace features.Table 9 defines the manifest fields as an evidence ledger for shipped edits.Source: §B.3 · p. 35, Table 9 · p. 36

C Anatomy of an Evolution Step

  • The round-10 GAIA/Sonnet 4.6 cycle illustrates Digester compression, Planner synthesis, Evolver editing, and Critic judgment through a composite intervention.It was selected as a richer example because it combined a tool, prompt, and configuration change and produced the run’s largest single-round gain.Source: C Anatomy of an Evolution Step · p. 36

C.1 Worked Example: GAIA / Sonnet 4.6, Round 10

  • The round-10 edit addressed repeated zero-character Wikipedia fetches with a MediaWiki API tool, a prompt instruction, and restored round-8 configuration.The Critic approved this composite edit over a revert-only candidate after verifying full-string tool output and considering persistent source-access failures.Source: §C.1 · pp. 36–37
  • GAIA pass rate rose from 74.8% at R9 to 79.6% at R10 (+4.9pp), with five tasks flipping to pass and a 0.71 hit rate.This was the run’s greatest improvement; five of seven tasks predicted to be affected flipped to pass, the highest hit rate for any ship across all 19 runs.Source: §C.1 · p. 37

D Additional Results

  • Additional benchmark results are organized around failure-cluster breakdowns, per-model distributions of shipped harness levers, and model-by-lever effectiveness heatmaps.The panels respectively show what fails, how each model evolves, and whether evolution closes failures.Source: D Additional Results · p. 37

D.1 GAIA

GAIA failures concentrate in evidence access and reasoning, and HarnessX adapts all four harness levers to address them. Tool edits reduce retrieval failures, while prompt edits target reasoning; visual and document-heavy errors remain difficult to resolve.

Failure clusters and their causes

  • Blocked-source failures dominate GAIA at 39%, followed by reasoning at 33%, figure/visual and document/table parsing at 11% each, and scope ambiguity at 6%.Blocked-source failures involve inaccessible or incomplete evidence; the other clusters involve multi-hop inference, visual grounding, structured-document extraction, and query disambiguation.Source: §D.1 · pp. 37–38

Per-model evolution logic

  • Sonnet used all four levers, GPT-5.4 focused on prompts, and Qwen3.5’s sole tools edit had the highest yield of any cell.Sonnet shipped 11 prompt, 7 processor, 7 config, and 6 tools edits; GPT-5.4’s prompt ships were 45% of its total. Across models, prompts targeted behavioral clusters while tools addressed mechanical failures prose could not fix.Source: §D.1 · p. 38

Did evolution close the clusters?

  • WikiTextFetch replaced unreliable browser-based Wikipedia retrieval with a MediaWiki API call, reducing blocked-source failures, while prompt edits encouraged verification to reduce reasoning errors.Figure/visual and document/table parsing failures remained harder to reduce, leaving residual errors concentrated in visual and document-heavy tasks.Source: §D.1 · p. 38

D.2 ALFWorld

ALFWorld failures are dominated by inefficient search and step limits, while HarnessX’s improvements depend on model strength: stronger models benefit mainly from prompt edits, and weaker models need structural support.

D.2 ALFWorld

  • Search / step-ceiling failures account for 89% of ALFWorld failures, compared with 7% prompt-rule side effects and 4% object-type confusion.Prompt-rule side effects and long-path failures were only partially reduced; some episodes still exceeded the interaction budget.Source: §D.2 · p. 39, Figure 9 · p. 40
  • Sonnet improves almost entirely through prompt edits, while GPT-5.4 adds a step-budget processor and Qwen3.5 needs prompt, config, and processor edits.Qwen3.5’s processor intercepts reasoning text and re-emits tool calls when needed; structural levers are recruited when prompting is insufficient.Source: §D.2 · p. 39
  • Qwen3.5’s processor and config edits hit 0.84 and 0.71, supporting +44.0pp improvement; Sonnet’s prompt edits hit 0.49 versus its processor’s 0.14.Qwen3.5’s structural edits re-emitted missed tool calls and adjusted execution budgets, bringing it closer to closed-model runs.Source: §D.2 · p. 39, Figure 9 · p. 40

D.3 WebShop · D.4 τ 3-Bench · D.5 SWE-bench Verified

Across WebShop, τ 3-Bench, and SWE-bench Verified, evolution targets benchmark-specific failure modes with different harness levers; its gains vary with the model’s capability and remaining headroom.

D.3 WebShop

  • WebShop failures shifted from early search and pagination loops toward product judgment, led by wrong product (46%), with pagination loops at 21%, colour matching at 17%, and attribute checks at 17%.Prompt rules and advisory processors address navigation loops, while colour-matching tools and config edits target attribute matching and long-session context; the tool did not improve performance in this run.Source: §D.3 · p. 40, Figure 10 · p. 41
  • 60% to a peak of 76%: WebShop improved as prompt edits consistently helped (hit-rates 0.37–0.50), while config edits aided the two weaker models.Advisory processors produced only modest gains (0.20–0.25), and the colour-matching tool had a 0.0 hit-rate, leaving residual navigation and product-judgment errors.Source: §D.3 · pp. 40–41, Figure 10 · p. 41

D.4 τ 3-Bench

  • In τ 3-Bench, premature or unverified action (28%) and wrong selection or count (24%) together made up over half of non-interrupt failures.Prompt and processor edits were used for every model, with zero tool edits; they encode policy ordering and enforce it mid-dialogue, while config was the sharpest lever where used (Qwen3.5 0.67; GPT-5.4 0.33 hit-rate).Source: §D.4 · p. 41, Figure 11 · p. 42
  • τ 3-Bench’s remaining failures include incomplete multi-step fixes (16%), missed steps or sub-tasks (14%), policy misinterpretation (13%), and capability-boundary confusion (5%).The boundary failures arise when telecom faults are on the user’s handset, beyond the agent’s device-side tools; wrong-selection and policy-judgment errors remain harder than premature-action and missed-step clusters.Source: §D.4 · p. 41
  • GPT-5.4 started lowest at 76.2% and gained most (+14.5pp); Sonnet 4.6 gained +5.4pp, while near-ceiling Qwen3.5-9B gained +1.1pp.The loop was not monotone: Sonnet’s telecom run reached 100% at R4, fell to 80.7% at R7, then recovered to 99.1% by R9.Source: §D.4 · p. 41

D.5 SWE-bench Verified

  • SWE-bench Verified failures were mainly incomplete fixes (62%) and wrong diagnoses (19%), with a mechanical tail of no edit (6%), anchor mismatch (5%), and budget exhaustion (4%).The failures were under-fixes rather than reward-hacking: the harness applies the gold test patch before the model patch and blocks test-file writes.Source: §D.5 · p. 42, Figure 12 · p. 43
  • SWE-bench evolution was prompt-first with zero tool edits; its secondary lever shifted from processor edits for Sonnet to config edits for GPT-5.4 and a varied mix for Qwen3.5.Prompt edits broadened fix scope and config edits restored workflow pacing, but the 9B model could not reliably execute the predicted fixes.Source: §D.5 · p. 42, Figure 12 · p. 43
  • GPT-5.4 rose from 45% to a 64% peak and Sonnet stabilized near 87%, while Qwen3.5-9B peaked at 42% with zero durable gains.Strong-model productive-lever hit-rates reached 0.48 for GPT-5.4 config edits and 0.40 for Sonnet prompt and processor edits; Qwen3.5-9B lever hit-rates were 0.05, 0.05, and 0.06, respectively.Source: §D.5 · p. 42, Figure 12 · p. 43

E Reproducibility and Artifacts · E.1 Per-Run Directory Layout

HarnessX writes each evolution run to a self-describing directory whose logged artifacts let readers reconstruct decisions. The layout includes run-level records, per-round artifacts, and raw rollouts.

E.1 Per-Run Directory Layout

  • Each run is stored under runs/<run_name>/ with an index, round journal, pass-rate trajectory, scoreboard, and structured audit log.The journal has one entry per round, curves.json records per-round pass rates, and audit.jsonl logs stage, gate, and commit events.Source: §E.1 · p. 43
  • Per-round R<n>/ artifacts include planner cross-trace synthesis, candidate change manifests, and applied configurations with asset files.The manifests are named candidates/C-R<n>-NN.md; applied artifacts are stored under applied/C-R<n>-NN/.Source: §E.1 · p. 43
  • Each round records a ship or no_op decision, per-candidate verdicts, and tasks worsened relative to the previous round.These records appear in decision.md, verdicts/V-C-R<n>-NN.md, and regressions.md, respectively.Source: §E.1 · p. 43
  • Run-level data includes task history by round and task, outcomes for historical ships, and rejected candidates.These artifacts are stored in data/task_history.jsonl, data/ship_outcomes.json, and data/rejected_candidates.jsonl.Source: §E.1 · p. 43
  • Per-task failure analyses are stored as digests/*.md within each round’s artifacts.Source: §E.1 · p. 43
  • Raw rollout records are stored as JSONL files under each round’s trajectories/ directory.Source: §E.1 · p. 43

Related papers

Loading 2606.14249v3…