PaperlayerView exact revision on arXiv

Source-linked AI summary

DarwinX: Evolving Agent Harnesses Through Natural Selection

Yifan Zhang, Yutong Dai, Juntao Tan, Luyu Yang, Rishi Mullur, Thai Hoang, Zhiyuan Hu, James Zhu, Phil Mui, Silvio Savarese, Ran Xu, Zeyuan Chen

arXiv:2608.07545v1cs.NEcs.AIcs.LGcs.SE

TL;DR

Existing harness self-improvement is path-dependent and can regress other tasks, so DarwinX evolves a population of harnesses through preserve-and-extend selection. Across benchmarks, the frozen-model approach improves performance and transfers to a different benchmark, while transferred gains remain narrow.

  • Problem

    Single-lineage harness evolution is path-dependent and can improve one task family while regressing another, limiting robust cross-task self-improvement.

  • Method

    DarwinX selects among harness variants by preserving solved tasks while extending coverage to fragile or failing tasks, enabling improvements to compound.

  • Results

    Across benchmarks, DarwinX improves frozen-model agents, including WebArena-Infinity audit-clean pass@1 from 43.5% to 93.0% and TerminalWorld held-out performance to 28/41 tasks.

  • Takeaways & Limitations

    Harness selection can improve general agent competence across tasks and verifiers without changing the underlying model.

  • Takeaways & Limitations

    Cross-benchmark transfer was measured in only one direction, and the transferred gain was within a narrow band around the reference.

Abstract

from arXiv · show

An LLM agent's capability depends not only on model weights but on its harness: prompts, tools, skills, and control flow. Self-improvement loops already edit harnesses, yet single-lineage search is path-dependent and local wins often regress other tasks. We introduce DarwinX, which treats self-evolution as selection over a population of harnesses with the model frozen: a preserve-and-extend contract admits only variants that extend coverage without regressing, an archive keeps alternative lineages for recombination, and failure-, teacher-, and self-derived evidence share one edit interface. Fitness comes from each benchmark's own verifier: no gold solutions, no hand-picked winners. Across four benchmarks that progressively separate the evolution signal from the test, one loop adds about 17 points on average: Terminal-Bench 2.1 rises +7.7 to 83.2% on a matched base and to the verified frontier at 84.7% on a stronger one; TerminalWorld's held-out split reaches 68.3%, ahead of every off-the-shelf agent; WebArena-Infinity real-task pass@1 rises from 43.5% to 93.0% audit-clean; and a Terminal-Bench 2.1 harness transfers unchanged to SWE-bench Verified. What evolves is general agent competence, not benchmark-specific patches, so it survives changes of task, verifier, and base model. A frozen model need not be a fixed agent: harness selection turns evaluation compute into durable capability.

1 Introduction

DarwinX treats self-evolution as natural selection over harness variants with a frozen model, using preserve-and-extend selection, archived lineages, recombination, and modular evidence. Across four increasingly separated evaluation regimes, it improves benchmark performance and transfers beyond the evolution signal.

  • An agent’s capability depends on its harness—prompts, tools, memory, and control flow—as much as on its underlying model.
  • DarwinX addresses path dependence and cross-task interference by selecting harness variants under a preserve-and-extend contract and recombining complementary lineages.The archive retains alternative branches, preventing complementary improvements from remaining isolated.
  • Selection uses each benchmark’s avg@k solve rate under its own verifier, with no gold solutions or hand-picked winners, while keeping the underlying model frozen.Candidates are harness edits rather than model-weight updates.
  • The proposal interface unifies failure-derived diagnosis, teacher-derived demonstrations, and self-derived rollout contrast into harness edits.
  • 84.7% is the verified Terminal-Bench 2.1 frontier; TerminalWorld reaches 68.3%, WebArena-Infinity rises from 43.5% to 93.0% audit-clean, and the harness transfers unchanged to SWE-bench Verified.The four regimes separate evolution from testing through in-domain evolution, held-out task generalization, synthetic-to-real generalization, and cross-benchmark transfer.

2 DarwinX

DarwinX evolves a frozen model’s harness through population-based selection, preserving solved capabilities while accumulating complementary improvements. Its archive, confirmation rules, and shared evidence interface let alternative lineages contribute edits across prompts, skills, tools, and control flow.

  • Core method: DarwinX repeatedly proposes additive harness edits and retains them only when they extend task coverage without breaking inherited capabilities.The editable harness includes prompts, memory, tools, control flow, and source code, while the base model remains frozen.
  • Population archive: Population selection replaces single-lineage hill climbing by archiving alternative branches, including weaker variants that may supply complementary edits for recombination.Merged children are kept only when they inherit both parents’ wins, so recombination can add coverage rather than trade capabilities.
  • Selection and confirmation: Exploration admits bounded-downside variants, while stricter avg@k re-tests and preservation probes determine which variants may steer future search.This two-stage design is permissive about trying edits but strict about trusting noisy or lucky results.
  • Population archive: Cumulative lineage gain ranks parents, allowing improvements to compound instead of repeatedly restarting from the baseline.Parent selection balances exploiting confirmed high-gain nodes with broadening across the wider population.
  • Learning signals: Any evidence source that explains how a harness should change can provide evolutionary pressure through DarwinX’s shared signal interface.The framework separates the evolution mechanism from a fixed training recipe, allowing diverse learning signals to drive edits.

3 Evaluation Design

DarwinX evaluates variants with repeated binary avg@k measurements that distinguish agent failures from infrastructure failures and require gains to survive preservation checks. A shared failure-mode memory guides harness proposals toward systemic capabilities, allowing a frozen model to improve through its harness alone.

  • Measurement and validation: Binary avg@k is used throughout, with task-budget timeouts counted as real failures and genuine infrastructure failures separated from agent behavior.
  • Measurement and validation: Preservation probes re-sample each lineage’s known solved set before promotion, while deferred candidates receive confirmation at full avg@k.This ensures local wins survive the measurement regime used for final selection rather than reflecting a lucky rollout.
  • Failure-guided evolution: A failure-mode classifier aggregates trial themes into shared memory Kg, updated after every evaluated variant as Kg+1 = Agg(Kg, worked, regressed, themes).
  • Failure-guided evolution: The dominant theme informs both proposer and verifier, steering search toward global capabilities for systemic bottlenecks rather than per-task patches.The paper gives efficient setup for timeout-dominated tasks as an example of such a capability.
  • Generalization: These mechanisms let a frozen model keep gaining capability through its harness alone, motivating tests of generalization beyond the searched tasks.

3 Evaluation Design · 4 In-Domain Test-Time Evolution

DarwinX is evaluated as general harness optimization across benchmarks that increasingly separate evolution signals from tests, with matched-model comparisons isolating harness gains. In-domain Terminal-Bench evolution shows substantial frozen-model improvement, targeted compute allocation, preservation of solved capabilities, and robustness under strict reliability and reward-hacking audits.

  • 3 Evaluation Design: DarwinX evaluates general harness optimization through four benchmarks ordered by increasing separation between evolution signal and test, plus an explanatory ablation.The primary comparison is base Monet versus evolved Monet on the same frozen model, tasks, and verifier.
  • 3.1 Research questions and benchmarks: TerminalWorld evolves on 94 training tasks and evaluates frozen harnesses with single-attempt pass@1 on 41 disjoint held-out tasks.This tests held-out task generalization while keeping the modality and verifier family fixed.
  • 3.1 Research questions and benchmarks: The best Terminal-Bench 2.1 harness transfers unchanged to all 500 SWE-bench Verified issues, using the official test harness for pass@1 grading.SWE-V is treated only as a transfer target because its in-loop signal measured trajectory completion rather than official test resolution.
  • 4 In-Domain Test-Time Evolution: 84.7% on a frozen GPT-5.6 Sol at medium effort reaches the verified Terminal-Bench 2.1 frontier, matching or exceeding Claude Code + Fable 5 at 83.8%.On frozen GPT-5.5, DarwinX reaches 83.2% at high effort, level with Codex + GPT-5.5 at 83.1%.
  • 4 In-Domain Test-Time Evolution: +7.7 points lifts base Monet from 75.5% to 83.2% on frozen GPT-5.5 under the strict leaderboard protocol, while Terminus 2 reaches 78.0%.Because model, tasks, verifier, and effort are controlled, the comparison attributes the gain to the harness.
  • 4.1 The gain is the harness, not compute: 22 vs. 11 turns and 380K vs. 89K tokens occur on six newly solved tasks, while already-solved tasks change only from 12 to 13 turns.The evolved harness concentrates additional verify-and-retry computation where reasoning was previously insufficient rather than increasing effort uniformly.

5 Held-Out Task Generalization

DarwinX generalizes from 94 training tasks to 41 unseen TerminalWorld tasks, reaching 68.3% held-out pass@1 and outperforming off-the-shelf agents. The gain depends on preserving diverse complementary harnesses rather than maximizing the in-loop proxy alone.

  • Held-out generalization: 68.3% held-out pass@1: Monet (DarwinX) resolves 28/41 TerminalWorld tasks, the best result and above every evaluated off-the-shelf agent.The held-out split is disjoint from training, and selection receives no held-out-task reward.
  • Held-out generalization: +7.3 points: the matched Opus 4.8 comparison improves from 25 to 28 tasks, while the matched GPT-5.5 pair moves from 20 to 23.The same three-task absolute improvement appears on both bases.
  • Proxy overfitting: 31.7-point gap: training-subset performance saturates from 0.505 to 1.000, while held-out pass@1 reaches only 68.3%.The proxy-maximizing variant is not the best generalizer.
  • Archive diversity: 28 tasks: the merged harness exceeds four specialists solving 24, 25, 26, and 27 distinct held-out subsets.This supports retaining an archive and selecting among complementary variants rather than following a single proxy-saturated incumbent.
  • Robustness caveats: 56.1% on GPT-5.5 versus 61.0% for Terminus-2: held-out transfer across base models is imperfect.With only 41 tasks, the one-task margin over Claude Code is suggestive rather than statistically decisive.

6 Synthetic-to-Real Generalization

DarwinX evolves Monet on synthetic browser intents, then evaluates the frozen harness on 1,260 disjoint real WebArena-Infinity tasks whose deterministic verifiers and task suites remain unseen during evolution. On the same frozen GPT-5.5 model, audit-clean pass@1 rises from 43.5% to 93.0%, while validity violations sharply decrease.

  • Evolution signal and held-out evaluation: 1,260 disjoint real tasks are evaluated with deterministic verifiers, while evolution uses 300 synthetic intents and LLM-judged avg@3 screening plus avg@5 confirmation.Neither the real tasks nor their verifiers influences selection.
  • Headline result: 93.0% audit-clean pass@1 is achieved by Monet (DarwinX) on the official 1,260-task suite, exceeding GPT-5.5 + Browser Use at 86.1% and Gemini 3 Flash + Browser Use at 69.3%.These comparisons are conservative because external baselines were not re-audited.
  • Gain over base Monet: +49.5 points takes the evolved harness from 43.5% to 93.0% audit-clean pass@1 against base Monet on the same frozen GPT-5.5 model.Every application improves, with the largest gains on Elation prescriptions (+75.0), Gmail (+73.3), and Gmail accounts/contacts (+70.0).
  • Two-stage detector: 99.4% coverage is achieved for evolved trajectories versus 99.0% for base trajectories by the two-stage validity detector and independent judge.The detector labels trajectories Valid, Invalid, Invalid-Attempted, or Review, with unresolved cases sent for human review.
  • Mechanism shift: 1,171/1,260 (93.0%) tasks are solved by the evolved node, versus 548/1,260 (43.5%) for base Monet, with invalid trajectories falling from 293 to 17.Under stricter accounting that drops Review and unaudited successes, the evolved harness still achieves 1,170/1,260 = 92.9%.

7 Cross-Benchmark Transfer

DarwinX transfers beyond terminal benchmarks: its preservation-based framework improves audit-clean browser performance and an unchanged Terminal-Bench harness achieves strong SWE-bench Verified results. Evolution also reduces invalid trajectories while adding contract-oriented browser procedures.

  • Invalid trajectories: 293 to 17 invalid trajectories remained after evolution, with all residual failures consisting of raw-state mutations.The residual mutations were concentrated in a single application.
  • Harness changes: The evolved harness adds four contract-oriented browser skills and changes the system prompt to verify rendered state and persistence.It derives acceptance contracts, inspects client-visible state, and uses app-owned semantic operations instead of abandoning the application surface when stuck.
  • Cross-domain evidence: 49.5 points: audit-clean held-out real-task performance rises from 43.5% to 93.0% when interaction modality and reward source change.The same population and preservation-based selection framework is retained, providing evidence that DarwinX is not specific to terminal benchmarks.
  • Repository-level transfer: 421/500 (84.2%) official pass@1: an unchanged Terminal-Bench 2.1-specialized harness exceeds the 80.8% fix-skill reference on SWE-bench Verified.The transferred agent received no SWE-bench feedback and was evaluated on all 500 issues using the official test harness.

8 Ablation: What Evolution Changes

Evolution primarily adds verification and artifact-contract behaviors to the harness, concentrating gains in tasks with the most headroom while avoiding cluster-level regressions. The attribution is exploratory because the added skills were co-selected rather than independently randomized.

  • Transfer scope: 84.2% on SWE-V is measured by its official test harness, while SWE-V evolution and reverse-direction transfer are excluded because the in-loop signal is unsuitable.Transfer is consequently measured in one direction only.
  • Attribution design: 83.2% avg@5 versus 75.5% for base Monet anchors the Terminal-Bench 2.1 comparison under the strict leaderboard rule.The analysis compares the skill-bundle diff and where benchmark gains land.
  • Harness changes: Seven added harness skills all belong to the verification / artifact-contract family and require explicit acceptance checks or real tool execution before finalizing.None adds domain knowledge.
  • Where gains land: +14.8 points in ML & scientific-computing and +13.8 in data/database are the largest gains, reaching 75% and 98%, respectively.These clusters are where verification against a contract unblocks fragile multi-step work.
  • Where gains land: System administration changes from 92% to 98%, security from 85% to 84% within noise, and no cluster regresses beyond the per-task noise band.The results are consistent with larger gains where the frozen base has more headroom.

9 Discussion and Limitations · 9 Discussion and Limitations · 10 Related Work and Positioning

DarwinX’s evidence supports harness-level gains across modalities and held-out tasks, while its attribution, transfer, diversity, and audit claims remain bounded by experimental and measurement limitations. Relative to prior work, it makes the full agent harness—not merely prompts, workflows, or skills—the learnable component under a frozen model.

  • 9 Discussion and Limitations: The preserve-and-extend rule fixes solved clusters while targeting procedural failures in setup, verification, dependencies, and multi-step tool use.The evidence characterizes the remaining difficulty as procedural rather than knowledge-bound, with improvements attributed to verification and artifact contracts rather than a stronger model.
  • 8.3 Cross-benchmark cross-checks: 49.5 points: WAI audit-clean pass@1 rises while confirmed-invalid rate falls from 23.5% to 1.4%, especially on state-changing applications.The browser results indicate that verification-before-finalization and contract-aware tool use recur across interaction modality and reward source.
  • 8.3 Cross-benchmark cross-checks: 28: the merged TerminalWorld harness solves 28 held-out tasks, versus 24–27 for individual specialists, showing complementary archive behaviors.The authors present this as evidence that diversity, rather than one isolated skill, contributes to population performance.
  • 9 Discussion and Limitations: 75.5 → 83.2%: matched-model TB2.1 improves, while WAI moves from 43.5 →93.0% audit-clean and TerminalWorld reaches 28/41.These are the strongest matched-model and disjoint-held-out results, with GPT-5.5 frozen for TB2.1 and WAI.
  • 9 Discussion and Limitations: 84.2%: a TB2.1-evolved harness reaches this score on SWE-bench Verified, ahead of the fix-skill reference but within a narrow band.Transfer was measured in one direction, and the gain is smaller than the in-domain improvements.
  • 9 Discussion and Limitations: Population search depends on diverse wins, while recombination’s contribution relative to single-lineage mutation still requires controlled ablation.The archive can preserve and combine variants, but inheritance is useful only when complementary improvements are first discovered.
  • 9 Discussion and Limitations: Attribution is system-level rather than operator-level because archive, selection, recombination, inference effort, and model settings are not independently randomized.TerminalWorld’s 41 held-out tasks make one solve worth 2.4 points, and its 25/41 versus 28/41 comparison has McNemar p=0.45, so evidence is suggestive rather than decisive.
  • 10 Related Work and Positioning: DarwinX differs from prompt, workflow, and skill optimizers by making the harness—including tools, control flow, and implementation—the learnable component.The frozen-model setting makes deltas attributable to the harness, while future extensions could couple harness selection with weight updates and preserve harness assets across model generations.

11 Conclusion

DarwinX improves a frozen-model agent by selecting among harness variants, using preservation, archives, and verifier-based evidence to evolve capability without training new weights. Across four regimes, it achieves broad gains while leaving component-level causality and controlled ablations open.

  • Core framework: DarwinX selects over a population of harness variants rather than training new weights, using measured fitness from each benchmark’s verifier without gold solutions or hand-picked winners.Its preserve-and-extend contract admits non-regressing improvements, while archives retain alternative lineages for recombination and a shared interface converts failure-, teacher-, and self-derived evidence into edits.
  • Results: About 17 points average gain across four regimes demonstrates improvement with the model frozen throughout.The regimes progressively separate evolution signal from test performance.
  • Results: 75.5% to 83.2% on Terminal-Bench 2.1 with GPT-5.5, and 84.7% on GPT-5.6 Sol, establishes strong verified performance.The 84.7% result is described as leaderboard-frontier performance.
  • Results: 43.5% to 93.0% audit-clean pass@1 on WebArena-Infinity coincides with invalid trajectories falling from 293 to 17.Capability and compliance improve together under the audit-clean evaluation.
  • Population design and transfer: 24 to 27 held-out TerminalWorld tasks are solved by specialists, while the merged harness beats every specialist.An unchanged Terminal-Bench 2.1 harness also reaches 84.2% on SWE-bench Verified with no in-domain feedback.
  • Limitations: The complete-system experiments do not isolate the archive, parent selector, or merge operator, so recurring behaviors explain gains plausibly rather than causally.Controlled separation under matched budgets is identified as the natural next step.

A Detailed Method Positioning … D.1 Constructing the Synthetic Evolution Set

DarwinX is positioned as a frozen-model harness-evolution method distinguished by how candidates are searched, promoted, and retained, with benchmark-specific protocols separating evolution data from reported evaluation. Its WebArena-Infinity evolution set is synthetic, independently constructed, and disjoint from the reported suite, making the result a transfer across intents, reward source, and partly applications.

  • A Detailed Method Positioning: DarwinX differs from prior self-improving-agent methods primarily in how candidates are searched, promoted, and retained rather than in what they edit.The comparison covers optimizers improving one designated artifact under a fixed outer loop and agents editing their executable scaffold.
  • B Evaluation Details: Every matched comparison freezes the base model, so DarwinX changes only the harness.HarnessX is the exception because it additionally co-trains the model; model co-evolution is outside DarwinX’s scope.
  • C TerminalWorld Robustness Details: 28/41 held-out TerminalWorld tasks remain solved by the DarwinX-evolved harness after infrastructure-error retries, while specialist results improve by +5 to +10 resolved tasks.The DarwinX harness already had 4 errored trials versus 12–17 for specialists, and its held-out margin therefore does not depend on the retry policy.
  • C TerminalWorld Robustness Details: A separately skill-bundled pre-TerminalWorld reference also reaches 28/41, limiting the claim to archive diversity and preservation-based selection recovering a harness that beats every off-the-shelf agent.The result does not establish that TerminalWorld-specific search improves every possible starting harness.
  • D WebArena-Infinity Benchmark Details: WebArena-Infinity evolution never observes a real benchmark task: its 300 intents come from an unchanged document-grounded synthesis pipeline built independently of this work.The synthetic applications and ten reported applications are not the same set, widening separation beyond intent-level disjointness.
  • D.1 Constructing the Synthetic Evolution Set: 11,279 raw intents across 18 applications are filtered to 8,013 retained intents (71%), using quality heuristics, Jaccard near-duplicate removal at 0.70, and TF–IDF uniqueness pruning.Seeds span information retrieval, data entry, data modification, navigation, and multi-step workflow; the benchmark suites are never read and deduplication is intra-pool only.
  • D.1 Constructing the Synthetic Evolution Set: The evolution split contains 300 intents balanced as 25 per application across 12 applications, alongside a disjoint 120-intent synthetic held-out split and a 24-intent smoke set.Synthetic trajectories use an LLM judge because the copied intents have no deterministic verifier, whereas the real suite retains programmatic scoring.
  • D.1 Constructing the Synthetic Evolution Set: The reported result transfers across intents, reward source, and partly applications: Gmail’s 60 of 1,260 real tasks is never seen during evolution, while three synthetic applications are absent from the reported suite.Nine of ten reported applications have synthetic counterparts; Elation patient communication, Figma slides, and Figma text and typography appear only synthetically.

D.2 Anti-cheating Detection Pipeline · D.3 Baseline Provenance and Raw Results

The paper enforces audit validity through provenance rules and a two-stage detector, then distinguishes controlled same-model harness comparisons from public reference points. Audit-clean results remain strong, while the evolved harness adds verification-before-finalization behaviors across task distributions.

  • D.2 Anti-cheating Detection Pipeline: A valid trajectory uses only client-accessible application surfaces, and every scored state change must preserve business logic under the same authority.Accepted causes include UI actions, product APIs, and application-defined semantic mutators.
  • D.2 Anti-cheating Detection Pipeline: Five rejected mechanism classes cover privileged host knowledge, evaluation-plane access, raw-state fabrication, direct database manipulation, and exploits or benchmark modification.The labels describe observed mechanisms rather than agent intent.
  • D.2 Anti-cheating Detection Pipeline: A two-stage detector combines grounded static analysis with independent LLM review of flagged trajectories, assigning Valid, Invalid, Invalid-Attempted, or Review.Static analysis de-obfuscates JavaScript, tracks tainted scored data, and detects host, evaluation-plane, database, and exploit access; disagreements remain for human review.
  • D.3 Baseline Provenance and Raw Results: 23.7-point margin over the strongest public agent is measured conservatively because only the authors’ trajectories undergo validity auditing and flagged successes count as failures.Applying the same audit uniformly to external systems could only widen the margin.
  • D.3 Baseline Provenance and Raw Results: Raw overall scores are 53.0% for base and 94.4% for Monet (DarwinX), falling to 41.9/92.9% when Review and unaudited successes are also removed.The largest audit drops occur on state-change-heavy applications such as Elation-Rx, Gmail, and Xero.
  • D.3 Baseline Provenance and Raw Results: The WAI evolution produced four added skills and one rewritten prompt rule centered on explicit acceptance contracts, rendered-UI and backing-state checks, bounded fallback, and persistence verification.This verification-before-finalization behavior also accounts for Terminal-Bench 2.1 gains despite a different task distribution and signal.

E Outlook and Broader Impact

The outlook extends DarwinX toward co-evolving models and harnesses, reusable harness assets across model generations, auditable and policy-constrained self-improvement, and deployment through refreshed proxy suites. These directions remain partly prospective because the paper does not evaluate relaxed freezing, general compliance probes, or deployed continual evolution.

  • Co-evolving the model and the harness: Co-evolving model weights and harnesses could turn verified newly solved tasks into a curriculum, but moving both layers weakens attribution.The proposed approach would alternate phases with one layer held fixed and re-score the population after weight updates.
  • The harness as an asset across model generations: A harness may become a reusable asset across model generations, but the paper has not measured survival after a base-model swap or warm-start reselection cost.The transfer experiment runs a harness selected on a GPT base unchanged on an Opus base, while the broader measurement remains future work.
  • Auditability is a property of the substrate: Human-readable harness diffs paired with promotion evidence make auditability an oversight primitive that weight-space self-improvement does not provide.This property follows from operating in the harness substrate, with Tables 13 and 14 recording the WAI run’s edits and evidence.
  • Preservation as a policy interface: Preservation can serve as a policy interface by protecting compliance probes alongside capability, although the general version remains untested.WAI is a partial instance in which compliance and capability are scored jointly and both improve.
  • From offline evolution to a deployed loop: Deployment would require refreshed proxy suites because production tasks often lack verifiers, while noise-aware avg@k costs k rollouts per candidate per task.The proposed path is periodic offline evolution against refreshed proxies followed by deployment of the selected harness, rather than online evolution.
Loading 2608.07545v1…