PaperlayerView exact revision on arXiv

Source-linked AI summary

Evolving Agents in the Dark: Retrospective Harness Optimization via Self-Preference

Wenbo Pan, Shujie Liu, Chin-Yew Lin, Jingying Zeng, Xianfeng Tang, Xiangyang Zhou, Yan Lu, Xiaohua Jia

arXiv:2606.05922v2cs.AIcs.CLcs.LG

TL;DR

Existing harness optimization typically needs labeled validation data, which is difficult to obtain in deployment. RHO instead learns from past trajectories and consistently improves performance across three domains, raising SWE-Bench Pro pass rate from 59% to 78% in one round without external grading.

  • Problem

    Existing harness-optimization methods rely on validation sets, but representative labeled data for future tasks is difficult to collect in practical deployment.

  • Method

    RHO selects challenging past tasks, re-solves them repeatedly, uses self-validation and self-consistency, and chooses harness updates through pairwise self-preference.

  • Results

    RHO consistently improves performance across software engineering, technical work, and knowledge work; SWE-Bench Pro pass rate rises from 59% to 78% after one round without validation-set grading.

  • Takeaways & Limitations

    Past trajectories can provide sufficient self-supervised signal for harness improvement, reshaping agent behavior and sustaining higher accuracy during long-horizon sessions.

  • Takeaways & Limitations

    RHO assumes tasks can be cleanly reset and repeatedly attempted, leaving one-shot or irreversible tasks outside its target setting.

Abstract

from arXiv · show

AI agents rely on a harness of skills, tools, and workflows to solve complex problems. Continually improving this harness is essential for adapting to new tasks. However, existing optimization methods typically require ground-truth validation sets, yet such labeled data is difficult to acquire in practical deployment settings. To address this problem, we introduce Retrospective Harness Optimization (RHO), a self-supervised method that optimizes the agent harness using only past trajectories. Specifically, RHO selects a diverse coreset of challenging tasks from past trajectories and re-solves them in parallel. The agent analyzes these rollouts using self-validation and self-consistency, then generates candidate harness updates and selects the most effective one by its own pairwise self-preference. We evaluate RHO across three diverse domains, spanning software engineering, technical work, and knowledge work. Notably, a single optimization round improves the pass rate on SWE-Bench Pro from 59% to 78% without any external grading. Furthermore, our analysis demonstrates that RHO effectively targets prior failure modes. As a result, the optimized harness alters the agent's behavior patterns and sustains higher accuracy during long-horizon sessions.

1 Introduction

Retrospective Harness Optimization (RHO) improves an agent harness from unlabeled past trajectories, avoiding reliance on a labeled validation set. It uses retrospective diagnostics and self-preference, and a single round raises SWE-Bench Pro pass rate from 59% to 78%.

  • Motivation and approach: RHO addresses the difficulty of collecting validation sets that accurately estimate future-task distributions by optimizing the harness from past trajectories without ground-truth labels.The approach is self-supervised and uses the agent’s internal self-preference to guide optimization.
  • RHO pipeline: RHO selects a difficulty-diverse task coreset, re-solves tasks in parallel, diagnoses failures through self-validation and self-consistency, and selects harness updates by pairwise self-preference.The pipeline samples multiple candidate harnesses and retains the one whose rollouts are most preferred over the baseline.
  • Results: 78% pass rate on SWE-Bench Pro follows from a single retrospective optimization round, up from 59%, without grading against a validation set.RHO consistently improves performance across software engineering, technical work, and knowledge work.
  • Analysis: RHO designs skills and tools targeting typical past-task failure modes, reshaping action patterns and sustaining higher accuracy during long-horizon sessions.The paper also quantitatively analyzes the contributions of the retrospective diagnostic signals.
  • Contributions: Retrospective analysis consistently outperforms straightforward experience accumulation and surpasses validation-feedback-driven evolution under a comparable optimization budget.The contributions frame RHO as improving the full harness, including memory, context, skills, and tools, using unlabeled trajectories.

2 Related Work

Related work divides agent improvement into harness optimization, which edits prompts, parameters, or workflow code around a fixed model, and self-improvement from past experience using self-judgment instead of ground-truth labels.

  • Harness optimization: Harness optimization edits prompts, program parameters, or workflow code surrounding a fixed model.This line includes methods based on labeled metrics, such as LLM-as-optimizer search, declarative pipeline compilation, textual-gradient updates, and reflective prompt evolution.
  • Agent self-improvement: Agent self-improvement uses an agent’s self-judgment over past trajectories in place of ground-truth labels.The supplied related-work passage introduces this as a second line of agent improvement.
  • Agent self-improvement: Existing self-improvement systems include Dynamic Cheatsheet, ReasoningBank, and MemMA, which curate strategies, distill reasoning, or repair memory banks using self-generated probes.Dynamic Cheatsheet maintains reusable strategies and code snippets; ReasoningBank distills strategies from self-judged successes and failures; MemMA coordinates memory updates across agents.

3 Problem Setting

The paper models an agent harness as persistent tools, prompts, and skills that produce task-solving trajectories, while harness utility remains latent because future-task validation and success measurement are unavailable. RHO therefore replaces direct utility optimization with a self-preference estimator.

  • Formalization: A harness h is a persistent collection of tools, prompts, and skills used by an agent to solve a task t.The agent executes a loop of reasoning, acting, and observing.
  • Formalization: Each execution produces a trajectory τ = solve(h, t) recording information read, chain of thought, tools used, and final output.Repeated task executions form a trajectory dataset D.
  • Problem: Harness utility U is latent because estimating true utility requires representative future tasks and a mechanism to calculate agent success with a specific harness.The paper states that U cannot be directly observed or optimized.
  • RHO formulation: A single RHO round uses one backbone for difficulty judging, solving, optimizing, and ranking, with no ground-truth label.The operators differ in their inputs while sharing the same backbone.
  • Problem: RHO substitutes latent utility with a self-preference estimator rather than directly optimizing U.The approach is motivated by the inability to directly observe true harness utility.

4 Retrospective Harness Optimization

RHO is a self-supervised method that improves an agent harness using only past trajectories. It selects a representative coreset, derives improvement signals from parallel rollouts, and chooses among candidate harnesses by pairwise self-preference.

  • Pipeline: RHO optimizes the harness using only past trajectories through coreset selection, group rollout, and best-of-N harness proposal.The pipeline first defines the optimization target, then extracts signals from parallel rollouts, and finally retains the most preferred candidate.
  • Coreset selection: The coreset captures challenging and diverse scenarios, avoiding prohibitive optimization over every trajectory and dilution by trivial signals.A determinantal point process selects k trajectories, with θ = 0.7 balancing difficulty and diversity.
  • Group rollout: For each coreset task, G parallel agent solves are compared to identify underperforming runs and formulate contrastive harness-improvement instructions.Self-preference analysis examines dimensions including self-consistency, using contradictions among trajectories to encourage more consistent behavior.
  • Best-of-N harness proposal: Because harness optimization is stochastic, RHO generates N candidate harnesses in parallel and retains the candidate with maximum relative advantage only when Sj > 0.The selection is intended to favor improvements that generalize to future tasks.

5 Experiments and Results

RHO improves performance across SWE-Bench Pro, Terminal-Bench 2, and GAIA-2 without validation-based grading, while adding harness components that address prior failure modes and alter agent behavior. On SWE-Bench Pro, RHO reaches a 0.78 pass rate versus 0.62 for matched-budget Meta-Harness, without held-out labels.

  • Experimental setup: RHO is evaluated using past benchmark trajectories and a held-out test set across SWE-Bench Pro, Terminal-Bench 2, and GAIA-2.The experiments run vanilla Codex on trajectory sets to generate the data required for RHO.
  • Main results: RHO delivers consistent improvements across all three benchmarks, whereas the validation-free baselines do not.The baselines are Dynamic Cheatsheet, ReasoningBank, and Sleep-time Compute.
  • Main results: 19% absolute improvement on SWE-Bench Pro is achieved without validation-based grading.RHO’s advantage is attributed to flexible optimization that can create new tools, skills, and instructions, while baseline gains are smaller and vary across datasets.
  • Harness changes: RHO adds multiple skills and tools across all three benchmarks, often targeting failure modes discovered by analyzing past failures.Examples include locating the Go toolchain outside the default path and removing Python cache directories before producing a final diff.
  • Comparison with validation-feedback optimization: 0.78 pass rate is achieved by RHO on SWE-Bench Pro versus 0.62 for matched-single-round Meta-Harness.Meta-Harness reaches a 0.80 performance ceiling after 10 rounds, requiring roughly three times RHO’s optimization-phase compute and held-out labels.
  • Behavioral changes: RHO sustains longer working sessions and shifts per-step actions toward verification on SWE-Bench Pro and toward execution on Terminal-Bench 2 and GAIA-2.This behavior shift is summarized in Figure 4.

6 Discussion

The discussion examines how RHO’s coreset selection and retrospective signals shape optimization outcomes. DPP selection balances task coverage and difficulty, while best-of-N proposal selection improves robustness despite only moderate candidate variance.

  • Behavioral mechanisms: The discussion proposes tool-call frequency and cumulative success by agent steps as behavioral lenses for understanding how newly created skills and tools improve future task performance.Token-level attribution over long reasoning chains is identified as a finer-grained complement to action-level statistics.
  • Coreset selection: DPP coreset selection balances coverage and difficulty, whereas difficulty-only or diversity-only selection underperforms, with only their combination reaching the top gain.Difficulty-only selection clusters samples in a narrow task-distribution region because the language model favors certain task types.
  • Coreset selection: Difficulty-only selection clusters chosen tasks narrowly and fails to include other task types in the coreset, producing no meaningful performance improvement.The clustering is attributed to the language model judging certain task types as inherently more difficult.
  • Candidate consistency: Best-of-N selection deploys a harness that scores higher than the worst candidate across all three benchmarks, even though the preferred harness is not always the highest-scoring candidate.The generated harnesses show only moderate variance, and even the lowest-scoring candidate meaningfully improves performance over baseline.
  • Retrospective analysis: RHO’s retrospective analysis compares self-validation and self-consistency with raw-trajectory optimization and separately tests the contribution of each diagnostic signal.The ablation removes self-validation and self-consistency independently and introduces a raw trajectory baseline that bypasses separate ranking analysis.

7 Conclusion

RHO reframes harness improvement as a retrospective, self-supervised process driven by an agent’s past experience rather than external ground-truth feedback. Re-solving past tasks and comparing outcomes exposes harness failures and enables consistent held-out gains across software engineering, technical work, and knowledge work.

  • 7 Conclusion: RHO enables agents to improve their harnesses from their own past experience without externally provided ground-truth feedback.The method treats harness improvement as a retrospective process that agents can run autonomously.
  • 7 Conclusion: Re-solving past tasks and comparing outcomes reveals where the harness fails and what changes could fix it.The agent’s existing trajectories provide the signal needed for this improvement loop.
  • 7 Conclusion: Across software engineering, technical work, and knowledge work, RHO produces consistent held-out gains and reshapes agent behavior.The conclusion presents this self-supervised loop as a step toward agents that continue improving from deployment experience.

Limitations

RHO improves an agent harness from past trajectories without external grading, but this self-supervised setup imposes environmental constraints. Its repeated group rollouts require cleanly resettable environments and exclude one-shot or irreversible tasks.

  • Feedback assumptions: RHO operates without ground-truth feedback or external grading, which limits the settings it can target.The method relies on the agent’s own past trajectories rather than externally validated feedback.
  • Environment assumptions: Repeated group rollouts assume environments reset cleanly and tolerate multiple attempts, excluding one-shot or irreversible tasks.Each coreset task is replayed several times during optimization.

Ethics Statement

RHO can persistently alter agent behavior based on model-generated judgments, potentially amplifying mistaken preferences, unsafe procedures, or biased rules. Safe deployment therefore requires auditing, human approval, and domain-specific checks.

  • Ethics Statement: RHO may amplify mistaken preferences, unsafe procedures, or biased behavioral rules when its evaluator prefers them.The risk arises because RHO modifies persistent agent behavior from model-generated judgments.
  • Ethics Statement: Deployments should keep full audit logs, require human approval for sensitive harness edits, and apply domain-specific safety checks before high-impact use.These safeguards should precede applying accepted harnesses to high-impact tasks.

Reproducibility Statement … C Hyperparameters and Infrastructure

The appendices provide an audit-ready account of RHO’s implementation, prompts, related-work comparison, and experimental configuration. They specify the operators, rollout analyses, optimization and ranking procedures, reproducibility records, and hyperparameter rationale.

  • Reproducibility Statement: Every run persists prompts, completions, trajectories, diagnoses, candidate harnesses, diffs, configs, scores, metadata, and held-out reports.The draft’s numbers are direct reads from recorded run reports, with supporting details distributed across Appendices A–G.
  • A Comparison with Related Work: Table 5 compares methods by edited harness surface, feedback signal, cost regime, and whether they satisfy RHO’s three criteria.The criteria are label-free optimization, full-harness editing, and a single offline retrospective pass.
  • B Prompts: RHO’s five agent operators are instantiated by verbatim prompts for solve, difficulty judging, diagnosis, optimize, and rank.Placeholders such as {name} are filled at call time using the values specified for each prompt block.
  • B.1 Solve: The solve wrapper materializes each harness and task in a fresh workspace and is reused unchanged for baseline and candidate rollouts.This keeps solve’s varying input limited to the harness while task-shaping guidance remains in the harness directory.
  • B.2 Coreset Selection (Difficulty Judge): The difficulty judge outputs r_i ∈[0, 10] and an abstract fingerprint from a scrubbed, head/tail-truncated prior trajectory digest.The normalized difficulty r_i/10 enters the DPP kernel, while the fingerprint becomes a unit vector x_i defining S = XX⊤.
  • B.3 Diagnosis: Diagnosis evaluates three trajectories, analyzes failure modes and inconsistencies, proposes one high-level harness direction, and assigns severity from 0.0 to 1.0.Severity acts as a soft attention weight for subsequent optimization rather than ground truth.
  • B.4 Optimization: Optimization gives independent candidates the harness and severity-sorted diagnoses, requiring cross-task pattern matching and discouraging task-specific hardcoded fixes.Recurring high-severity failure modes and inconsistency causes receive priority, while low-severity tasks generally receive less influence.
  • B.5 Pairwise Ranking: Pairwise ranking returns an integer in [-10, 10], averages oriented scores across the coreset, and accepts a candidate only when S_j > 0.The strict-positive gate keeps the current harness when self-preference is nonpositive.

D Pipeline Implementation Details … D.7 Persistence

The appendix specifies the executable mechanics of RHO, including filesystem-based harness handling, isolated operator roles, pairwise ranking, candidate editing and acceptance, and persistent audit records.

  • D Pipeline Implementation Details: RHO’s implementation appendix fixes the mechanics that make the main-text equations and operator signatures executable.These design choices govern RHO’s behavior without changing the equations or operator signatures.
  • D.1 Harness Representation and Mounting: A harness is a schema-free directory whose prose, scripts, and configuration files are mounted into the agent’s working directory.solve mounts the harness read-only by convention, whereas optimize receives a writable fresh copy; identical outputs are treated as unchanged candidates.
  • D.2 Role Separation and Workspace Isolation: All operators use the same backbone, with role separation enforced by fresh workspaces containing only the inputs each operator should see.solve, diagnosis, optimize, and rank therefore differ through workspace contents rather than backbone changes.
  • D.3 Order, Parsing, and Failure Handling in Pairwise Ranking: rank compares each candidate with a fixed baseline trajectory, presents the candidate first, negates the parsed scalar, and returns zero on parse or execution failure.The judge must emit one integer in [−10, 10] with a one-sentence rationale, and calls are not retried.
  • D.4 Diagnosis vs. Ranking Inputs: Diagnosis consumes all G rollouts for one task to produce one severity-weighted instruction, while candidate ranking remains strictly pairwise against a constant reference.G > 1 sharpens diagnosis through cross-trajectory inconsistency but does not vote on candidate winners.
  • D.5 Optimizer Action Space: The optimizer edits the harness directly as a filesystem, may add, remove, or modify any file, and returns the resulting directory state rather than a parsed diff.Diagnosis instructions are sorted by severity, but severity acts as a soft attention weight in [0, 1].
  • D.6 Acceptance Gate and No-Ops: An update is accepted only when the best surviving candidate has a strictly positive mean pairwise score; otherwise RHO leaves the harness unchanged.Optimizer failures, timeouts, identical harnesses, and candidates with S_j ≤0 are dropped before the gate.
  • D.7 Persistence: Persistence records operator trajectories, diagnosis instructions, candidate harnesses and diffs, pairwise scores, mean scores, and acceptance flags for later audit, re-grading, and ablation.Each trajectory includes its full event stream, final message, workspace diff, and wall-clock time.

E Dataset Specifications … E.3 GAIA-2

The appendix specifies reproducible data sources, partitions, solve interfaces, and grading procedures for SWE-Bench Pro, Terminal-Bench 2, and GAIA-2. GAIA-2 additionally evaluates behavior under partial observation and asynchronous events.

  • E Dataset Specifications: The benchmark partitions and graders are reproducible from a clean checkout using pinned upstream commits.The appendix documents upstream inputs, splits, solve-time observations, and grading procedures.
  • E.1 SWE-Bench Pro: SWE-Bench Pro evaluates long-horizon software-engineering tasks that pass only when multi-file patches satisfy the upstream test set.The benchmark uses the test split and official evaluator scripts with per-instance Docker images pinned to a fixed commit.
  • E.1 SWE-Bench Pro: 100 training instances and 100 held-out test instances are selected by seeded SHA-256 hash ordering, with remaining rows unused.Each task provides an issue prompt and a fresh repository clone at its base commit.
  • E.1 SWE-Bench Pro: A SWE-Bench Pro task passes iff every FAIL_TO_PASS and PASS_TO_PASS test resolves correctly after patch application in the official Docker image.Binary hunks and generated paths are stripped before scoring, and each task has a one-hour Docker wall-clock budget.
  • E.2 Terminal-Bench 2: Terminal-Bench 2 contains executable command-line tasks where failure is a missed reward, making state inspection and command chaining central harness concerns.The upstream fixed-commit release contains 89 tasks.
  • E.3 GAIA-2: GAIA-2 differs from the coding benchmarks because its environment evolves independently of the agent, requiring behavior under partial observation and asynchronous events.Its harness must encode responses to changing environment state rather than only stable build conventions.
  • E.3 GAIA-2: 200 GAIA-2 mini-configuration validation scenarios are split into 100 training and 100 held-out scenarios by the same seeded hash-ordering scheme.Each scenario includes an asynchronous event stream and an upstream write-action verifier, and evaluation reports the held-out slice.

F Baseline Implementations … F.4 Meta-Harness (Lee et al., 2026)

The appendix compares four harness-optimization baselines under a shared Codex gpt-5.5 evaluation setup, distinguishing them by what their offline phases persist and how solvers consume it. Meta-Harness uniquely incorporates labeled validation feedback, whereas RHO uses the same empty harness, coreset, held-out split, and grader without that feedback.

  • F Baseline Implementations: All baselines use the same Codex gpt-5.5 backbone at high reasoning effort, while trajectory-only methods share a 10-task, 3-candidate coreset budget.The documented variation is limited to offline persistence and solver consumption.
  • F.1 Dynamic Cheatsheet (Suzgun et al., 2025): Dynamic Cheatsheet maintains a running markdown file of reusable facts and procedures, rewritten sequentially from selected task transcripts and read during normal harness lookup.Each memory item includes a description, worked example, and usage count.
  • F.2 ReasoningBank (Ouyang et al., 2025): ReasoningBank stores reusable reasoning patterns in an external JSONL bank with precomputed embeddings and retrieves them by similarity at inference time.For each selected training task, a judge first assesses trajectory success before extraction.
  • F.3 Sleep-time Compute (Lin et al., 2025): Sleep-time Compute converts selected past traces into bounded markdown memory blocks inside the harness and prepends compact notes to the agent’s inference context.Its offline agent edits memories using structured insert, replace, rethink, and finish tools following Letta semantics.
  • F.4 Meta-Harness (Lee et al., 2026): Meta-Harness maintains candidate harness directories and search history, alternating a proposer that emits rewrites with an evaluator that scores them on a labeled validation set.The search history records per-candidate validation pass rates.
  • F.4 Meta-Harness (Lee et al., 2026): RHO and Meta-Harness start from the same empty harness, use the same selected-task coreset, and share the held-out split and grader; Meta-Harness additionally consumes validation grades.Table 2 measures this validation-feedback axis.

G Optimization-Phase Compute Cost … H.3 GAIA-2

RHO’s optimization-phase compute is comparable to trajectory-only baselines at matched budgets, while its optimized artifacts encode benchmark-specific verification and validation workflows. Meta-Harness incurs a substantially larger offline footprint, and wall-clock performance depends on parallelization and dataset-specific invocation cost.

  • G Optimization-Phase Compute Cost: At matched coreset budget (k = 10, N = 3), RHO is in the same order of magnitude as Sleep-time Compute and Dynamic Cheatsheet but costs more than ReasoningBank.The matched-budget Meta-Harness configuration is the only baseline with a qualitatively different cost profile.
  • G.1 Accounting Scope: RHO’s offline accounting separates Codex agent invocations by role and auxiliary LLM calls, with persisted trajectories recording solve, diagnose, optimize, and rank metadata.Table 7 counts ROLLOUT, AFTER, RANK, and TEST invocations for SWE-Bench Pro with k = 10 and Mtest = 100.
  • G.2 Per-Method Decomposition: ReasoningBank uses one solve per training task without proposing a candidate harness, whereas Dynamic Cheatsheet and Sleep-time Compute use G = 3 rollouts before optimization.ReasoningBank adds k = 10 Codex invocations and 2k auxiliary LLM calls for judging and memory extraction.
  • G.3 Wall-Clock Time: Trajectory-only methods achieve 5–10× wall-clock speedup through parallelization, while ReasoningBank reaches 3.5–4.5× because its offline phase is sequential.Σcodex measures serial-equivalent invocation time, whereas end-to-end time uses 10 concurrent agent calls and graders.
  • G.3 Wall-Clock Time: ±4%: At matched coreset and N = 3, RHO’s cost is within ±4% of Sleep-time Compute on every dataset, while invocation costs vary by more than 2× across datasets.GAIA-2 averages around 150 s per invocation, Terminal-Bench 2 around 300 s, and SWE-Bench Pro around 400 s.
  • G.4 Matched-Budget Meta-Harness: 1,210: Meta-Harness’s upstream default would require approximately 1,210 SWE-Bench Pro agent invocations, nearly twelve times RHO’s 103, while its 10-round matched configuration uses 320.The 320-invocation configuration is 3.1× RHO’s optimization-phase count and excludes the shared held-out evaluation.
  • G.5 Coreset Selection Cost (Shared): Coreset selection is a shared one-time cost: SWE-Bench Pro uses 100 auxiliary LLM calls plus one batched embedding call, then reuses the selected k = 10 task ids across methods.This cost is paid once per dataset and is not attributed to any individual method.
  • H.1 SWE-Bench Pro: The optimized harness artifacts are complete benchmark-specific directories containing Markdown instructions, skills, and executable tools; SWE-Bench Pro emphasizes contract-led repair and direct smoke verification.Its repair-verify tool discovers non-PATH toolchains, runs cheap syntax or package checks, reports generated artifacts, and accepts an extra task-specific command.
Loading 2606.05922v2…