HarnessCompass: Guiding Automatic Harness Evolution toward Generalizable and Effective Agent Harnesses

• Manual harness tuning requires developers to inspect trajectories, diagnose failures, and revise harnesses, a loop that struggles to keep pace with advancing base models.Source: Introduction · p. 1
• Existing methods can overfit search tasks, rely only on trajectories that obscure harness friction, and jointly edit components whose changes interfere.Source: Introduction · p. 1
• HarnessCompass constrains edits to task-agnostic changes, grounds first-person agent feedback in trajectories, and optimizes components separately before integration.These steps aim to prevent task-specific shortcuts, verify candidate feedback, and reduce cross-component interference while retaining synergy.Source: Introduction · p. 2
• Pass@1 rose from 54% to 66% with GPT-5.4 on SWE-bench Verified after five evolution iterations, surpassing AHE in final Pass@1 and evolutionary efficiency.Source: Introduction · p. 2
• The evolved harness transfers to held-out tasks and other models, improving generalization over prior automatic harness evolution methods.Source: Introduction · p. 2

Related Works

• Harnesses shape agent perception, tool interaction, and failure recovery; SWE-agent and OpenHands illustrate their importance in software-engineering agents (Zhang et al. 2026b; Yang et al. 2024b; Wang et al. 2025).Reasoning-and-acting established an interaction loop for LLM agents (Yao et al. 2023), and scaffold changes can yield gains comparable to upgrading the underlying model.Source: Related Works · p. 2
• Darwin Gödel Machine and Live-SWE-agent evolve agent implementations, while existing systems lack explicit mechanisms to ensure effectiveness and generality (Zhang et al. 2025a; Xia et al. 2025).A survey organizes the emerging field around what, when, and how agents evolve (Gao et al. 2025).Source: Related Works · p. 3
• Automatic optimization spans prompts, tools, agent structures, and workflows, but harness evolution can overfit search tasks and generalize poorly to disjoint evaluation sets.Examples include OPRO and TextGrad for prompts (Yang et al. 2024a; Yuksekgonul et al. 2024), Self-Supervised Prompt Optimization (Xiang et al. 2025), ToolMaker and ToolLibGen (Wölflein et al. 2025; Yue et al. 2025), ADAS and AFlow (Hu, Lu, and Clune 2025; Zhang et al. 2025b), and end-to-end optimization (Evers-Hood et al. 2025). Automatic harness evolution jointly optimizes the full harness through agent–environment interactions (Lin et al. 2026), while evaluation identifies overfitting and poor generalization (Wang et al. 2026).Source: Related Works · p. 3

HarnessCompass

• HarnessCompass disciplines an iterative evaluate–analyze–improve loop with task-agnostic constraints, first-person evidence, and component-wise optimization.These principles target overfitting to search tasks, limited trajectory-only evidence, and interference across components.Source: HarnessCompass · p. 3

Preliminaries

• The base model stays fixed, and evolution starts from a minimal harness so later components are introduced and measured through rollouts.The harness has seven component types: structural components that execute code and control interactions, and guidance components that shape behavior.Source: Preliminaries · pp. 3–4

Constrained Evolution

• The generalization gate admits reusable decision criteria rather than task-specific rules and requires capability edits in executable components and guidance edits in prompts or memory.It rejects references to specific task instances, test functions, private symbols, or task-exclusive token branches, and prevents guidance from being encoded as executable logic.Source: Constrained Evolution · p. 4

Proactive Feedback

• First-person blind and hindsight reports are reconciled, retained only when trajectory-supported, and grouped into component-level evidence for the meta agent.Groups receive confidence scores based on report agreement, affected-task count, and consistency between report types; edits still require trajectory support and are reverted if they do not help.Source: Proactive Feedback · p. 4

Component-wise Optimization

• Structural and guidance variants are evolved on separate tracks, then R3 merges independently beneficial loser edits into the higher-Pass@1 winner while removing conflicts, regressions, and redundancy.R3 uses Revision, Recombination, and Refinement, with the winner taking precedence in file-level conflicts.Source: Component-wise Optimization · p. 5

Setup

• The evaluation uses 50 SWE-Bench Verified tasks for evolution and 450 unseen tasks for held-out evaluation, with GPT-5.4 in non-thinking mode and H0 and AHE as baselines.Pass@1 is averaged across rollouts and tasks, with each task weighted equally in Total; Turns counts evolution iterations.Source: Setup · p. 5

Main Results

• 66.0% Pass@1 on the evolution sample beats AHE by 3 points; Held-Out and Total Pass@1 are 60.4% vs 54.7% and 61.0% vs 55.5%, with 5 vs 20 Turns.The seed harness scored 54.0% on the evolution sample and 51.8% Total; both automatic methods used the same seed and base model.Source: Table 1 · p. 5, Main Results · pp. 5–6

Evolution Efficiency

• HarnessCompass climbs faster and reaches a higher performance level than AHE, which improves slowly and levels off below it after about 20 iterations.The generalization gate keeps edits transferable and avoids give-back from reverting task-specific fixes, while R3 folds useful edits from the losing track into each round rather than discarding half its exploration.Source: Figure 2 · p. 6, Evolution Efficiency · p. 7

Ablation Study

• 62.0% Sample Pass@1 after adding the generalization gate, up from 54.0%, accompanies a Held-Out increase from 51.6% to 58.4%.Proactive feedback then raises Sample Pass@1 to 66.0%, but Held-Out falls to 55.8% and evolution takes 12 Turns; R3 integration maintains 66.0% Sample, restores Held-Out to 60.4%, and cuts Turns to 5. The gate promotes transfer, feedback expands the optimization signal, and R3 reconciles complementary tracks while removing redundant or non-transferable edits.Source: Table 2 · p. 6, Ablation Study · pp. 6–7

Generalization to Held-Out Tasks and Base Models

• 73.8% Total Pass@1 on Claude-Sonnet-4.6, up from 70.0%, shows that the frozen GPT-5.4-evolved harness transfers to another base model.On Claude-Sonnet-4.6, Sample Pass@1 rises from 68.0% to 76.0% and Held-Out from 70.2% to 73.6%, without further evolution.Source: Table 3 · p. 6, Generalization to Held-Out Tasks and… · p. 7

Where the Gains Land Across Repositories

• 35.2% to 58.0% on Sphinx-doc and 59.3% to 70.1% on django are examples of the evolved harness’s broadly distributed GPT-5.4 gains.Gains are largest where the seed is weakest and tasks are harness-sensitive; scikit-learn barely moves from its already-strong 73.4%. A similar pattern appears on Claude-Sonnet-4.6, including astropy rising from 45.5% to 63.6%.Source: Figure 3 · p. 7, Where the Gains Land Across Repositories · p. 7

Conclusion

• HarnessCompass uses task-agnostic constraints, first-person agent feedback, and isolated component tracks to reduce overfitting and cross-component interference while enriching trajectory evidence.The framework optimizes harness components separately before consolidating them into a unified harness.Source: Conclusion · p. 7
• HarnessCompass achieves stronger effectiveness and generalization than prior methods, suggesting disciplined evolution is as important as search itself.These findings motivate future work on harnesses that are both effective and broadly generalizable.Source: Conclusion · p. 8

Appendix · Prompts

• The prompt set includes the seed harness H0 and a generalization gate for constrained evolution.These prompts are organized according to the design principles they implement.Source: Prompts · p. 8
• A four-stage first-person reflection pipeline implements proactive feedback.Source: Prompts · p. 8
• Track constraints and the R3 integrator implement component-wise optimization.Source: Prompts · p. 8
• The appendix also includes trajectory-analysis prompts inherited from the standard evaluate–analyze–improve loop.Source: Prompts · p. 8
• All prompts are reproduced verbatim, with Jinja placeholders and Python format fields instantiated by the harness at runtime.Examples include {{ ws }}, {{ iteration }}, {pre_json}, and {loser_diff}.Source: Prompts · p. 8
• Non-ASCII characters in the reproduced prompts are transliterated for typesetting.Source: Prompts · p. 8

Seed Harness H0

• H0 contains only a shell tool and system prompt; every component in later harnesses is introduced by the evolution loop.The seed has no middleware, skills, sub-agents, or long-term memory.Source: Seed Harness H0 · p. 8
• The system prompt frames the agent as non-interactive, directing it to use the shell to inspect, edit, build, and test the repository without asking questions.It also prefers short replies, requests brief warnings before destructive commands, and directs long-running processes to run in the background.Source: Seed Harness H0 · p. 8

Generalization Gate

• Every harness edit must encode a reusable principle with an applicability criterion for unseen tasks, never task-specific answers, identifiers, symbols, or keyword-matching rules.The gate applies across harness surfaces, and any rule that cannot be made task-agnostic must be discarded.Source: Generalization Gate · p. 8
• Structural changes to middleware, tools, and sub-agents must perform a deterministic action triggered by an execution event and return a result, not merely inject advisory text.Advice belongs in system prompts or long-term memory, where it is directly governed by the gate.Source: Generalization Gate · p. 9
• Memory entries must pair a transferable principle with an applicability criterion; merge duplicate or contradictory lessons into a deciding criterion and prune task-specific or non-generalizing entries.Memory is always-on global context, so the gate forbids routing task-specific fixes into short-term memory at runtime.Source: Generalization Gate · p. 9

Pipeline overview

• For each failed task, three calls use the trajectory-generating base model and a fourth uses an analyzer; all share the trajectory, but blind reflection withholds the verdict.The analyzer serves as an external checker.Source: Proactive Feedback · p. 9

Reflection

• Blind reflection records harness friction before the verdict; hindsight reflection adds the verifier result and failure attribution, allowing feedback to be discarded when failure was not harness-caused.Both reports solicit existing-feature friction and missing-capability requests.Source: Proactive Feedback · pp. 9–10

Reconciliation

• Reconciliation merges the blind and hindsight reports into items, recording agreement for use in the aggregated pattern’s confidence score.Source: Proactive Feedback · p. 10

Grounding and routing

• Grounding requires trace evidence for existing-feature complaints, keeps missing-capability requests only when the trajectory shows a plausible gap, and routes each survivor to structural or guidance tracks.Structural fixes must be concrete code actions; prose-only advice is classified as guidance.Source: Proactive Feedback · pp. 10–11

Aggregation and use

• Fixed aggregation and confidence scoring require no additional model calls; the resulting evidence enters the meta-agent query, is filtered by the active variant’s track, and remains subordinate to the trajectory.Evolution should extract cross-task mechanisms rather than copy task-specific friction.Source: Proactive Feedback · p. 11

Proactive Feedback

• When acting on a pattern, cite its feedback_ids in the manifest’s agent_feedback_evidence.The passage also lists candidate_fix and affected_tasks alongside feedback_ids.Source: Proactive Feedback · p. 12
• Set feedback_confidence to the pattern’s confidence.Source: Proactive Feedback · p. 12

Component-wise optimization

• Each round runs two meta-agents on the same evidence with different constraints that partition seven component types, producing disjoint edits.The variants therefore cannot interfere within a round.Source: Component-wise Optimization · p. 12
• Structural changes target middleware, tools, or sub-agents and must be generic mechanisms that deterministically change agent behavior, not advisory text.The structural constraint rejects task-specific code and guidance disguised as structural changes.Source: Component-wise Optimization · p. 12
• Guidance changes target prompts, skills, tool descriptions, or memory and must state cross-task principles with explicit applicability criteria.Contradictory per-task memory rules should be merged or pruned, and lessons that cannot be made task-agnostic discarded.Source: Component-wise Optimization · p. 12
• Integration uses the higher-scoring variant as its base and reviews loser changes individually, keeping independent positives while dropping regressions, task-specific hacks, or uncertain changes.Multiple genuinely useful changes may be retained; a direct conflict is resolved in favor of the winner.Source: Component-wise Optimization · pp. 12–13
• The integration pass removes confirmed redundancy by keeping deterministic behavior over advisory guidance, while preserving capabilities not shown to be duplicates.An advisory item is removed only when the integrator can identify the component that already enforces the same behavior.Source: Component-wise Optimization · p. 13

Change Manifest

• Each meta-agent edit is recorded in a machine-readable manifest that later rounds use to attribute score changes to edits and determine rollbacks.The final two fields link each edit to the first-person evidence motivating it, enabling measurement of how often proactive feedback leads to committed changes.Source: Change Manifest · p. 13
• The schema records each change’s type, rationale, files, failure pattern, predicted fixes, risk tasks, chosen constraint level, and agent-feedback evidence and confidence.Source: Change Manifest · p. 14
• When a change addresses self-reported friction, its feedback IDs and confidence are recorded as supplemental candidate evidence; trajectory evidence and the falsification loop remain required.Changes not driven by agent feedback leave the feedback fields empty.Source: Change Manifest · p. 14

Per-Repository Results

• Small-repository Pass@1 changes have coarse resolution, so the authors caution against over-interpreting them.Resolution is 6.25 points for psf, 25 for mwaskom, and 50 for pallets; mwaskom’s +25.0 GPT-5.4 gain is one additional success among four trials.Source: Per-Repository Results · p. 14
• +3.8 points in aggregate when the GPT-5.4-evolved harness transfers to Claude-Sonnet-4.6, although repository-level effects vary.The harness gains 18.1 points on astropy but drops 10.5 on pytest and 4.5 on sphinx; the authors treat aggregate transfer as the primary result.Source: Per-Repository Results · p. 14, Table 4 · p. 15

Rewards Overfitting

• Artifact inspection offers a qualitative mechanistic check on whether evolution learned reusable engineering competence or merely memorized tasks from its evolution set.The authors caution that memory inspection alone neither proves generalization nor shows that every gated rule will transfer.Source: Rewards Overfitting · p. 14, Interpretation · p. 15

Two kinds of accumulated memory

• Ungated memory stores task-specific recipes, whereas gated memory records general principles with applicability conditions for decisions on unseen codebases.Examples of gated principles include tracing a failing value to the function that emits it and changing a shared helper only when multiple consumers share a broken contract.Source: Two kinds of accumulated memory · pp. 14–15

Per-Repository Results

• +9.2 points under GPT-5.4: HarnessCompass improves eight of twelve repositories, with gains concentrated in larger repositories.django gains 10.8 points and sympy gains 4.7 points; together they account for 61% of benchmark tasks and contribute substantially to the overall gain.Source: Per-Repository Results · p. 14, Table 4 · p. 15

Interpretation

• The Generalization Gate blocks task identifiers, task-specific symbols, and file paths while requiring applicability conditions, replacing patch catalogues with method-level knowledge.This rules out direct task-specific recipes as the explanation for ungated gains, but does not itself prove that gated rules generalize.Source: Interpretation · p. 15

Future Work

• Future work could evolve rubrics to detect intermediate agent behaviors linked to downstream failures and corrective mechanisms to prevent error propagation.Proposed mechanisms include trajectory-level filtering, targeted feedback generation, or reward modification, potentially improving supervision signals for reinforcement learning.Source: Future Work · p. 16