PaperlayerView exact revision on arXiv

Source-linked AI summary

HarnessPro: Recursive Harness Self-Improvement with Pareto Optimization for Performance-Cost Trade-offs

Zhuoshi Pan, ZhiYuan Feng, Yukuo Cen, Honglin Lin, Junru Lu, Zhongxia Chen, Shuo Yang, H. Vicky Zhao, Jie Tang

arXiv:2610.26770v2cs.LGcs.CL

TL;DR

Recursive harness self-improvement can raise performance through greater exploration and verification, but often at higher token and latency costs. HarnessPro evolves harnesses using trajectory feedback and generated tests, then selects options for different budgets. Across its reported evaluations, it improves performance while reducing execution cost, with latency benefits also reported.

  • Problem

    Recursive harness improvements often pursue performance gains through more exploration and verification, increasing token consumption and latency despite the practical importance of execution cost.

    Source: §1 · p. 1

  • Method

    HarnessPro uses specialized agents to diagnose trajectories, plan and implement harness updates, and evaluate candidates with fixed LLM-generated tests before selecting budget-oriented harnesses.

    Source: §1 · p. 1

  • Results

    Across three benchmarks, the heavy mode achieves the highest pass rate at lower cost than RHO; on Terminal-Bench 2 with GPT-5.5 and Codex, it exceeds RHO by 4.5 percentage points with 33.9% lower mean cost.

    Source: §1 · p. 2, §5.2 · p. 7

  • Takeaways & Limitations

    HarnessPro offers lite, medium, and heavy alternatives for different deployment budgets rather than returning a single harness.

    Source: §6 · p. 9

    Latency is treated as an auxiliary metric because model response times fluctuate with server load and network conditions.

    Source: §3 · p. 3

Abstract

from arXiv · show

Harness development plays a crucial role in building capable AI agents. Recent approaches improve harnesses through recursive self-improvement (RSI), but their performance gains often come with higher execution costs. We introduce HarnessPro, a label-free method for evolving harnesses to improve task performance while reducing execution cost. In this setting, candidate harnesses are evaluated using LLM-generated tests without access to ground-truth solutions or official tests. Specialized agents diagnose execution trajectories, plan updates with different performance-cost priorities, and implement them as candidate harnesses. Rather than returning a single harness, HarnessPro uses normalized weighted squared-distance selection to provide cost-focused (lite), balanced (medium), and performance-focused (heavy) harnesses for different deployment budgets. Selected harnesses enter the next iteration, where execution feedback guides further updates. Our experiments cover Terminal-Bench 2, SWE-Bench Pro, and ProgramBench, using GPT-5.5 with Codex and GLM-5.2 with Claude Code. Across these benchmarks using GPT-5.5 with Codex, the heavy mode of HarnessPro achieves an average pass-rate gain of 2.0 percentage points and an average per-task cost reduction of 26.1% relative to RHO. This advantage also holds when using GLM-5.2 with Claude Code. Code is available at https://github.com/pzs19/HarnessPro

1 INTRODUCTION

HarnessPro targets the performance–cost trade-off in recursive harness self-improvement with a label-free, multi-agent approach that evolves harnesses from execution feedback. It offers budget-oriented harness choices and reports improved performance with lower cost across evaluated settings.

1 INTRODUCTION

  • HarnessPro uses specialized agents to diagnose execution trajectories, plan updates for different performance–cost priorities, and implement executable harness candidates evaluated with fixed LLM-generated tests.The tests require neither ground-truth solutions nor official tests; candidates that underperform the initial harness or are Pareto-dominated are discarded.Source: §1 · p. 1
  • Normalized weighted squared-distance selection retains cost-focused lite, balanced medium, and performance-focused heavy harnesses for different deployment budgets.Selected harnesses enter subsequent epochs, where updates continue under different performance–cost priorities.Source: §1 · pp. 1–2
  • On Terminal-Bench 2 with GPT-5.5 and Codex, heavy exceeds RHO by 4.5 percentage points in pass rate while reducing mean cost per task by 33.9%.For cost-sensitive deployments, medium delivers performance comparable to RHO at substantially lower cost.Source: §1 · p. 2
  • Evolved harnesses dynamically allocate reasoning effort and tool budgets, use runtime hooks to steer execution toward convergence, and streamline prompts while retaining needed context and verification.The analysis attributes reduced avoidable overhead to these mechanisms without relying simply on context compression or increased reasoning effort.Source: §1 · p. 2
  • HARNESSPRO analyzes execution trajectories and recursively improves harnesses; across three benchmarks, it improves performance while reducing cost and latency, yielding a series of harnesses with different performance-cost trade-offs.Source: §1 · p. 2

2 RELATED WORK

Prior work improves agents through search, harness modification, component optimization, and inference-time budget allocation. HarnessPro instead evolves the whole harness from past trajectories and returns multiple performance–cost options.

2 RELATED WORK

  • RHO guides harness evolution toward task success using LLM comparisons of execution trajectories without ground-truth labels, whereas HarnessPro optimizes task success and monetary cost and returns trade-off tiers.The tiers are lite, medium, and heavy.Source: §2 · p. 2
  • Related approaches optimize particular components or allocate inference-time budgets, while HarnessPro uses past trajectories to evolve instructions, skills, tools, hooks, and verification rules across the whole harness.Examples include skill and memory optimization, reusable strategies, and budget-aware allocation of tokens or tool calls.Source: §2 · p. 2

3 PROBLEM SETTING

The problem setting treats a harness as the runtime layer governing model interactions and evaluates it by task success, execution cost, and latency. HarnessPro seeks performance–cost Pareto trade-offs in a broad, discrete search space where reducing computation indiscriminately can harm performance.

3 PROBLEM SETTING

  • A harness governs a model’s interactions and execution, shaping the information and actions available and influencing both task success and resource use.The runtime layer includes prompts, tools, skills, context and memory mechanisms, hooks, and verification policies.Source: §3 · pp. 2–3
  • Pass rate is the fraction of test tasks whose final artifacts pass all official tests, with training, validation, and test sets used for updates, selection, and final evaluation, respectively.Source: §3 · p. 3
  • Execution cost aggregates uncached input, cached input, and output token usage across model calls, while latency measures wall-clock time from agent execution start to completion.Latency excludes grading time and is treated as auxiliary because response times fluctuate with server and network conditions.Source: §3 · p. 3
  • The objective is to find Pareto-frontier harnesses and select lite, medium, and heavy operating points, but the search spans a vast discrete space and extra reasoning or verification can raise cost.Indiscriminate reductions to exploration or verification may hurt performance, so the setting calls for distinguishing necessary computation from avoidable overhead.Source: §3 · p. 3

4 HARNESSPRO

HARNESSPRO recursively evolves executable harnesses from rollout evidence, using specialized agents to diagnose and implement updates. Fixed generated tests and validation-based filtering and selection guide the next iteration and preserve distinct performance-cost priorities.

4 HARNESSPRO

  • HARNESSPRO analyzes coreset trajectories to identify failures, unproductive exploration, and redundant computation before iteratively modifying executable harnesses.The process begins with a seed harness and carries selected candidates into the next epoch.Source: §4 · p. 3, §4.1 · p. 4

4.1 OVERVIEW

  • The label-free process generates tests from task descriptions and initial-harness trajectories, then uses those fixed tests to compare candidates on validation tasks.Validation and test sets remain hidden from the planner, and final evaluation uses the held-out test set after evolution.Source: §4.1 · p. 4, §4.3 · p. 6

4.2 TRAJECTORY ANALYSIS AND HARNESS EVOLUTION

  • The trajectory analyzer jointly examines repeated rollouts, while the planner proposes cost-focused, balanced, and performance-focused plans that the editor implements as executable candidates.The planner can compare candidates’ code differences to identify effective mechanisms and remaining bottlenecks.Source: §4.2 · pp. 4–6

4.3 EVALUATION AND CANDIDATE SELECTION

  • A test-synthesis agent creates evaluation tests without ground-truth solutions, official tests, or grader feedback; generated-test failure recall rises from 41.7% to 83.3% versus direct LLM judgments.The same generated tests remain fixed throughout evolution for fair candidate comparison.Source: §4.3 · p. 6
  • Candidates below the initial pass-rate floor or Pareto-dominated are removed, then normalized weighted squared-distance selection retains representatives for lite, medium, and heavy modes.Filtering and selection use validation metrics, with at most three harnesses retained for the next epoch.Source: §4.3 · p. 6

5 EXPERIMENTS

HARNESSPRO is evaluated across three benchmarks and two model–agent pairings, with heavy mode outperforming RHO at lower cost and recursive evolution improving retained performance–cost trade-offs. The experiments also examine evolved harness mechanisms and ablations of routing, budget hooks, and reasoning effort.

5.1 EXPERIMENTAL SETUP

  • The evaluation uses 10-task training sets, held-out test sets, and shared data splits and grading settings across methods on three benchmarks and two model–agent pairings.HARNESSPRO uses three epochs on Terminal-Bench 2 and SWE-Bench Pro and four on ProgramBench; metrics are reported over repeated runs.Source: §5.1 · p. 7
  • Against Vanilla, Dynamic Cheatsheet, ReasoningBank, Sleep-time Compute, and RHO, the study also compares RHO variants that add cost-aware instructions or repeated iterations.RHO+Iteration runs for the same number of epochs as HARNESSPRO, selecting each next starting harness with RHO’s best-of-N procedure.Source: §5.1 · p. 7

5.2 MAIN RESULTS

  • With GLM-5.2 and Claude Code, heavy surpasses RHO on all three benchmarks at lower cost, while reducing SWE-Bench Pro mean latency from 520 to 412 seconds.The medium and lite modes outperform Vanilla in pass rate, often at substantially lower cost.Source: §5.2 · p. 7
  • On Terminal-Bench 2, heavy reaches 80.8% pass rate, 4.5 percentage points above RHO with 33.9% lower mean cost; on ProgramBench it reaches 69.6% versus 68.8% at 17.8% lower cost.For Terminal-Bench 2, heavy’s 80.8% pass rate is the mean across three runs; on ProgramBench, mean cost is $2.95 versus RHO’s $3.59.Source: §5.2 · p. 7, Table 2 · p. 8

5.3 ANALYSIS OF EVOLVED HARNESSES

  • Evolved harnesses select task-specific reasoning profiles and tool-call budgets, enforce budgets with runtime reminders, and reduce redundant context through output compression and streamlined instructions.On Terminal-Bench 2, one medium profile triggers reminders after 14 and 26 tool calls and blocks further calls after 44.Source: §5.3 · p. 8

5.4 EVOLUTION DYNAMICS

  • Across benchmarks, recursive evolution advances the retained performance–cost frontier; on SWE-Bench Pro, medium improves both metrics only by epoch 3, reaching 79.0% pass rate at $1.26 per task.Figure 4 connects retained harnesses across epochs; intermediate test-set metrics were for visualization and not selection.Source: §5.4 · p. 8, Figure 4 · p. 9

5.5 ABLATION STUDY

  • The ablation table covers RHO variants with dynamic routing, budget hooks, or fixed low reasoning effort, but the supplied passage does not report their outcomes.Dynamic routing selects reasoning effort from task descriptions; budget hooks append reminders to tool results.Source: Table 3 · p. 9

6 CONCLUSION

The conclusion presents HARNESSPRO as a label-free, multi-agent approach that recursively improves harnesses while offering alternatives for different deployment budgets. Experiments across three benchmarks report performance and cost improvements, and the authors identify evolving mechanisms and recursive progress as contributors.

6 CONCLUSION

  • HARNESSPRO coordinates trajectory diagnosis, harness modification, and LLM-generated-test evaluation to evolve harnesses for improved performance and lower execution cost.It offers cost-focused lite, balanced medium, and performance-focused heavy alternatives.Source: §6 · p. 9
  • The authors report that generative AI assisted software implementation and manuscript drafting, while LLMs perform trajectory analysis, harness updates, and test synthesis within the method.They state that AI-assisted code and text were manually reviewed and reported results checked.Source: AI USE STATEMENT · p. 10
  • The reproducibility statement points to the evolution procedure, optimization objective, experimental settings, appendix instructions, pricing assumptions, and frozen harness artifacts.The procedure and objective are described in Sections 3 and 4, with pseudocode in Algorithm 1.Source: REPRODUCIBILITY STATEMENT · p. 10

A.1 DPP-BASED CORESET SELECTION

The coreset procedure selects a fixed set of challenging and diverse training tasks before harness evolution. It uses trajectory-derived failure fingerprints and a greedy determinant-based approximation rather than sampling from the k-DPP.

A.1 DPP-BASED CORESET SELECTION

  • For each candidate task, a language-model judge scores difficulty and assigns an abstract failure-mode fingerprint from its initial-harness trajectory.The fingerprint omits task-specific names so similarity reflects shared challenges rather than superficial project details.Source: §A.1 · p. 13
  • The method combines cosine similarity between normalized fingerprint embeddings with difficulty weights in a DPP kernel, whose subset determinant penalizes redundant fingerprints.The weights favor challenging tasks; ε = 0.1 and θ = 0.7 follow RHO.Source: §A.1 · p. 13
  • Instead of sampling from the k-DPP, the procedure greedily adds tasks to approximate its most probable subset, stops at 10 tasks, and keeps that coreset fixed across epochs.Neither official test results nor validation outcomes enter coreset selection.Source: §A.1 · p. 13

A.2 ADDITIONAL IMPLEMENTATION DETAILS

The appendix reports implementation settings and gives distinct search guidance for cost-focused, balanced, and performance-focused evolution. The instructions emphasize evidence-based changes, preserving correctness, and evaluating quality–cost trade-offs rather than treating predicted benefits as established results.

A.2 ADDITIONAL IMPLEMENTATION DETAILS

  • Model reasoning defaults and mode weights are fixed across epochs, while generation settings, token limits, solver budgets, and grading limits vary by model or benchmark.Lite, medium, and heavy use weights (1, 2), (1, 1), and (2, 1), respectively; wall-clock limits are distinct from harness tool-call budgets.Source: §A.2 · p. 13
  • The appendix specifies task resource settings and token-price assumptions used to compute execution cost from token usage.Terminal-Bench 2 and ProgramBench allocate 8 CPUs and 32 GiB per solver; SWE-Bench Pro has no explicit per-task CPU or memory limit.Source: §A.2 · p. 14, Table 4 · p. 14

B EVOLUTION INSTRUCTIONS

  • Cost-focused instructions prioritize average cost reduction while preserving correctness, using trajectories to identify redundant work and accounting for unsuccessful attempts and regression risks.Possible search directions include task-adaptive reasoning, economical context and tool feedback, and improved convergence or recovery; these are optional rather than prescribed designs.Source: B EVOLUTION INSTRUCTIONS · p. 15
  • Balanced instructions treat reliability and average execution cost jointly, favoring removal of low-value work while preserving or strengthening reasoning and validation that matter.They caution that additional computation does not necessarily improve quality and that combining changes can add overhead and complexity.Source: B EVOLUTION INSTRUCTIONS · p. 15
  • Performance-focused instructions target consequential failure patterns and invest effort where it may improve success, rather than increasing computation uniformly across tasks.They identify task-adaptive effort, selective investigation or verification, and context or recovery management as optional hypotheses to derive from evidence.Source: B EVOLUTION INSTRUCTIONS · p. 16

C.1 TRAJECTORY DIAGNOSES

Trajectory diagnoses distinguish incomplete Caffe–CIFAR-10 runs from successful but costly Mailman runs. They separate shared bottlenecks from run-specific detours and potential risks, identifying avoidable work without treating unobserved failures as observed.

Case 1 Caffe–CIFAR-10

  • No Caffe–CIFAR-10 run reached conversion or training; high build parallelism caused OOM failures, and long builds and downloads produced no final model or log.The analysis separates a shared build bottleneck from run-specific stalls and a potential downstream risk.Source: §C.1 · p. 17
  • The Caffe diagnosis flags a likely OpenCV-related data-preparation blocker for run 0, but explicitly notes that the trace did not show this failure.Run 0 configured Caffe without OpenCV even though the stock CIFAR pipeline’s mean-generation step requires it.Source: §C.1 · p. 17
  • Run 0 recovered from an incorrect source tag by trying the nonexistent 1.0.0 branch, inspecting tags, and using 1.0; this added commands before the build but caused no lasting quality loss.Source: §C.1 · p. 17

Case 2 Mailman

  • All three Mailman runs passed generated tests, while the diagnosis identifies broad exploration, repeated evaluations, and run-specific detours that increased cost without changing the final outcome.The broad exploration contributed to trace totals of 1,058,946, 1,763,487, and 2,517,861 input tokens in runs 0, 1, and 2, respectively.Source: §C.1 · p. 18
  • Mailman’s manual smoke tests provided useful delivery checks, but later and earlier evaluations repeated covered behavior and added little end-to-end coverage.A run-2 smoke test temporarily caused confirmation mail to bounce; the agent diagnosed and corrected it without changing the final outcome.Source: §C.1 · p. 18
  • Mailman delivery passed for valid Unix users, but disabling SMTP-time recipient validation could allow invalid local recipients through SMTP and fail later; that risk was not observed in tests.Two failed evaluations and map debugging preceded the fallback.Source: §C.1 · p. 18

C.2 EVOLUTION PLANS

The evolution-plan examples show distinct performance–cost priorities, while the test-synthesizer configuration specifies how candidate tests are authored and checked without access to official tests. The appendix also identifies the files and runtime setup for evolved harness artifacts.

C.2 EVOLUTION PLANS

  • The medium plan proposes task-adaptive routing and lightweight execution support, with a router that selects reasoning effort and helpers that preserve logs while limiting noisy output.The plan notes possible under-allocation on deceptively simple tasks and routing overhead that can be net-negative on very small tasks.Source: §C.2 · p. 19
  • The heavy plan emphasizes public-contract and semantic artifact verification, bounded exploration, and cheaper command output, while acknowledging that stronger task-local checks can add execution cost.Its guidance-only changes add neither a second solver session nor a uniform extra reasoning pass, but depend on solver compliance and logged command output.Source: §C.2 · p. 20
  • The lite plan revises a router observed to choose high or xhigh for most sampled tasks, favoring medium or low where appropriate while retaining escalation for uncertainty.The proposed savings remain a hypothesis until measured on task rollouts, and shifting effort downward could hurt quality when public descriptions hide difficulty or verification is weak.Source: §C.2 · p. 21

C.3 TEST SYNTHESIZER CONFIGURATION

  • The test-synthesizer workflow uses public task evidence to author executable tests, validates them against a frozen Empty product, and reviews the suite without official tests, graders, or labels.The Empty product is an implementation and execution-diagnostic context, not an answer key; valid product failures are preserved rather than repaired to make Empty pass.Source: §C.3 · pp. 22–23
  • Test design maps documented public requirements to meaningful behavioral checks, chooses the strongest justified oracle, and avoids deriving expected values or paths from Empty.The configuration specifies ten distinct executable cases and requires a completed validation run without infrastructure error after executable edits.Source: §C.3 · p. 22

D EVOLVED HARNESS ARTIFACTS

  • The evolved-artifacts appendix presents medium harness configurations for three benchmarks and shows Terminal-Bench 2 lite and heavy differences; its runtime uses a separate GPT-5.5 call to select an execution profile from the public task description.The evaluator reads policy settings and generates a Codex configuration that loads the instruction file and registers the hook.Source: D EVOLVED HARNESS ARTIFACTS · p. 23

D.1 TERMINAL-BENCH 2

The Terminal-Bench 2 medium harness runtime files include a profile rubric, policy, hook, and model instructions. A separate GPT-5.5 call selects an execution profile using the public task description.

D.1 TERMINAL-BENCH 2

  • A separate GPT-5.5 call uses the rubric to select an execution profile from the public task description, while the evaluator reads policy settings and generates a Codex configuration that loads instructions and registers the hook.The listed runtime files are model instructions, profile rubric, policy, and hook.Source: §D.1 · p. 24

D.1.1 MEDIUM MODE

The medium configuration routes tasks to execution profiles using observable task characteristics and expected success per dollar. Runtime policies enforce profile-specific tool budgets; the lite configuration shortens those budgets and adjusts reasoning settings.

D.1.1 MEDIUM MODE

  • The rubric selects one of five profiles—compact, lean, standard, deep, or research—from task characteristics, optimizing expected success per dollar without relying on remembered outcomes.Unknown oracle quality defaults to standard; the rubric applies its routing gates in order.Source: §D.1.1 · pp. 25, 27
  • Profile choice depends on whether task evidence supports compact work, bounded lean work, a standard default, deep iteration with a strong local oracle, or multi-source research.The deep gate requires an explicitly described or constructible strong local oracle plus a qualifying task shape; lean requires bounded work, few uncertainties, a credible plan, direct checks, and about eight or fewer evidence-producing calls.Source: §D.1.1 · pp. 25–26
  • The medium policy assigns profile-specific reasoning effort and staged tool thresholds, while runtime hooks enforce tool-call limits and issue phase reminders.Thresholds count attempted calls, and the policy's history token limit applies per output rather than to the model context window.Source: §D.1.1 · p. 29

D.1.2 LITE MODE

  • Lite mode retains the shared rubric while adjusting tool budgets and reasoning settings from medium; listed finalization thresholds fall from 14 to 8, 22 to 16, 44 to 38, 54 to 48, and 46 to 40, with deep and research effort medium.Source: §D.1.2 · p. 30

D.1.3 HEAVY MODE

This section details heavy-mode harness configurations across benchmarks, including expanded reasoning and tool budgets, task-adaptive routing for SWE-Bench Pro, and fixed-profile reconstruction for ProgramBench. It also describes ProgramBench safeguards that separate the black-box reference from the submitted implementation and use tool-count thresholds to guide exploration and finalization.

D.1.3 HEAVY MODE

  • Heavy mode keeps medium’s instructions and hooks but changes routing and policy profiles, with larger tool budgets and reasoning effort.Profiles range from compact low effort with a 14-call stop threshold to research high effort with up to about 60 calls for source reconstruction.Source: §D.1.3 · p. 31

D.2 SWE-BENCH PRO

  • SWE-Bench Pro medium routes tasks among five reasoning and tool-budget profiles, then extracts the repository contract, makes focused repairs, and verifies task-shaped behavior.The rubric limits lean routing to narrowly scoped repairs with a focused check that covers the success path and stated edge cases.Source: §D.2 · pp. 32, 34

D.3 PROGRAMBENCH

  • ProgramBench medium uses a fixed GPT-5.5 high-effort profile to build an independent program from bundled documentation and public-interface probes, then validate and rebuild it offline.The workflow forbids source retrieval, internet access, decompilation, tracing, and reuse of the reference executable.Source: §D.3 · pp. 37–38
  • The candidate must be built independently without overwriting or submitting the black-box reference, and the final deliverable must be rebuilt from validated source after comparison.The reference is a development oracle for permitted comparisons; reserve time for an offline rebuild and end-to-end validation.Source: §D.3 · pp. 38–39
  • The policy sets a 3000-token history-retention limit, 4500 maximum characters, high reasoning effort, and economy, strict, and finalize thresholds of 36, 60, and 76 tools.The history limit is per-output retention, not model context size; strict reminders recur every eight tools.Source: §D.3 · p. 39
  • The hook applies economy, convergence, and finalization stages using tool-count thresholds, then denies further tool calls after finalization plus the larger of eight or the reminder interval.Its guidance shifts from covering documented semantic families, to hypothesis-driven probes, to implementation completion and final checks.Source: §D.3 · p. 41

Related papers

Loading 2610.26770v2…