PaperlayerView exact revision on arXiv

Source-linked AI summary

HARBOR: Automated Harness Optimization

Biswa Sengupta, Jinhua Wang

arXiv:2604.20938v1cs.LGcs.AI

TL;DR

Long-horizon agents rely heavily on harness code, yet teams tune its features by hand. HARBOR formalizes harness configuration as constrained noisy Bayesian optimization and, in one 89-task run, matched the manual peak while beating the all-on stack by five passes.

  • Problem

    Harnesses dominate much of long-horizon agents' implementation, while teams tune their configurations by hand through feature additions and ablations.

    Source: §I · p. 1

  • Method

    HARBOR formalizes harness configuration as constrained noisy optimization over a structured, cost-heterogeneous space, using Bayesian optimization, a per-task-difficulty meta-controller, and a cold-start-aware evaluator.

    Source: §I · p. 2

  • Results

    At the deployment-cost tier matching the manual D stack, HARBOR's two-flag configuration scored 17/89, five passes above D's 12/89, after about 3.5 full-suite-equivalent search units.

    Source: §VIII · p. 10

  • Takeaways & Limitations

    The end-to-end run provides first-order evidence that automated harness configuration search is tractable with the assembled optimization ingredients.

    Source: §IX · p. 12

    The evidence is scoped to a single agent-benchmark setting, with no claim of transfer across agent families.

    Source: §IX · p. 12

Abstract

from arXiv · show

Long-horizon language-model agents are dominated, in lines of code and in operational complexity, not by their underlying model but by the harness that wraps it: context compaction, tool caching, semantic memory, trajectory reuse, speculative tool prediction, and the glue that binds the model to a sandboxed execution environment. We argue that harness design is a first-class machine-learning problem and that automated configuration search dominates manual stacking once the flag space exceeds a handful of bits. We defend this claim in two steps. First, we formalize automated harness optimization as constrained noisy Bayesian optimization over a mixed-variable, cost-heterogeneous configuration space with cold-start-corrected rewards and a posterior chance-constrained safety check, and give a reference solver, HARBOR (Harness Axis-aligned Regularized Bayesian Optimization Routine), built from a block-additive SAAS surrogate, multi-fidelity cost-aware acquisition, and TuRBO trust regions. Second, we instantiate the problem in a flag-gated harness over a production coding agent and report a controlled four-round manual-tuning case study against a fixed task suite and an end-to-end HARBOR run. The formulation itself is task-class agnostic: the configuration space, reward correction, acquisition, and safety check apply to any agent harness with a bounded flag space and a reproducible task suite.

I. INTRODUCTION

The introduction frames harness configuration, rather than individual features, as a central optimization problem for long-horizon agents. It motivates automation with a negative manual-tuning case and presents HARBOR as a constrained-search approach.

I. INTRODUCTION

  • Only round B beat the all-flags-off baseline, scoring 17/89 versus 15/89; rounds C and D fell to 13/89 and 12/89.The reported explanation includes harmful self-evaluation, cache corruption from compressed observations, cold-start effects, and silent integration bugs.Source: §I · p. 1
  • The manual rounds exposed failures including low-capability self-evaluators, ACON summaries stored instead of raw tool results, cold-start effects, and silent integration bugs.The PASTE next-tool predictor was never wired into the runner, and reflection memory was not propagated between containers.Source: §I · pp. 1–2
  • Its schematic scores flag-space and fidelity inputs with a block-additive SAAS surrogate, selects batches through cost-aware acquisition, and outputs a safety-checked Pareto front of pass rate and cost.Runtime telemetry also feeds silent-flag auto-exclusion, which prunes dimensions from the surrogate.Source: Figure 1 · p. 2
  • The paper characterizes the harness as a high-dimensional flag-valued space with six non-trivial interaction dimensions, where feature value is class-specific and stacking can have negative returns.It argues that class-level composition and component-internal tuning matter more than simply adding published techniques.Source: §I · p. 2
  • HARBOR formalizes harness optimization as constrained noisy optimization over a structured, cost-heterogeneous configuration space, with a cold-start-aware evaluator and per-task-difficulty meta-controller.Its reference algorithm combines Bayesian optimization over flag bundles with these evaluator and controller components.Source: §I · p. 2
  • The empirical claims are explicitly in-the-small, while the formulation is presented as applicable to any agent-orchestration harness with a bounded flag space and reproducible task suite.Coding is described as a measurement convenience rather than a restriction of the formulation.Source: §I · p. 2

II. RELATED WORK

Related work spans harness architecture, harness-code evolution, agent benchmarks, individual harness components, and expensive-black-box optimization. The paper positions its contribution as configuration search rather than harness-code generation.

II. RELATED WORK

  • The paper treats a subset of the high-level harness feature-flag space over a single API gateway as an optimization problem, not an architectural one.Prior work describes harness layers including externalized memory, context compaction, prompt-prefix caching, and AI gateways.Source: §II · p. 2
  • Meta-Harness evolves harness code, whereas Automated Harness Optimization searches harness configurations with cost-aware Bayesian optimization; the approaches are complementary.The passage reports Meta-Harness's +7.7 absolute accuracy points on LawBench with 4× fewer context tokens against hand-engineered baselines.Source: §II · pp. 2–3
  • Terminal-Bench and SWE-bench Verified are described as standard long-horizon coding-agent targets, and task independence can amplify cold-start penalties for cross-task state.The paper says its B result is consistent with the lower end of Terminus-KIRA's reported 10-percentage-point harness-only improvements.Source: §II · p. 3
  • The evaluated harness features draw on literature about caching, reflection, memory trees, speculative execution, observation compression, and tiered embedding compression; the paper observes that stacking them is not additive.Binary embedding and Matryoshka-style two-stage retrieval are mentioned as unimplemented next steps.Source: §II · p. 3
  • The paper draws on Bayesian optimization, multi-fidelity methods, and BO-plus-bandit approaches for expensive evaluations, and says it applies them to harness configuration rather than harness code.It describes this as, to its knowledge, the first such application for LLM-agent harness configuration.Source: §II · p. 3

III. THE CODEX-PY HARNESS

This section describes codex-py as a layered production harness whose runtime and infrastructure support tool execution, context management, memory, resilience, and opt-in extensions.

II. THE CODEX-PY HARNESS

  • The codex-py port is organized into six architectural layers and includes 34 tool handlers plus a three-stage approval pipeline.The layers are agent, security, protocol, integration, presentation, and infrastructure.Source: §III · p. 3
  • A stateful runtime turn loop owns dispatch, tool invocation, and exit conditions, while file and shell handlers declare read/write and concurrency behavior.The runtime emits typed events consumed by presentation and telemetry layers.Source: §III · p. 3
  • Context management combines microcompaction, token-threshold snipping, and LLM-driven summarization, with several flag-gated policies layered on top.The substrate also includes post-compact file restoration and ghost snapshots.Source: §III · p. 3
  • Cross-session memory uses a compressed index of task-embedding and hint pairs alongside a trajectory-based experience library, retrieving few-shot hints for similar new tasks.The passage describes both backends as persisting compressed embeddings across sessions.Source: §III · p. 3
  • A categorized-retry resilience family is complemented by a runtime self-evaluation gate that runs a checklist-driven review before submission.The gate is described as being in the style of Terminus-KIRA.Source: §III · p. 3
  • Enhancement flags resolve by runtime override, build-time compile flag, environment variable, then default, keeping additions opt-in so all flags off reproduces the Rust-parity baseline.The paper identifies nine relevant default-off flags for its experiments.Source: §III · p. 3

IV. THE HARNESS FEATURE LAYER OVER COD E X-P Y

The feature layer packages flag-gated caching, memory, compression, trajectory reuse, prediction, and published techniques over codex-py. The section also describes telemetry that preserves requested-versus-realized flag state when extensions fail.

IV. THE HARNESS FEATURE LAYER OVER COD E X-P Y

  • TURBOQUANT is an online, data-oblivious two-stage vector quantizer used by compression-heavy extensions, with a rotation-based per-coordinate quantizer followed by a one-bit QJL residual transform.The passage says it matches the information-theoretic distortion-rate lower bound within an approximately 2.7 constant.Source: §IV · p. 4
  • PolarCache combines exact and compressed-inner-product similarity keys, file modification times, and intent-canonical hashes; PolarMemoryIndex stores task embeddings and hints with a memory-type facet.The memory facet separates Reflexion-style reflections from trajectory hints.Source: §IV · p. 4
  • The tiered compressor keeps recent turns at full precision and compresses the tail, while trajectory replay injects successful or failing trajectories as few-shot hints when similarity exceeds a threshold.PASTE predicts and speculatively fires tools using trajectory-derived tuples and a probability threshold, with an online path based on dictionary lookup.Source: §IV · p. 4
  • Three later-round flags add a critic-style self-evaluation call, compressed summaries in place of large tool observations, and failure reflections stored in cross-session memory.These are described as self-evaluation, ACON observation compression, and Reflexion-style reflection, respectively.Source: §IV · p. 4
  • A session-scoped coordinator wires extensions into runtime pathways and demotes lazy-init failures to disabled while preserving the requested flag in telemetry.The requested-versus-realized record can expose discrepancies such as a write-side counter without a matching consumer-side counter.Source: §IV · p. 4

V. CASE STUDY: MANUAL TUNING, A–D

Across four manual tuning rounds, only B beat the all-flags-off baseline; later additions reduced full-suite performance. The regressions exposed model-dependent behavior, feature interactions, and integration bugs that the manual loop missed.

A →B: the one clean win

  • 17/89 tasks passed in round B, versus 15/89 for the all-flags-off baseline; B was the manual search’s peak.B combined four changes, with zero errors introduced.Source: §V · p. 5

B →C: the literature bites back

  • On gpt-5.4-nano, C’s self-evaluation gate usually changed passing answers into failing ones, unlike the reported net win with Claude Sonnet.The cache canonicalization rules produced zero canonical hits, while prompt-prefix-first saved tokens without a separately credited pass-rate gain.Source: §V · p. 5

C →D: regressions

  • 13/89 tasks passed in C and 12/89 in D, both below B’s 17/89; D also lost nine tasks relative to B.D gained two tasks and regained three lost by C, but ended five tasks below B.Source: §V · p. 5

C →D: the bundle makes it worse

  • In D, ACON compressed tool results before caching, so tasks needing line-level detail reasoned from truncated outputs and failed.The Reflexion retrieval and PASTE prediction counters were both identically zero because their integrations did not work across containers or invoke the runner entry point.Source: §V · p. 5

What the four rounds teach us

  • The Oracle union passed 81/89 tasks versus B’s 17/89, while the manual loop missed integration bugs and small per-flag effects required full-suite sweeps to clear noise.The authors identify non-additive feature interactions, cold-start bounds on cross-session features, and model-specific transfer as lessons from the rounds.Source: §V · p. 5

VI. AUTOMATED HARNESS OPTIMIZATION

The paper formalizes harness tuning as constrained, noisy optimization over mixed, cost-heterogeneous flag configurations. HARBOR combines cold-start correction, a sparse block-additive surrogate, multi-fidelity cost-aware acquisition, and posterior safety constraints.

Configuration space

  • Harness configurations span about 40 Boolean, numeric, or discrete dimensions, but silent-flag detection typically reduces the effective search to about five.The nominal configuration set exceeds 240; the SAAS prior is designed to tolerate the ambient dimension while optimization work is dominated by effective dimensions.Source: §VI · p. 5

Problem formulation

  • Automated Harness Optimization constrains deployed per-task cost and regression from a measured all-flags-off baseline, while a separate search budget caps cumulative optimization evaluations.The deployment budget applies to the returned configuration at inference time; the search budget limits the outer-loop history.Source: §VI · pp. 5–6

Property I: Cold-start-corrected reward

  • Cold-start correction estimates fully warmed pass rate from observed pass rate, the baseline, and warm-up fraction, with uncertainty inflated when warm-up evidence is weak.The system warm fraction uses the minimum across warm-state flags; at zero estimated warm-up, the observation receives zero-precision evidence.Source: §VI · p. 7

Property II: Mixed-input, block-additive surrogate

  • The block-additive mixed-input surrogate uses sparse per-layer effects and pairwise cross-block interactions, while excluding interactions of three or more blocks.The cross-block scale is shrunk toward zero unless evidence supports interactions, such as ACON↔cache; its components are only coarsely interpretable under arbitrary input measures.Source: §VI · p. 7

Property III: Multi-fidelity, cost-aware, safe acquisition

  • Multi-fidelity acquisition uses stratified task subsets and cost-aware qNEHVI to trade information per dollar across fidelities; its cost ratio is an engineering heuristic, not a theorem.The acquisition compares candidates against a fidelity-consistent Pareto frontier and applies a posterior chance constraint to each batch candidate to limit regression from baseline.Source: §VI · pp. 7–8

Reference algorithm: HARBOR

  • HARBOR combines the sparse block-additive surrogate, cold-start correction, multi-fidelity constrained acquisition, and TuRBO local moves, with meta-learned warm starts when prior logs exist.Fig. 2 labels GP posterior mean and standard deviation, fidelity task subsets, per-block kernel scales, and observed block projections.Source: §VI · p. 8, Figure 2 · p. 8

VII. EXPERIMENTAL SETUP

The evaluation uses gpt-5.4-nano on the fixed 89-task Terminal-Bench 2 suite with the same task specifications across tuning rounds. The experimental HARBOR implementation approximates parts of the reference Bayesian machinery and uses telemetry checks before full-suite runs.

Benchmark and models

  • Terminal-Bench 2 contains 89 Docker-isolated terminal tasks, and a task passes when its verifier awards reward 1.0.All headline numbers use the full suite; gpt-5.4-nano is served through the OpenAI Responses API.Source: §VII · p. 9

Metric and telemetry

  • The reported metric is pass rate, supplemented by session-averaged telemetry counters for caching, compression, memory, and prediction behavior.These structured counters also feed the warm-start estimator.Source: §VII · p. 9

Implementation

  • The HARBOR artifact uses weighted ridge regression, a linear tensor-product kernel on Boolean inputs, and per-candidate Monte Carlo EHVI instead of the reference solver’s corresponding Bayesian components.The implementation retains the listed correctness-critical elements of the reference specification, including the chance constraint and warm-start variance inflation.Source: §VII · p. 9

Preflight smoke verification

  • Before an m=89 evaluation, a short smoke test checks whether enabled flags emit telemetry, accounting for each flag’s firing threshold.Configurations with any remaining RED flag are vetoed; a zero counter is only downgraded when the smoke trajectory was too short to reach the flag’s threshold.Source: §VII · p. 9

Silent-flag auto-exclusion

  • After evaluations, HARBOR prunes flags whose warm counters remain below threshold across the required silent-flag observations.This auto-exclusion targets flags that are empirically silent on the current model–benchmark pair.Source: §VII · p. 9

VIII. RESULTS

HARBOR’s two-flag configuration matched the manual B peak and improved on the D all-on stack, while the manual rounds exposed regressions and integration bugs. The results also show a substantial gap between the best single configuration and the union of tasks passed across configurations.

Baselines

  • The 81/89 Oracle is the union of tasks passed by any tested configuration; B’s 17 passes captured only 21% of that union.Both models used the same codex-py runtime, tool handlers, and sandbox; the A–D rounds used gpt-5.4-nano because its API quota supported the full sweep.Source: §VIII · p. 9

Manual tuning and feature deltas

  • Only baseline→B improved (15→17); B→C fell 4 and B→D fell 5, while B’s cache-key normalization delivered the cache win and D’s ACON gate stored compressed summaries that harmed detail-dependent tasks.C canonicalization added zero hits; the B–D gap sat at the edge of the ±3-pass Wilson noise floor but was repeatable across independent sweeps.Source: §VIII · pp. 9–10

Wall-clock budget accounting

  • At concurrency 4, an 89-task sweep took 121.9 minutes for the baseline and 139.8 minutes for D-all-on; API token quota, not wall-clock, constrained iteration speed.The sweeps were dominated by per-task container setup and package installation rather than model latency.Source: §VIII · p. 10

HARBOR validation

  • 17/89 (19.10%) matched the manual B peak and exceeded D’s 12/89 (13.48%) by 5 passes (+5.6pp), using two flags rather than D’s eight.HARBOR ran on the same 89-task suite with gpt-5.4-nano; the effective configuration reduced to compression-only because memory retrieval was zero.Source: §VIII · p. 10
  • Across five HARBOR-proposed evaluations, the mean was 16.6/89 versus D’s 12/89 and the 15/89 all-flags-off baseline; two runs had silent flags, making the 19/89 peak a lucky draw.The silent-flag runs were effectively draws from the flagless-baseline distribution; the compression-only signal was the strongest positive single-flag effect.Source: §VIII · p. 10

Token-savings accounting

  • 104,731 tokens across 574 compressed turns: HARBOR’s two-flag pick retained 96.97% of D’s compressor volume with a quarter of D’s flag count and a +5-pass improvement.D-all-on elided 108,004 compressor tokens across 592 compressed turns; its separate ACON summaries corrupted downstream cache reads.Source: §VIII · p. 11

Bug-catching and container persistence

  • Telemetry caught zero-retrieval reflection memory and a PASTE predictor that never executed; per-task container recycling explained the memory loss, prompting shared-state synchronization.The remediation used a harness-controlled state directory, upload/evaluate/download synchronization, and a file lock for concurrent sync-back.Source: §VIII · pp. 10–11

IX. DISCUSSION

A single HARBOR run matched the manual peak and beat the all-on stack, while the discussion stresses that harness features are model-dependent and the evidence remains empirical and narrowly scoped.

IX. DISCUSSION

  • A harness feature can be perfectly implemented yet lose when its behavior depends on the self-evaluator’s capability.The paper argues that this latent dependency makes harness engineering an optimization problem, not merely a systems problem.Source: §IX · p. 11
  • Five native features persist across sessions, so each task-independent benchmark session starts empty and requires a warm-start-aware evaluator.Source: §IX · p. 11
  • Prompt-prefix-first layout saved about 80% of prefix tokens on a 58 KB system prompt, but its benefit appears in wall-clock time rather than pass rate.The paper therefore folds wall-clock into the deployment constraint.Source: §IX · p. 11
  • Structured telemetry can expose integration bugs: writes with no consumer reads and cache byte-size far below raw tool output are diagnostic signatures.The discussion connects these checks to observability systems and property-check frameworks.Source: §IX · p. 11
  • The real-harness trace exercised trust-region shrinking, cheapest-fidelity selection, and silent-flag exclusion; cheap-fidelity baseline reuse also produced a coverage failure on T8.The reported bias preserves ranking, and stratified Tm is the stated fix.Source: §IX · p. 11
  • AHO is amortized over a few invocations per agent, task-suite, and model epoch because changes to the task distribution, agent, or model require a new optimizer run.The empirical claims are limited to one agent-benchmark pair and do not claim transfer across agent families or regret bounds.Source: §IX · pp. 11–12
  • Repeated runs can tighten the per-block SAAS scales monotonically and accumulate information across epochs when the block structure is preserved.The reported block-ANOVA decomposition already showed few dominant blocks at T=19.Source: §IX · p. 11
  • 17/89: HARBOR matched manual round B and beat the all-on round D by five passes at about 3.5 full-suite-equivalent search units.The B–Oracle gap was 17 versus 81 passes.Source: §IX · p. 12

APPENDIX A A REPRESENTATIVE HARBOR RUN ON THE REAL HARNESS

The appendix reports one complete, representative execution of Algorithm 2 on the real gpt-5.4-nano and Terminal-Bench 2 harness.

APPENDIX A A REPRESENTATIVE HARBOR RUN ON THE REAL HARNESS

  • The reported run was a typical trajectory, not a cherry-picked best, and used 19 function evaluations.Source: Appendix A · p. 12
  • The search consumed 9.826 of a nominal 10 full-suite-equivalent budget units.Source: Appendix A · p. 12
  • The run took approximately 10–11 hours of wall-clock time on the real gpt-5.4-nano×TB-2 harness.Source: Appendix A · p. 12

A. Trajectory and final artefact

The final artifact is a four-flag configuration scored at the cheapest fidelity, with a three-point Pareto frontier under the deployment-cost limit.

A. Trajectory and final artefact

B. Algorithmic-machinery checklist

The run exercised the specified search components, with every evaluation at the cheapest fidelity and no safety-constraint rejections.

B. Algorithmic-machinery checklist

C. Three findings

The run exposes three constraints on the reference algorithm: small-m stratification was violated, Bernoulli noise obscured rankings, and cost-aware escalation preserved the search budget.

a) The stratified-Tm assumption is load-bearing

  • The seeded-prefix shuffle gave the baseline Sobol-init point 0/8 on T8, collapsing the flagless base-rate estimate and violating the stratification assumption at small m.The resulting bias is driven by querying the base rate away from its estimation fidelity, while the shift is x-constant and preserves ranking.Source: C. Three findings · p. 12

b) Bernoulli noise dominates the signal at m=8

  • At m=8, three distinct configurations tied at 2/8 (25%), while the observed 0/8-to-2/8 spread fell within the approximately 13% Bernoulli standard error.The run therefore treated rankings at this fidelity as weak evidence and deferred a costly m=89 recheck.Source: C. Three findings · p. 12

c) Cost-aware escalation held the budget

  • The solver never escalated to m=89, avoiding truncation of Sobol coverage below the recommended 12-point minimum when its cost-adjusted information gain did not justify escalation.The acquisition compares KG(x, m)/c(m) across fidelities.Source: C. Three findings · pp. 12–13
Loading 2604.20938v1…