MemoHarness: Agent Harnesses That Learn from Experience

1 Introduction

• Most automated improvement methods optimize narrower artifacts, while deployed harnesses are often reused across cases despite differences in domain, reasoning depth, retrieval needs, and output format.The harness includes context construction, tool access, orchestration, memory, decoding, and output handling.Source: §1 · pp. 1–2
• MemoHarness decomposes the harness into six editable control surfaces: context, tool, generation, orchestration, memory, and output.This structures harness search as edits across dimensions rather than one opaque prompt.Source: §1 · p. 2
• Its dual-layer experience bank stores per-case execution entries and distilled global patterns, making search diagnostic rather than driven only by benchmark scores.The search ranks configurations by correctness first, using execution cost only as a tiebreaker.Source: §1 · p. 2
• At test time, MemoHarness retrieves similar cases and relevant patterns to adapt the global harness without test-time feedback or additional search.The adaptation uses the visible test case and accumulated training experience.Source: §1 · pp. 2–3
• 0.806 vs. 0.722 on the primary shell-agent benchmark beats the strongest fixed-harness baseline; held-out base models gain +0.098 in average task success.The paper also reports selective transfer to unseen suites and reduced per-task dollar cost relative to the strongest commercial baselines in that comparison.Source: §1 · p. 2

2 Method

• MemoHarness combines a structured six-dimensional harness space, a dual-layer experience bank, and correctness-first global search followed by test-time case adaptation.The method optimizes a global base harness during search, then applies a case-specific adaptation rule at evaluation.Source: §2 · p. 3

2.1 Motivation

• A single benchmark-level configuration can mismatch cases with different domains, reasoning depth, retrieval needs, or output formats, motivating diagnostic search and case-specific reuse of experience.The paper notes that a harness strong on average may still be systematically mismatched to particular case types.Source: §2.1 · pp. 3–4

2.2 Problem Formulation

• The problem formulation distinguishes labeled search cases from unlabeled evaluation cases and represents a harness as choices across six functional control dimensions.At evaluation, adaptation may use visible inputs and case features plus the experience bank, but not the unknown reference output.Source: §2.2 · p. 4

D2 Tool / D6 Output

• Tool controls determine when and how the harness calls external tools or retrievers, while output controls transform raw model output through answer extraction, schema validation, or fallback selection.These are distinct stages within the harness's inference flow.Source: §2.2 · p. 5, D2 Tool · p. 5, D6 Output · p. 5

2.3 Six-Dimensional Harness Space

• The six-dimensional harness space turns an opaque harness into separable control surfaces for structured editing, targeted diagnosis, and repair.The dimensions follow inference's temporal information flow, while making cross-dimensional interactions explicit.Source: §2.3 · p. 5, Table 1 · p. 5

2.4 Dual-Layer Experience Bank

• The experience bank pairs per-case execution entries with distilled global patterns that summarize recurring phenomena, supporting evidence, and expected effects of targeted harness changes.Structured queries return bounded slices of patterns, filtered case entries, and aggregated statistics rather than exposing the entire bank to the controller.Source: §2.4 · pp. 5–6

2.5 Training-Time Harness Optimization

• Training selects the global harness by primary reward first, using lower token usage only to break ties between candidates with equal reward.Search executes proposed harnesses on labeled cases, records trajectories, rewards, costs, and diagnoses, and periodically distills persistent failure clusters into global patterns.Source: §2.5 · pp. 6–7

2.6 Test-Time Case Adaptation Without Feedback

• At test time, MemoHarness retrieves global patterns, similar successful and failed cases, and feature-conditioned evidence to specialize the global harness once for each case, without evaluation-time learning or reselection.The resulting harness is executed for that case; simple cases can remain lightweight, while retrieval-heavy, multi-step, or format-sensitive cases can invoke richer orchestration when retrieved evidence warrants it.Source: §2.6 · p. 7

3.1 Setup

• The evaluation spans shell agency, code generation, and financial reasoning, with six external suites and six additional base models used for transfer tests.Terminal-Bench comparisons use released or benchmark-provided baselines and validation-selected harnesses.Source: §3.1 · p. 8

RQ1 Does adaptive harness optimization improve task success?

• MemoHarness reaches 0.806 on Terminal-Bench, beating the strongest fixed-harness baseline, Codex, by +0.084.Other baselines trail by +0.250 to +0.445; product-specific comparisons use the closest released configuration rather than a model-controlled scaffold ablation.Source: RQ1 Does adaptive harness optimization… · p. 8, Figure 2 · p. 8

RQ2 How does harness quality evolve during search?

• The final harness improves over base on all three benchmarks: Terminal-Bench 0.722→0.806, LiveCodeBench 0.900→0.967, and FinanceAgent 0.600→0.767.The shipped checkpoint is validation-selected, so it can fall below an in-training peak.Source: RQ2 How does harness quality evolve… · pp. 8–9, Figure 3 · p. 9
• FinanceAgent rises from 42.5% to 65.0% near iterations 8–9, while LiveCodeBench stays within a 91.2%–95.0% band.Later FinanceAgent rounds continue to find beneficial edits; LiveCodeBench is near saturation and has less headroom.Source: RQ2 How does harness quality evolve… · p. 9, Figure 4 · p. 9

RQ3 Do learned harness changes transfer to unseen suites?

• Transfer is selective: Terminal-Bench search lifts MMMLU by +0.030, StrongReject by +0.030, and SWE-Bench Pro by +0.059, while saturated suites show no movement.LiveCodeBench and FinanceAgent searches improve different suites; the strongest transfer is associated with long-horizon, toolcentric search, but component attribution remains untested.Source: RQ3 Do learned harness changes… · p. 10, Table 2 · p. 10

RQ4 Does the learned harness transfer across base models?

• Every evaluated model improves over its base setting in the transfer study, with mean gain +0.098 and gains ranging from +0.038 on GPT-4.1 to +0.233 on GLM-5.GPT-5.3-Codex retains the same +0.084 in-domain improvement reported in Figure 2.Source: RQ4 Does the learned harness transfer… · p. 10

RQ5 Is adaptive harnessing cost-effective?

• With 13.32M of 14.18M input tokens cached, MemoHarness costs $6.89, below Codex and Claude Code at higher success; Terminus and OpenCode cost less but score lower.The cost comparison uses offline public list prices and depends on the caching assumption; evaluation is on Terminal-Bench’s 18-task held-out split.Source: RQ5 Is adaptive harnessing… · p. 11, Table 4 · p. 11

• MemoHarness combines a structured six-dimensional harness space, a dual-layer experience bank, and test-time case adaptation to reuse past execution experience.The conclusion characterizes this combination as making harness search diagnostic and reusable.Source: §4 · p. 11
• The results suggest improving the control layer around an LLM can complement model scaling and manual harness engineering.Source: §4 · p. 11
• Future work should examine fully unsupervised search, larger-scale validation, component attribution, and online experience accumulation across deployments.Source: §4 · p. 11

B Implementation details

• Appendix B covers implementation details.Source: Appendix Contents · p. 14

C Experimental details

• Appendix C covers experimental details.Source: Appendix Contents · p. 14

D Models used

• Appendix D lists the models used.Source: Appendix Contents · p. 14

E Comparison scope

• Appendix E covers comparison scope.Source: Appendix Contents · p. 14

F Related work

• Appendix F covers related work.Source: Appendix Contents · p. 14

G Operation-level diagnostic

• Appendix G presents an operation-level diagnostic.Source: Appendix Contents · p. 14

• The primary Terminal-Bench evaluation uses an 18-task held-out split and reports point estimates without confidence intervals or significance tests.The authors frame the results as evidence under a fixed evaluation protocol, not a full statistical characterization of harness performance.Source: A Limitations · p. 14
• Some comparisons are system-level because baselines may differ in model, tool, or product interfaces rather than provide a pure scaffold transplant.Source: A Limitations · p. 15
• The experiments do not fully isolate every component, including the experience bank, global patterns, and case-specific test-time adaptation in all settings.Source: A Limitations · p. 15
• The cost analysis depends on observed experience cacheability, so deployments with lower cache reuse may have a different cost profile.Source: A Limitations · p. 15
• The reference implementation uses practical heuristic controllers and diagnostic operators; the authors identify more general controllers as future work.Source: A Limitations · p. 15

• The reference implementation instantiates the abstract harness, diagnostic, and cost components without altering MemoHarness’s algorithmic structure.These choices specify how the abstract objects are realized in the reported experiments.Source: B Implementation Details · p. 15
• A harness bundle pairs a structured D1–D6 policy file with operating instructions, a persistent playbook, and the distilled memory in scope.Edits update both the dimension summary and the accompanying instructions.Source: B Implementation Details · p. 15
• Reported experiments start from a minimal harness with demonstrations, retrieval, structured scaffolding, cross-call memory, and output validators disabled.Source: B Implementation Details · p. 15
• The diagnostic maps observable execution evidence to one of six dimensions, producing an intentionally coarse per-case diagnosis.Repeated failures can be distilled into cross-case patterns with supporting evidence and a dimension-level repair suggestion.Source: B Implementation Details · p. 15
• Token usage is the search-time cost quantity used for selection, while dollar cost is computed offline from token counts and model list prices.Call count and latency are logged for debugging and reporting but do not affect selection.Source: B Implementation Details · pp. 15–16
• Global-pattern distillation runs on a fixed schedule and when repeated failures accumulate, with a per-round cap to bound prompt budget.The distiller uses failure histories, reward trends, dimension assignments, and configuration snapshots.Source: B Implementation Details · p. 16

Benchmarks

• Terminal-Bench uses an 80/20 split of its 89 tasks, yielding 18 held-out tasks for absolute performance and cost reporting.LiveCodeBench tests single-shot code generation, while FinanceAgent covers multi-step financial reasoning with documents and tool calls.Source: C Experimental Details · p. 16

Cross-dataset evaluation

• Six external suites extend cross-dataset evaluation across question answering, code repair, safety refusal, verifiable and legal reasoning, and software engineering.The suites are MMMLU, HumanEvalFix, StrongReject, Reasoning-Gym-Easy, LawBench, and SWE-Bench Pro.Source: C Experimental Details · p. 16

Baselines

• Terminal-Bench comparisons use four released harnesses or reproducible configurations, with Codex the strongest baseline in this setup.The comparison includes Codex, Claude Code, OpenCode, and Terminus; model-agnostic generators use GPT-5.3-Codex where possible.Source: C Experimental Details · p. 16

Models and protocol

• Harness search uses GPT-5.3-Codex, while six other models from four families test cross-model transfer without retraining.Table 5 identifies the seven evaluated models and distinguishes proprietary API access from publicly released weights.Source: C Experimental Details · p. 16, Table 5 · p. 17

Evaluation protocol

• Evaluation uses the harness selected by validation at search end, averages results across repeated runs, and calculates dollar costs from public GPT-5.3-Codex token prices.Terminal-Bench cost reporting averages token counts over repeated evaluation runs.Source: C Experimental Details · p. 17

Search settings

• Search runs for 10 outer iterations, distills global patterns after five new entries or three consecutive same-case failures, and uses deterministic generation.The dataset split is fixed at 80/20 with seed 42; generation uses temperature 0.0, top-p 1.0, one candidate, and an 8192-token maximum.Source: C Experimental Details · p. 17

Source model

• GPT-5.3-Codex is the source model used during harness search.The remaining six models are reserved for the RQ4 transfer study.Source: D Models Used · p. 17

Transfer models

• Six additional base models execute the search-derived harness at test time with no further training.They represent six models from four families.Source: D Models Used · p. 17

Model access

• Table 5 distinguishes proprietary models served only by API from models with publicly released weights.Source: D Models Used · p. 17

E Comparison Scope

• Meta-Harness is treated as related work, not a direct baseline, because no suitable public implementation was available when experiments were finalized in early 2026.The authors exclude later-released reference code and avoid a reimplementation whose design choices would complicate attribution.Source: E Comparison Scope · p. 17

F.1 Optimization for Agents

• Prior optimization spans prompts and self-feedback [Yang et al., 2023; Madaan et al., 2023], pipelines and workflows [Khattab et al., 2023; Zhang et al., 2024], and harness code.The paper characterizes these approaches primarily as pre-deployment optimization, with case-level harness adaptation at deployment less explored.Source: §F.1 · p. 18

F.2 Harness Engineering

• Practitioner and emerging academic work treat harness design as consequential, but the authors report no known prior work combining reusable diagnostic experience with test-time case adaptation.Examples include production guidance on context and tools (Anthropic, 2024; LangChain, 2025) and harness-focused research (Pan et al., 2026).Source: §F.2 · pp. 18–19

Operation-level associations

• New curl, echo, and grep operations are weakly or negatively associated with reward improvement in this analysis.The authors say this operation-level decomposition is possible because the experience bank stores per-case traces alongside scalar scores.Source: G Operation-Level Diagnostic on… · p. 19
• +17 to +60 pp lift is reported for newly added cat, sed, which, and test operations, indicating association with reward-increasing transitions.The analysis interprets these operations as often appearing when inspection or condition-checking gaps are repaired, not as proof of causation.Source: G Operation-Level Diagnostic on… · p. 19, Table 6 · p. 20

Analysis design

• Table 6 measures each newly added operation on fixed-task adjacent-iteration pairs against an all-transition baseline of approximately 13.2%.It reports occurrence counts, reward-increase counts, positive-transition rates, percentage-point lift, and relative lift.Source: Table 6 · p. 20