PaperlayerView exact revision on arXiv

Source-linked AI summary

Natural-Language Agent Harnesses

Linyue Pan, Lexiao Zou, Shuo Guo, Jingchen Ni, Hai-Tao Zheng

arXiv:2603.25723v2cs.CLcs.AI

TL;DR

Agent harnesses are often buried in coupled controller code, making their reusable policies difficult to inspect and compare. This paper represents harness policies as executable natural-language documents interpreted by a shared runtime, finding competitive task outcomes across coding, terminal-use, and computer-use benchmarks while making policies shorter and easier to inspect.

  • Problem

    Agent harnesses are typically bundled with implementation details, making reusable harness policies difficult to inspect, port, compare, and ablate.

  • Method

    The paper introduces editable Natural-Language Agent Harnesses executed by a shared Intelligent Harness Runtime under common runtime semantics.

  • Results

    Across coding, terminal-use, and computer-use benchmarks, IHR-executed NLAHs remain competitive with native harnesses while exposing much shorter policies that support mechanism audits and module ablations.

  • Takeaways & Limitations

    Explicit natural-language harness policies make harness design a more inspectable experimental object for process-level and mechanism-level analysis.

  • Takeaways & Limitations

    The prototype runtime’s main mechanism weakness is information loss across parent-child handoff boundaries, with recall falling to 0.322 and 0.553 in the reported settings.

Abstract

from arXiv · show

Agent performance is strongly shaped by the surrounding harness: the external execution system around a model that organizes a task run. Yet this logic is usually buried in tightly coupled controller code, which makes harnesses hard to inspect, compare, transfer, and ablate. This paper asks whether the reusable design pattern of an agent harness can be represented as an executable natural-language object. We introduce Natural-Language Agent Harnesses (NLAHs), editable documents that describe run-level harness policy, and Intelligent Harness Runtime (IHR), a shared runtime that interprets these documents into agent calls, handoffs, state updates, validation gates, and artifact contracts. Across coding, terminal-use, and computer-use benchmarks, IHR-executed NLAHs achieve comparable task outcomes to code and prompted realizations, while exposing much shorter static harness policies. Module ablations further show that explicit harness modules are analyzable. These results suggest that agent harnesses can be turned from incidental glue around models into scientific representation objects.

1 Introduction

The paper proposes representing agent harness policies as executable natural-language documents, with a shared runtime translating them into auditable execution mechanisms. It evaluates whether this separation preserves performance while enabling concise, inspectable, and modular harness analysis.

  • Agent harnesses organize tools, state, failure recovery, validation, and delegation, but strongly affect measured performance.
  • Code harnesses entangle prompts, adapters, parsers, validation, artifacts, retries, context, and benchmark assumptions, hindering inspection, portability, comparison, and ablation.
  • NLAHs externalize run-level harness policy as editable natural-language text, while IHR executes it through agent calls using code and runtime mechanisms for tools, parsing, sandboxing, and logging.
  • The evaluation asks whether IHR-executed NLAHs preserve task performance, materialize intended harness mechanisms, and support module-level interventions.
  • Across three benchmark families, NLAHs shape agent behavior with comparable task outcomes, concise static harness policies, and support for module-level analysis.

2 Preliminaries

The preliminaries define models, agents, and harnesses as layered execution concepts. A harness organizes model-centered interaction with tools, state, observations, validation, recovery, delegation, and stopping decisions.

  • A model is a callable learned function mapping context c to output y, with context potentially containing text, images, or video.
  • An agent wraps one or more model calls with external interaction, maintaining execution state, observing feedback, and deciding whether to act, request information, validate, or stop.A single model call is a degenerate agent call when the agent may call the model.
  • NLAH describes harness behavior at the execution level where prompts, tools, state, validation, and delegation operate.In the NLAH+IHR framework, NLAH stores readable policy, IHR supplies shared execution semantics, and scripts or adapters perform exact operations.
  • A harness is the external execution system that turns a base model into an agent by organizing prompts, tools, state, observations, validation, recovery, delegation, and stopping.Harness engineering covers designing, implementing, adapting, debugging, and evaluating agent harnesses.

3 Methodology

NLAH+IHR separates a minimal agent substrate, shared runtime semantics, and editable per-harness policy, while retaining deterministic code for precision-critical operations. NLAHs make task stages, state, evidence, validation, retries, and module boundaries explicit in executable natural-language policy.

  • System architecture: NLAH+IHR has four layers, including a code-form minimal base agent whose only external model-facing tool is a terminal.The terminal supports file operations, processes, event recording, and launching child agents.
  • System architecture: The base agent and adapters provide the machine interface, IHR supplies shared execution semantics, and each NLAH supplies per-harness policy.This separation makes the changing harness policy distinct from the stable execution machinery.
  • System architecture: IHR uses the base agent as an orchestrator and delegates substantive work to child agents, preserving visible boundaries between harness control and task execution.Even a nominally single-agent harness is realized as a parent orchestrator plus one executor child; multi-role or multi-branch harnesses use separate children.
  • Policy design: NLAHs specify run-level decisions such as state creation, verifier timing, preserved evidence, retry conditions, and run closure, while runtime mechanisms instantiate them.Natural language covers policy, whereas tests, parsers, sandboxing, benchmark adapters, and artifact validators remain deterministic code.
  • Policy design: Effective NLAHs define contracts, stages, state, evidence, enforceable conditions, and independently ablatable module boundaries rather than vague behavioral advice.Recommended clauses identify inputs and outputs, mechanism-use conditions, stored artifacts, verifier evidence, and explicit module names for controlled removal or alteration.

4 Experimental Design

The experiments compare code, prompted NLAH, and IHR-executed NLAH realizations across coding, terminal-use, and computer-use benchmarks. They examine task outcomes, harness mechanisms, and module-level ablations.

  • Harness Mechanism Realization: RQ2 tests whether IHR-executed NLAHs preserve workflow structure, contract enforcement, tool use, recovery, and information handoff.The evaluation asks whether intended harness mechanisms are materialized during execution.
  • Module Ablation: RQ3 tests whether natural-language harness modules can be cleanly ablated and analyzed at the module level.This isolates the analyzability of explicitly expressed harness modules.
  • Harness Realization: RQ1 compares three realizations of the same harness idea: Code Harness, Prompted NLAH, and IHR-executed NLAH.Code Harness provides strongest deterministic control; Prompted NLAH tests passive natural-language instructions; IHR adds runtime semantics for lifecycle, artifacts, state, contracts, and stopping.
  • Benchmark Families: The evaluation spans SWE-bench Verified, Terminal-Bench 2.0, and OSWorld, covering repository issue resolution, long-horizon Linux command-line tasks, and real desktop computer use.Their primary metrics are issue resolution rate, task success, and task success rate, respectively.
  • Experimental Setup: All experiments use Codex CLI 0.123.0 with gpt-5.4-mini at reasoning effort xhigh in Dockerized Ubuntu 24.04 environments.Servers provide 64 CPU cores and 251 GiB memory; per-task containers are capped at 32 vCPUs, 84 GiB memory, and 40 GiB storage.

5 Results

Across coding, terminal-use, and computer-use benchmarks, IHR-executed NLAHs preserve competitive task outcomes while exposing compact, inspectable harness policies and auditable mechanisms. Results also show that explicit modules help when they strengthen state and acceptance discipline, whereas handoff reliability and excessive branching remain limitations.

  • RQ1: IHR-executed NLAHs achieve task performance in the same regime as corresponding code harnesses across audited settings.Live-SWE reaches 73.0 versus 67.0 for native code and 77.0 for prompted NLAH; OSWorld reaches 46.3 versus 47.1 for code; MHTBA reaches 53.9 versus 36.0 for code and 57.3 for prompted NLAH.
  • RQ1: NLAHs reduce readable harness policy from 60.1k to 2.9k tokens on Live-SWE and from 10.5k to 0.8k tokens on MHTBA.The reduction separates high-level state handling, validation, recovery, candidate search, and completion gates from deterministic mechanisms.
  • RQ1: NLAHs preserve policy-guided behavior while allowing different concrete routes that satisfy the same completion contract.OSWorld cases retain staged observation, action selection, recovery, and completion checking while permitting shell commands, file edits, or package-level operations.
  • RQ2: Information Handoff Recall drops to 0.322 and 0.553 under parent-child execution, identifying handoff and orchestration reliability as the main runtime weakness.NLAH Orchestration Reliability is lower than Prompt on both Live-SWE and MHTBA, consistent with the prototype’s added execution overhead.
  • RQ2: IHR-executed NLAHs materialize auditable workflow, contract, verification, tool-use, recovery, and handoff signals in addition to competitive task outcomes.On Live-SWE, NLAH reaches 1.000 Artifact Contract, 0.933 Tool Call Success, and 0.992 Failed Tool Continuation; on MHTBA, the corresponding values are 0.955, 0.928, and 0.995.
  • RQ3: File-backed state improves performance from 73.0 to 75.6 on SWE and from 44.4 to 58.3 on OSWorld, while self-evolution reaches 78.8 on SWE and 52.8 on OSWorld.Evidence-backed answering adds +2.8 on both benchmarks; multi-candidate search increases Agent Calls from 1.1 to 5.7 on SWE and from 1.083 to 1.333 on OSWorld, but yields only +2.8 on OSWorld and drops SWE from 73.0 to 71.4.

6 Related Work

Related work shows that agent performance depends on execution scaffolds, operational knowledge can be packaged as reusable text, and natural language can serve as a programmable or executable medium. These strands motivate representing harness policies as inspectable natural-language objects.

  • Agent harnesses and scaffold-aware evaluation: Agent performance depends on execution scaffolds encompassing tools, feedback loops, state, validation, and workflow structure.Related directions include code-harness synthesis, scaffold-aware benchmarks, agent graph compilation, and multi-agent routing or orchestration.
  • Natural-language instruction carriers: Prompts, AGENTS.md, CLAUDE.md, AgentSkills, and related bundles package operational knowledge as reusable text attached to agent runs.Recent skill and memory research further studies learning, evolving, storing, and transferring reusable procedures.
  • Natural language as programs, workflows, and constraints: Prompt programming and language-model programming frameworks treat prompts and LLM calls as programmable objects, while other systems compile natural language into workflows, graphs, runtime constraints, or executable specifications.Examples include LMQL, DSPy, APPL, and SGLang.

7 Conclusion

The paper presents Natural-Language Agent Harnesses and the Intelligent Harness Runtime as compact, executable, analyzable representations of agent harness policy. Across coding, terminal-use, and computer-use benchmarks, IHR-executed NLAHs remain competitive with native harnesses while enabling shorter, more inspectable policies and mechanism-level analysis.

  • Conclusion: Natural-Language Agent Harnesses and the Intelligent Harness Runtime externalize agent harness policy as a compact, executable, and analyzable representation.IHR executes NLAHs under shared runtime semantics.
  • Conclusion: Across coding, terminal-use, and computer-use benchmarks, IHR-executed NLAHs remain competitive with native harnesses while making the policy layer much shorter and easier to inspect.The comparison spans all three benchmark categories named in the conclusion.
  • Conclusion: Mechanism audits and module ablations show that explicit harness documents support process-level inspection and mechanism-level analysis.These analyses examine both observable process behavior and individual harness mechanisms.

A Discussion

The discussion presents NLAHs as an explicit run-level harness-policy layer, while code, IHR, and deterministic hooks preserve exact executable mechanisms. This makes harnesses inspectable, auditable, and experimentally modifiable, while retaining natural language’s role in evidence, delegation, verification, and stopping policies.

  • NLAHs as a policy layer: NLAHs express harness policy—including roles, contracts, evidence requirements, retries, validation, state handoff, and stopping—while code handles exact mechanisms.IHR and deterministic hooks preserve executable precision for parsers, tool execution, sandboxing, adapters, logging, and validators.
  • Why explicit harness policy matters: NLAHs change harness design from hidden controller logic into a text-readable, executable, auditable, and module-wise modifiable experimental object.This enables researchers to examine which policy choices contribute to system behavior rather than comparing only whole systems.
  • Natural language remains useful at the harness level: Natural language remains useful for run-level policies specifying what evidence to preserve, when to delegate, how to verify, and when to stop.The discussion distinguishes this role from potentially brittle prompt-level tricks whose gains may diminish as models strengthen.
  • Toward harness representation science: Explicit harnesses become searchable and testable, enabling future retrieval, composition, mutation, and optimization of NLAH modules under shared runtime assumptions.This points toward a controlled science of harness representations focused on how policy choices produce differences between agent systems.

C MHTBA Code-Artifact Portability on TB2 · D Formalization and Definitions

The MHTBA code artifact transfers poorly from its Claude Opus 4.6 environment to GPT because its completion and stopping protocol often causes valid runs to time out. The paper defines NLAH’s boundary as editable task-family policy, while retaining shared execution, exact correctness, safety, and permissions in code or runtime components.

  • C MHTBA Code-Artifact Portability on TB2: Controlled RQ1 evaluates the released Meta-Harness TB2 code artifact under gpt-5.4-mini, reasoning effort xhigh, and one attempt to test cross-model portability.Its public usage reports Terminal-Bench 2.0 results under anthropic/claude-opus-4-6 with –n-attempts 5.
  • C MHTBA Code-Artifact Portability on TB2: 32/89 code score is driven partly by failed stopping rather than task-solving inability: 66/89 runs time out, including 21/66 with reward 1.0 and 45/66 failed timeouts.The reward-1.0 timeouts satisfy the verifier but the controller does not stop.
  • C MHTBA Code-Artifact Portability on TB2: In GPT-versus-natural-language disagreements, the failing code artifact often runs hundreds of episodes, consumes tens of millions of prompt tokens, and accumulates no-tool warnings.The pattern indicates a runtime-protocol symptom, although task-domain difficulty explains part of it.
  • C MHTBA Code-Artifact Portability on TB2: The completion gate requests a second completion-tool call after task_complete; text-only DONE responses clear pending completion and trigger continued looping.The controller issues a no-tool warning when GPT answers the confirmation prompt without another tool call.
  • C MHTBA Code-Artifact Portability on TB2: The tune-mjcf trace reaches verifier reward 1.0 yet times out after 3600 seconds, 186 episodes, and 5.4M input tokens because it cannot satisfy the exact two-call stopping protocol.The trace alternates task_complete calls, text-only DONE responses, warnings, and a no-op shell command.
  • D.2 NLAH expressivity boundary and code-harness mapping: The expressivity boundary assigns shared or machine-execution decisions to base-agent code or runtime policy, task-family-specific editable decisions to NLAH, and exact-correctness decisions to scripts or adapters.This classification distinguishes shared substrate, runtime semantics, editable policy, deterministic hooks, and model-internal behavior.
  • D.2 NLAH expressivity boundary and code-harness mapping: NLAH supports fast iteration, auditing, portability, and module-level experimentation, but interpretation uncertainty, model dependence, cost, and safety or permission risks limit production use.Production systems should retain safety, permissions, evaluation, and key parsing logic in code.

E Runtime and Implementation Details … F NLAH Modules

IHR makes NLAHs executable through a shared runtime that enforces delegated execution, explicit context and artifact semantics, and contract-first completion. Its modules specify auditable patterns for handoffs, evidence, verification, retries, candidate selection, delegation, compression, and memory.

  • E.1 Runtime-policy prompt: IHR realizes even single-agent harnesses as a runtime parent plus task child, keeping workspace work delegated and boundaries inspectable.The runtime reconstructs roles, stages, repetition, and independence requirements from NLAH text and realizes them as child-agent launches.
  • E.1 Runtime-policy prompt: When an NLAH is absent or incomplete, IHR constructs a minimal runnable delegated baseline and applies additional NLAH clauses as overlays.This baseline grounds task instructions in a runnable delegated substrate.
  • E.1 Runtime-policy prompt: IHR separates durable intermediate state under STATE_ROOT from judgeable deliverables under /sa-output/artifacts and requires inspectable evidence for staged or multi-role execution.The default STATE_ROOT is /sa-output/runtime, while state is written only when needed for reuse or auditability.
  • E.2 Realizing Harness Aspects with IHR: IHR materializes harness aspects through agent calls, code-backed tools, orchestration, path-addressable memory and retrieval, and contract-preserving context compression.NLAHs specify policies and contracts, while executable tools, files, adapters, and runtime operations implement them.
  • F NLAH Modules: The ROOT, HANDOFF, CHILD PACKET, and BOOKKEEPING modules make state, transfers, child inputs and outputs, launches, promotions, and recovery path-addressable.They maintain STATE_ROOT/RESPONSE.md, named files under STATE_ROOT, child TASK.md and RESPONSE.md files, task_history.jsonl, and artifacts/manifest.json.
  • F NLAH Modules: The ARTIFACT, STRUCTURE, CLAIM DISCIPLINE, and GATE modules require standalone evidence covering validation and uncertainty, provenance for major claims, and no release while critical claims remain deficient.Claims must identify direct observation or inference and provide minimal supporting spans or output segments when available.
  • F NLAH Modules: The VERIFIER, LOOP, BUDGET, DIVERSITY, SELECTION, and ESCALATION modules independently check candidates, retry failures, default to five attempts or candidates, diversify search, and prune or expand branches.Verification returns one primary verdict without repairing the candidate; retries respond to concrete failure signals, and candidate comparison uses task fit, evidence quality, coherence, and repair cost.

F.1 Additional RQ3 module observations

Additional module observations show that validation helps when aligned with benchmark acceptance criteria, while durable file-backed state is more reliable than aggressive compression. Memory modules diverge across benchmarks, with context compression consistently harmful and Markdown memory mixed.

  • Verifier improves both benchmarks, but gains are uneven: +0.2 on SWE and +8.4 on OSWorld.The passage attributes this benefit to keeping the verifier’s judgment close to the benchmark’s acceptance criterion.
  • Context compression hurts both benchmarks, dropping SWE from 73.0 to 72.0 and OSWorld from 44.4 to 36.1.
  • Markdown memory is mixed, hurting SWE by 2.8 points but helping OSWorld by 5.6.
  • File-backed state is positive in both settings, making durable path-addressable state more reliable than summarization or free-form memory.

G Limitations, Risks, and Broader Impact · Limitations · Broader impact and risks

NLAHs’ main limitation is natural-language imprecision, requiring executed runs to verify behavior rather than relying on policy text alone. Externalized harnesses may reduce development costs and improve reuse, but can also increase risks involving tool use, delegation, and artifact handling.

  • Limitations: Natural-language imprecision is the main limitation of NLAHs.NLAH policies may leave semantically important constraints under-specified.
  • Limitations: Different models may interpret the same NLAH constraints differently.The passage warns that important constraints can be interpreted differently across models.
  • Limitations: Paraphrase can weaken semantically important constraints in NLAHs.This makes policy wording an unreliable sole basis for inferring behavior.
  • Limitations: Exact mechanisms remain in code, while executed behavior must be checked through runs.The authors explicitly avoid inferring behavior from NLAH text alone.
  • Broader impact and risks: Externalizing harness modules can reduce development cost, improve comparability, and encourage reuse of robust workflows.These benefits arise from making harness logic portable and externally represented.
  • Broader impact and risks: Portable harness logic and scripts may lower the barrier to spreading risky workflows.The passage presents this as a broader-impact risk alongside the potential benefits of externalization.
  • Broader impact and risks: Harnesses can introduce attack surfaces for prompt injection, malicious tool grafting, or supply-chain contamination.These risks follow from harnesses mediating tool use, artifact handling, and delegation.
  • Broader impact and risks: Deployments should combine provenance tracking, review, permission control, and sandboxing.The supplied passage lists these measures as safeguards for deployment risks.
Loading 2603.25723v2…