PaperlayerView exact revision on arXiv

Source-linked AI summary

AI Harness Engineering: A Runtime Substrate for Foundation-Model Software Agents

Hailin Zhong, Shengxin Zhu

arXiv:2605.13357v1cs.SEcs.AI

TL;DR

Autonomous software-engineering agents remain unreliable, and the paper argues that the gap reflects not only model capability but also missing runtime support. It formalizes AI Harness Engineering, operationalizes it through an H0–H3 ladder and trace-based evaluation, and finds that higher harness levels produce richer auditable evidence. The paper therefore shifts attention toward model–harness–environment systems that establish verifiable, attributed, and maintainable changes.

  • Problem

    Autonomous software-engineering agents remain unreliable in realistic development settings, while the prevailing explanation locates the gap in model capability.

  • Method

    The paper formalizes AI Harness Engineering as a runtime substrate, defines its responsibilities, and evaluates progressively exposed support through an H0–H3 ladder and trace-based protocol.

  • Results

    Higher harness levels produce reproduction logs, failure attributions, deterministic requirement checks, and structured verification reports absent at lower levels.

  • Takeaways & Limitations

    Autonomous software engineering should be evaluated as a model–harness–environment system producing verifiable, attributed, and maintainable changes.

Abstract

from arXiv · show

Foundation models have transformed automated code generation, yet autonomous software-engineering agents remain unreliable in realistic development settings. The dominant explanation locates this gap in model capability. We propose a different locus: software-engineering capability emerges from a model-harness-environment system, in which a runtime substrate -- the harness -- mediates how a foundation-model agent observes a project, acts on it, receives feedback, and establishes that a change is complete. We formalize this substrate as an AI Harness Engineering and identify eleven component responsibilities: task specification, context selection, tool access, project memory, task state, observability, failure attribution, verification, permissions, entropy auditing, and intervention recording. We operationalize the harness through a four-level ladder (H0-H3) that progressively exposes runtime support to the agent, and we propose a trace-based evaluation protocol that converts each agent run into an auditable episode package. Applied to a controlled validation task, the framework yields episode packages whose evidence structure varies systematically with harness level: lower levels produce only a final patch, higher levels produce reproduction logs, failure attributions, deterministic requirement checks, and structured verification reports. The framework reframes the central question of autonomous software engineering from whether a foundation model can produce a patch to whether the model-harness-environment system can produce a verifiably correct, attributed, and maintainable change. We outline a research program for the runtime systems that foundation-model software agents will require.

Abstract

The paper concerns harness engineering for foundation-model software agents, focusing on runtime systems, evaluation, and verification.

  • The paper examines harness engineering for foundation models and autonomous software engineering.
  • Its focus includes runtime systems, agent evaluation, and verification for software-engineering agents.

Introduction

The introduction argues that reliable autonomous software engineering depends on the model, harness, and environment together, not model capability alone. It defines the harness as a runtime substrate, proposes a controlled H0–H3 ladder and trace-based evaluation, and reports richer evidence at higher harness levels.

  • Software engineering requires long-horizon context management, project memory, tools, feedback, validation, permissions, rollback, and maintenance discipline.
  • Autonomous software-engineering capability emerges from the model–harness–environment system rather than the model alone.
  • The paper defines AI Harness Engineering as a runtime layer managing context, tools, project memory, task state, observability, failure attribution, verification, and permissions.
  • The H0–H3 ladder progressively exposes runtime support, while the trace protocol records execution evidence and evaluates verification autonomy.
  • Higher harness levels produce reproduction logs, failure attributions, requirement-level verification, and structured reports that lower levels lack.
  • The framework distinguishes harness engineering from existing practitioner harnesses by defining components, controlled ablations, and episode evidence.

A runtime view of autonomous software engineering

Autonomous software engineering is a stateful, tool-mediated process whose capability depends on the model–harness–environment system rather than the model alone. The framework uses runtime evidence and failure attribution to distinguish missing harness support from model failure.

  • Software-engineering capability extends beyond coding ability to repository navigation, context selection, tool use, testing, failure interpretation, verification, documentation, and maintenance.
  • Runtime structure mediates task representations, repository observations, tool invocations, feedback, and completion decisions, so the same model can appear competent or incompetent depending on that support.
  • The autonomy gap is the difference between local coding ability and completing software tasks without runtime-substituting human help.
  • Common failures include inspecting the wrong files, misinterpreting tests, patching the wrong architectural layer, failing to preserve prior behavior, and declaring completion without evidence.
  • Human interventions diagnose missing responsibilities such as context management, observability, and entropy auditing, while M-HIR measures those missing-harness interventions.
  • The failure taxonomy distinguishes context, tool, feedback, verification, recovery, entropy, model, and unknown failures, enabling diagnosis beyond pass/fail evaluation.

The AI Harness Engineering

AI Harness Engineering is a runtime substrate that surrounds a foundation-model software agent and manages the resources, mediation, verification, and evidence needed for auditable behavior. It is evaluated as task-runtime infrastructure distinct from prompts, agent frameworks, interfaces, operating systems, and evaluation harnesses.

  • AI Harness Engineering is a runtime substrate surrounding a foundation-model software agent and managing the supports required for software-engineering behavior.
  • A harness governs how an agent observes a project, acts on it, receives feedback, and establishes completion, while remaining external to the model and evaluable through ablation and tracing.
  • Its design principles require explicit resources, traceable mediation, requirement-level verification, attribution before recovery, and maintenance or entropy awareness.
  • The eleven component responsibilities specify runtime contracts, characteristic unmanaged failure modes, and evidence artifacts for agent-on-repository execution.
  • The harness manages resources including context, tools, verification evidence, project memory, task state, human attention, permissions, failure signals, entropy, and test-time compute.
  • Unlike prompts, agent frameworks, ACIs, agent operating systems, evaluation harnesses, or DevOps infrastructure, a development harness configures runtime supports and records execution evidence for software agents.

A controlled harness ladder

The H0–H3 ladder provides a controlled way to vary runtime support while holding the task, repository, and initial state fixed. Each level progressively exposes tools, project context, memory, task state, observability, and verification artifacts.

  • A controlled harness ladder: The H0–H3 ladder progressively exposes runtime support while keeping the task, repository, and initial state constant.The ladder is designed to make the harness contribution empirically separable from the model’s.
  • A controlled harness ladder: H0 supplies only the task and repository, whereas H1 adds tool registries and protocols without project knowledge or verification discipline.H0 is the minimal comparison point; H1 makes the action surface explicit and traceable.
  • A controlled harness ladder: H2 adds agent-readable project memory, task state, and context selection, making context use part of the visible harness.These artifacts include architecture, testing conventions, and known failures.
  • A controlled harness ladder: H3 adds deterministic checks, reproduction, failure attribution, verification protocols, and a verification report.H3 therefore exposes the artifacts needed to turn completion into an evidentiary process.
  • A controlled harness ladder: The ladder measures process properties—including context inspection, failure reproduction and attribution, requirement verification, preserved behavior, entropy, and intervention—not task success alone.The visibility matrix operationally defines which artifacts are available at each level.

Trace-based evaluation

The evaluation protocol treats an agent run as an auditable episode rather than a single response. It records intermediate execution evidence and classifies final outcomes by both task behavior and verification quality.

  • Trace-based evaluation: Each episode produces an auditable package containing eight trace types, a patch, verification and final reports, and a final-outcome record.The pipeline converts an input pack through an agent episode into evidence that is then classified.
  • Trace-based evaluation: The protocol records action, tool, context, verification, failure-attribution, intervention, entropy, and outcome traces.These traces connect runtime resources to diagnosable failure types.
  • Trace-based evaluation: Verification traces identify the verification method, result, covered requirements, and the agent’s interpretation across reproduction, checks, tests, regression, lint, and review.Tool traces additionally record commands, exit codes, duration, timeout status, failure type, and recovery.
  • Trace-based evaluation: The five-label outcome taxonomy separates task behavior from evidence quality, allowing a correct patch to remain unverified and a failed patch to remain diagnostically useful.Labels include autonomous verified success, assisted verified success, unverified success, failed, and unsafe invalid.
  • Trace-based evaluation: The protocol supports population-level metrics including autonomous verified success, missing-harness intervention, verification autonomy, attribution completeness, tool recovery, and entropy delta.These metrics summarize episode packages for specified model, harness, task, and repository cells.

An illustrative case: a controlled validation task

A controlled login-validation task tests whether the ladder can produce distinct evidence packages under identical task conditions. All levels produce a working patch, but higher levels provide progressively richer diagnosis and verification evidence.

  • An illustrative case: a controlled validation task: The task requires rejecting empty passwords with “Password is required.” while preserving valid-login and invalid-non-empty-credential behavior.It also requires a test for the corrected behavior and continued regression coverage or documented instability.
  • An illustrative case: a controlled validation task: All four harness levels produce a working patch, while their episode packages differ in evidence structure.The case is designed to demonstrate operational feasibility rather than population-level performance differences.
  • An illustrative case: a controlled validation task: The empty-password probe returns “Invalid credentials.” instead of the expected “Password is required.”, identifying validation as the faulty layer.The fix rejects empty or whitespace-only passwords and adds a covering test.
  • An illustrative case: a controlled validation task: Higher levels expose tool traces at H1, context traces at H2, and reproduction logs, attribution logs, deterministic checks, and verification reports only at H3.The protocol records full-regression timeout instability rather than hiding it.
  • An illustrative case: a controlled validation task: The H3 workflow reproduces the failure, attributes it, applies a targeted fix, verifies required and preserved behavior, and reports evidence and limitations.Verification includes deterministic probes, targeted tests, and a bounded full-regression attempt.

Implications

The framework reframes autonomous software engineering as a property of the model–harness–environment system rather than the model alone. Its implications include explicit verification, auditable memory use, runtime treatment of instability, and repository affordances designed for agents.

  • Implications: The central question becomes whether the model–harness–environment system can produce a verifiable, attributed, maintainable change rather than merely a patch.This reframing treats runtime support as part of software-engineering capability.
  • Implications: Placing verification inside the harness requires reproduction, attribution, targeted fixing, requirement checks, and evidence reporting.This produces a transferable record of why the change is believed to be correct.
  • Implications: Tracing project-memory use makes it possible to inspect which artifacts influenced decisions and whether the agent ignored, misunderstood, or correctly applied them.Memory therefore becomes an analyzable runtime resource rather than an invisible prompt ingredient.
  • Implications: The harness treats tool instability and entropy as runtime resources by recording timeouts, recovery, and maintenance burdens such as residue, weakened tests, and unnecessary dependencies.This makes instability and long-term project degradation part of the evaluation loop.
  • Implications: Future repositories should expose agent-readable architecture, testing, task-state, verification, permission, entropy, and intervention affordances.The design question shifts toward what a repository must expose for an agent to work reliably.

Outlook

The paper frames AI Harness Engineering as a runtime-systems research program rather than a completed conclusion. It proposes evaluating harness effects across tasks, models, and long-horizon repository maintenance.

  • Outlook: The framework opens an empirical program in which the harness ladder functions as a controlled-ablation instrument and episode packages support population-level analysis.The protocol is intended to make harness effects statistically evaluable rather than establish a final result.
  • Multi-task evaluation: A balanced multi-task suite can stress distinct harness responsibilities, from verification and observability to entropy auditing, memory, failure attribution, recovery, and permissions.Different task classes are designed to illuminate different harness components.
  • Multi-model evaluation: A models × harness levels × tasks design can separate model effects from harness effects and reveal which components are model-agnostic or model-dependent.The proposed design explicitly tests interactions between model capability and runtime support.
  • Long-horizon evaluation: Long-horizon evaluation preserves repository state across tasks, audits accumulated entropy, and measures how prior episodes affect later success and maintenance burden.This tests whether richer harnesses reduce long-term maintenance costs as project memory and task state evolve.
  • AI-native repository design: Repository design becomes a research question because agent-readable documentation, test registries, and architecture maps shape context selection, recovery, and layer choice.The paper treats these as repository-artifact questions rather than questions about models or agents.
  • Runtime systems for agent-first development: The broader implication is a need for runtime systems analogous to operating systems but specialized for foundation-model agents, managing context, tools, memory, verification, permissions, recovery, oversight, entropy, cost, and risk.The paper names AI Harness Engineering as the missing layer and calls for building it.

Methods

The methods instantiate a controlled validation task across four harness levels while recording agent actions, context, verification, failures, interventions, and entropy. Deterministic adjudication then separates behavioral success, evidence quality, unsafe changes, assistance, and regression limitations.

  • Repository construction: The controlled repository contains a Node.js login application whose validator incorrectly accepts empty-string passwords, enabling objective checks and a layer-specific fix.The task requires rejecting the defect in the validator rather than applying a wrong-layer fix.
  • Harness instantiation: H0 through H3 progressively expose registries, guides, state and memory artifacts, deterministic checks, reproduction, attribution, verification, and reporting protocols.Evaluator notes remain hidden from the agent at every level.
  • Trace recording: The trace system records externally meaningful actions, tool outcomes, consulted memory, verification attempts, human interventions, and agent-introduced residue as structured episode data.The action trace captures meaningful operations rather than every internal token.
  • Deterministic behavioral checks: Three deterministic controller probes test empty passwords, valid credentials, and invalid non-empty credentials through short Node.js invocations.The checks are evaluator-side at all levels but agent-visible through the deterministic registry only at H3.
  • Outcome adjudication: Outcome labels distinguish autonomous verified success, unverified success, failure, unsafe invalidity, and assisted verified success according to deterministic checks, verification evidence, destructive edits, and human assistance.The taxonomy separates task behavior from evidence quality.
  • Full regression handling: Full regression runs use strict timeouts, and a timeout is recorded rather than silently retried or treated as success; at H3 it does not alone block autonomous verified success when deterministic coverage is complete and reported.Episodes run in isolated workspaces with fixed commits, explicit subprocess timeouts, and identical compute environments across levels.
Loading 2605.13357v1…