PaperlayerView exact revision on arXiv

Source-linked AI summary

Harness-Bench: Measuring Harness Effects across Models in Realistic Agent Workflows

Yilun Yao, Xinyu Tan, Chao-Hsuan Liu, Yaoming Li, Zhengyang Wang, Wenhan Yu, Zhewen Tan, Yuxuan Tian, Guangxiang Zhao, Lin Sun, Xiangzheng Zhang, Tong Yang

arXiv:2605.27922v1cs.AI

TL;DR

Existing agent benchmarks do not systematically isolate how harness configurations affect executable workflows. Harness-Bench evaluates model–harness pairings across shared tasks and conditions while preserving native execution behavior, finding substantial configuration-level variation and supporting configuration-level capability reporting.

  • Problem

    Existing benchmarks typically abstract away execution, evaluate complete systems, or hold the harness fixed, limiting diagnostic evidence about harness effects in realistic workflows.

  • Method

    Harness-Bench evaluates model–harness configurations on 106 sandboxed offline tasks under shared environments, budgets, timeouts, and evaluators while recording execution evidence.

  • Results

    Across 5,194 trajectories, Harness-Bench observes substantial variation across model–harness pairings in completion, process quality, efficiency, and failure behavior.

  • Takeaways & Limitations

    Agent capability should be reported at the model–harness configuration level rather than attributed to the base model alone.

  • Takeaways & Limitations

    The benchmark covers controlled sandboxed offline workflows, evaluates complete harness configurations, and partly relies on rubric-based or LLM-assisted process assessment.

Abstract

from arXiv · show

LLM agents are increasingly deployed as executable systems that use tools, modify workspaces, and produce concrete artifacts. In such workflows, performance depends not only on the base model, but also on the harness: the system layer that manages context, tools, state, constraints, permissions, tracing, and recovery. However, existing benchmarks typically abstract away execution, compare complete agent systems, or hold the harness fixed, making execution-layer variation difficult to study. We introduce Harness-Bench, a diagnostic benchmark for evaluating configuration-level harness effects in realistic agent workflows. Harness-Bench evaluates representative harness configurations across multiple model backends under shared task environments, budgets, and evaluation protocols, while preserving each harness's native execution behavior. The benchmark contains 106 sandboxed offline tasks constructed from practical agent-use patterns and manually reviewed for realism, solvability, oracle-checkability, and integrity. Each run records final artifacts, execution traces, usage statistics, and validator outputs, enabling analysis beyond final completion. Across 5,194 execution trajectories, we observe substantial variation in completion, process quality, efficiency, and failure behavior across model-harness pairings. These results suggest that agent capability should be reported at the model-harness configuration level rather than attributed to the base model alone. Our analysis further identifies recurring execution-alignment failures, where plausible reasoning becomes decoupled from tool feedback, workspace state, evidence, or verifiable output contracts. Harness-Bench provides a reproducible foundation for diagnosing and improving reliable, efficient, and auditable agent execution stacks.

1 Introduction

Harness-Bench addresses the lack of a diagnostic protocol for isolating harness effects in realistic agent workflows. It fixes external conditions while preserving native harness behavior, evaluates 106 sandboxed tasks, and analyzes 5,194 trajectories across completion, process, efficiency, and failure behavior.

  • Motivation: Existing benchmarks measure model capabilities or complete agent systems, but generally do not make the harness a variable of evaluation.This leaves configuration-level effects on success, cost, robustness, and traceability insufficiently characterized.
  • Benchmark design: Harness-Bench varies model–harness configurations under shared task environments, budgets, timeouts, and evaluators while preserving each harness’s native execution behavior.The resulting measurements are configuration-level diagnostics rather than causal decompositions of individual harness mechanisms.
  • Benchmark design: 106 sandboxed offline tasks cover realistic end-to-end workflows and were manually reviewed for realism, difficulty, solvability, and evaluation reliability.The suite uses task-specific configurations and evaluators while avoiding dependence on live services to improve reproducibility and independent scoring.
  • Contributions: 5,194 execution trajectories support analysis of completion, process quality, efficiency, and recurring failure symptoms across model–harness pairings.The benchmark includes task manifests, fixtures, evaluators, and execution traces.
  • Contributions: Harness-Bench supports reporting agent capability at the model–harness configuration level rather than attributing it to the base model alone.This conclusion follows from observed variation across model–harness pairings under the shared protocol.

2 Related Work

Prior benchmarks span static evaluation, executable environments, live online workflows, and broader execution infrastructure. Harness-Bench complements them by making the harness surrounding the model the evaluation variable in executable workflows.

  • LLM and agent benchmarks: Static benchmarks measure text-based capabilities, while agent benchmarks evaluate complete systems across software, terminal, web, and operating-system environments.The supplied related-work passages position these benchmarks as measures of model or end-to-end agent capability.
  • LLM and agent benchmarks: Long-horizon multimodal and live-web benchmarks extend evaluation toward persistent services, drifting state, and production-site complexity.These settings expose dimensions that sandboxed benchmarks do not directly cover.
  • Harnesses and harness engineering: Harness-Bench complements prior benchmarks by varying the harness around the model while fixing external task conditions and recording final artifacts and execution traces.Each evaluation combines a task, model backend, harness configuration, sandboxed environment, and evaluator.
  • Harnesses and harness engineering: Related systems research addresses infrastructure such as tool-use protocols, stateful frameworks, tracing, guardrails, memory, budget control, and recovery mechanisms.This work reflects growing attention to the execution stack that turns model outputs into external actions.

3 The Harness-Bench Benchmark

Harness-Bench evaluates model–harness configurations in executable workflows by varying the harness under shared task conditions while preserving native execution behavior. It combines sandboxed tasks, multi-source run evidence, and outcome- and process-level scoring for diagnostic comparison.

  • Harness-Bench studies model–harness configurations by varying the harness around a model in executable agent workflows.
  • A harness conditions model calls and converts model outputs into workspace actions through tools, context, permissions, budgets, tracing, and recovery.
  • The benchmark fixes task conditions, initial state, budgets, timeouts, and evaluators while preserving each harness’s native prompting, interfaces, state management, and recovery behavior.
  • 106 sandboxed tasks span eight workflow categories and require end-to-end deliverables checked from final workspace state and, when needed, execution traces.
  • Each run records final workspace state, execution trace, usage statistics, and validator outputs for completion, process, cost, permission, and failure analysis.
  • The aggregate score combines security, completion, and process measures, while separate metrics report robustness, tool use, consistency, tokens, and turns.

4 Experiments

Harness-Bench evaluates complete model–harness configurations under shared task conditions while preserving native execution behavior. Across the benchmark, results show substantial configuration-level variation in scores, process quality, efficiency, and harness dependence.

  • Experimental setup: The evaluation measures complete harness configurations under shared task conditions, preserving native prompting, tools, state management, and recovery behavior.Each trajectory uses fixed task, model, harness, workspace, budget, timeout, and evaluator conditions.
  • Experimental setup: 106 tasks, 6 configurable harnesses, and 8 API model backends produce 5,088 matrix trajectories, with 106 additional Codex trajectories.Codex is evaluated separately as a model-bound coding agent under its default model configuration.
  • Main results: 23.8 points separate NanoBot’s aggregate score of 76.2 from OpenClaw’s 52.4 among configurable harnesses under the same task set and model-backend pool.Codex scores 80.4 but is reported separately because it is not evaluated across the same backend matrix.
  • Main results: Higher-scoring harnesses tend to show stronger tool-use appropriateness, consistency, and robustness, while longer trajectories alone do not determine performance.NanoBot achieves the highest configurable-harness score while using fewer tokens than Hermes, ZeroClaw, NullClaw, and Moltis.
  • Harness dependence: Harness dependence is measured as variance across harness-level average scores for each backend over the fixed task suite, not repeated-run stochastic variance.The metric captures cross-harness variation under otherwise shared benchmark conditions.
  • Harness dependence: Stronger model backends tend to achieve higher mean scores and lower cross-harness variance, whereas weaker backends show greater sensitivity to the execution substrate.This pattern supports reporting results at the model–harness configuration level.
  • Harness dependence: Codex outperforms most GPT-backed configurable harnesses but remains slightly below NanoBot+GPT, serving as a practical reference rather than a controlled ablation.Category-level variation is larger for structured data analysis, tool sequencing, and workspace manipulation.
  • Interpretation: The experiments are descriptive measurements over a fixed task suite, with complete configurations and some LLM-assisted process assessment limiting interpretation.These caveats constrain claims about all possible agent workflows.

5 Analysis

The analysis frames failures as execution drift: locally plausible reasoning becomes decoupled from the artifacts, tools, evidence, state, or output contracts that evaluators check. It identifies recurring breakdowns at output and recovery boundaries and uses execution alignment to describe the needed correspondence.

  • Observed failure symptoms: Failure analysis uses oracle outcomes, process notes, failure notes, and structured fields to identify non-exclusive recurring symptoms among failed trajectories.A single trajectory may contain tool failure, missing artifact, and schema violation symptoms simultaneously.
  • Observed failure symptoms: The framework organizes symptoms by failures in output contracts, tool interfaces, evidence bases, committed artifacts, or continuation state.This separates executable-agent evaluation from final-answer-only evaluation.
  • Observed failure symptoms: Many failures occur when semantically plausible reasoning violates machine-checkable schemas, omits required ledgers, or fails to produce consumable artifacts.The issue is not necessarily task misunderstanding, but failure to render reasoning in verifiable form.
  • Observed failure symptoms: Other failures arise after partial progress when useful inputs or tool feedback are not converted into recovery, grounded claims, preserved state, or committed outputs.These failures occur despite relevant inspection or informative tool responses.
  • Execution alignment: Execution alignment measures correspondence among reasoning, observed workspace state, tool actions, and evaluator-checked conditions.The analysis treats breakdowns in this correspondence as execution drift.
  • Execution alignment: Harnesses affect performance partly by defining what counts as an obligation, observed evidence, recoverable failure, and completed work.Weak or implicit operational representations allow plausible reasoning to drift from judged task conditions.
  • Execution alignment: Harness-Bench evaluates whether the execution system carries reasoning into verified action by preserving correspondence among reasoning, workspace records, and evaluator checks.The analysis distinguishes this property from merely having more tools or a more permissive runtime.

6 Discussion

The discussion treats capability as a property of a model embedded in an execution system, while noting that stronger models still require reliable execution substrates. Harness-Bench is useful diagnostically, but its controlled offline scope limits direct claims about production performance or safety.

  • Harnesses as part of measured capability: Agent scores depend on environments, tools, tests, traces, and verifiers as well as on what the model can infer.This motivates interpreting measured capability at the model–harness level.
  • Harnesses as part of measured capability: Stronger models may need less prompt scaffolding, but still require permission boundaries, persistent state, interpretable traces, evidence records, and objective verification.The discussion recommends reporting both model and harness conditions for benchmark scores.
  • Limitations: Harness-Bench focuses on controlled sandboxed offline workflows, trading coverage of live services, user feedback, changing external state, and long-term production memory for reproducibility.Its scores are diagnostic measurements under a fixed protocol, not guarantees of real-world deployment performance or safety.

7 Conclusion

Harness-Bench makes execution-layer variation observable under fixed task conditions while preserving native harness behavior. Across 5,194 trajectories, it finds substantial differences across model–harness configurations and motivates configuration-level reporting.

  • Harness-Bench fixes external task conditions while preserving each harness’s native execution behavior.
  • Across 5,194 trajectories, the benchmark observes substantial differences across model–harness configurations.
  • The findings support reporting agent capability at the configuration level rather than by the base model alone.
  • The benchmark is intended to help diagnose and improve reliable, efficient, permission-aware, and auditable agent execution stacks.

B.1 Configurable Harnesses

The evaluation distinguishes configurable harnesses from model backends and from Codex, a model-bound coding agent. Harnesses retain native execution behavior while sharing task, budget, environment, and evaluation conditions.

  • Six configurable harnesses are evaluated under the main factorial setting, with categories describing design emphasis rather than benchmark performance.
  • Each configurable harness follows the same task environment, budget, and evaluation protocol while preserving its native execution behavior.
  • The API model backends span open-weight and closed-source frontier families and are paired with every configurable harness.
  • Codex is evaluated separately because it is a model-bound coding-agent stack that cannot be paired with arbitrary model backends.

C Category-Level Harness Dependence

Harness dependence varies across workflow categories under the fixed Harness-Bench protocol. It is greatest in workflows requiring structured manipulation, tool sequencing, workspace edits, and intermediate-state tracking, and lowest in more language-centric communication tasks.

  • Category-level harness dependence is measured as variance across configurable-harness average scores within each workflow category.The analysis uses harness-level averages rather than repeated stochastic runs, and category sizes differ.
  • Data, BI & Finance Analytics, Workspace/Tool Use, and Software Engineering show the largest cross-harness variance.These workflows often depend on structured data manipulation, tool sequencing, workspace edits, and intermediate-state tracking.
  • Office & Business Communication exhibits the lowest variance, suggesting that more language-centric tasks are less sensitive to harness configuration.
  • Harness effects are most visible when success requires alignment among reasoning, tools, state, and verifiable artifacts rather than language generation alone.

D Representative Harness-Bench Task Cards

Harness-Bench task cards specify workspace setup, objectives, expected artifacts, constraints, and oracle-grading signals for representative workflow families. Example tasks include document-and-spreadsheet business work and multi-round code repair.

  • Each task card summarizes workspace setup, the agent objective, expected artifacts, constraints, and the oracle-grading signal.
  • The document-and-spreadsheet business workflow is represented as a dedicated task-card family.
  • The multi-round code repair workflow is represented as another task-card family.
  • A code-repair task requires specific output files, minimal changes, preserved behavior, repair comments, and oracle grading based on completion, efficiency, and fix quality.

D.3 Database Migration Safety

The section presents task cards for database migration safety, policy-grounded customer support, and offline evidence auditing workflows. One workflow specifies concrete output artifacts, escalation constraints, and oracle checks.

  • The benchmark includes a task card for a database migration safety workflow.
  • The benchmark also includes task cards for policy-grounded customer support and offline evidence-auditing workflows.
  • The customer-support workflow requires routing decisions, reply templates, and escalation notes as workspace outputs.
  • Its constraints require conservative escalation for fraud holds, carrier contradictions, and delivered-status disputes, while VIP status cannot bypass evidence requirements or fraud holds.
  • Oracle grading checks ticket decisions, valid actions and priorities, policy-clause citations, escalation-team mapping, and reply-template coverage.
Loading 2605.27922v1…