PaperlayerView exact revision on arXiv

Source-linked AI summary

LongHorizon-Harness: Advancing Long-Horizon Agents for Real-World Tasks

Ziyu Ma, Hailang Huang, Shun Zou, Yong Wang, Shidong Yang, Yiming Hu, Fei Wei, XiangXiang Chu

arXiv:2608.01964v1cs.CV

TL;DR

Long-horizon agents struggle to preserve reliable task state as execution histories grow and completion assessment remains coupled to execution. LongHorizon-Harness separates state management from execution through independently audited MEA rounds, and reports consistent gains across benchmarks, models, and interaction domains.

  • Problem

    Long-horizon agents must sustain coherent progress across interdependent actions, but existing harnesses make task state difficult to track and allow incorrect completion judgments to influence later decisions.

  • Method

    LongHorizon-Harness maintains explicit task state outside execution, updates it only with independently verified environmental facts, and uses manager, fresh-context executor, and read-only auditor roles in an MEA loop.

  • Results

    LongHorizon-Harness delivers consistent gains across WeaveBench, OSWorld 2.0, and Terminal-Bench 2.1, with Qwen 3.7-Plus WeaveBench PassRate rising from 51.8% to 80.7%.

  • Takeaways & Limitations

    The results indicate that long-horizon agent capability depends not only on the model but also on the harness that organizes, verifies, and carries progress across rounds.

  • Takeaways & Limitations

    Benefits are smaller on several analytical categories when performance is dominated by an individual model capability.

Abstract

from arXiv · show

Large language model (LLM) agents increasingly undertake long-horizon tasks that require sustained reasoning, tool use, and revision across many interdependent steps. However, existing agent harnesses maintain task execution, task state, and completion assessment within a growing context, making the state difficult to track and allowing incorrect self-assessments to propagate into later decisions. We reformulate long-horizon execution as a task-state management problem and propose LongHorizon-Harness, which maintains the task state explicitly outside execution and updates it only with facts independently verified from the environment. Its Manage-Execute-Audit(MEA) loop uses a manager to maintain the task state and determine the next subtask, a fresh-context executor to perform it, and a read-only auditor to verify the resulting environment state before the next round. A lightweight AgentAdapter supports interchangeable model and harness backends without modifying their native agent loops. LongHorizon-Harness improves Qwen~3.7-Plus from 51.8% to 80.7% on WeaveBench, from 69.7% to 77.2% on Terminal-Bench~2.1, and from 2.8% to 8.3% on OSWorld~2.0. It also raises Claude Opus~4.7 from 20.0% to 34.3% on an OSWorld2.0 subset, demonstrating consistent gains across models, harnesses, and interaction domains.

1 Introduction

Long-horizon execution becomes unreliable as interdependent actions accumulate errors and growing context obscures task state. LongHorizon-Harness addresses this by separating explicit task-state management from execution and independently auditing each transition.

  • Motivation: Longer task horizons increase delegated work, but longer horizons alone do not make agent execution reliable.Advanced-agent task-completion horizons have doubled roughly every seven months, with a more recent trend of about four months.
  • Motivation: Long-horizon difficulty arises from sustaining coherent progress across many interdependent actions, including compounding errors, goal drift, and context rot.Earlier errors can distort later choices, while growing interaction history makes relevant information harder to retrieve and use.
  • Problem: Existing harnesses use the same growing context to execute tasks and maintain task state, making state increasingly difficult to track.Execution and completion assessment also remain coupled, allowing incorrect completion judgments to enter task state and influence subsequent decisions.
  • Approach: LongHorizon-Harness maintains explicit task state outside execution, updates it only with independently verified environmental facts, and derives each next subtask from that record and the original goal.Its MEA loop uses a manager, fresh-context executor, and read-only auditor; raw executor history is discarded after each round.

2 Method

LongHorizon-Harness executes long-horizon tasks through repeated MEA rounds, maintaining explicit audited state while isolating execution contexts. Separate manager, executor, and auditor roles constrain how state is proposed, changed, and verified.

  • 2.1 Overview: The harness advances a task through dynamically determined rounds, persisting only explicit task state and supporting audit reports across fresh execution episodes.The executor’s raw interaction trajectory is discarded after each round.
  • 2.1 Overview: Each MEA round has a manager construct a bounded contract, a fresh executor transform the environment, and an auditor inspect the result before state update.The loop can end when the audited state satisfies the task, progress is blocked, user input is required, or the round budget is exhausted.
  • 2.2 Manager: The manager owns persistent task state and makes decisions from the original task, current state, and accumulated audit reports without direct environment access.This prevents manager decisions from relying on unrecorded application or workspace observations.
  • 2.2 Manager: Task state records requirements, artifacts, and facts with completed, pending, blocked, or untrusted status and links to supporting audit evidence.Executor claims do not directly update persistent state; completion requires clean audit evidence.
  • 2.2 Manager: The manager selects an unresolved objective and creates a bounded contract containing its goal, acceptance criteria, constraints, dependencies, and relevant prior evidence.The contract is routed to a GUI or CLI executor according to the environment transition required.
  • 2.3 Executor: The executor is the only role permitted to intentionally modify the environment and reports actions, resulting state, artifacts, and encountered issues.Its execution report describes the outcome but does not establish contract completion.
  • 2.3 Executor: Each executor invocation is a fresh, budget-bounded episode that receives only current-round information and discards raw trajectory and internal reasoning afterward.The backend retains its native planning and tool-use loop through a common agent-adapter interface.
  • 2.4 Auditor: The auditor independently compares the resulting environment with the contract and records completion, integrity, and supported task-state findings.It uses read-only inspection, and audit conclusions must rely on direct environmental evidence rather than executor completion claims.

3 Experiments

LongHorizon-Harness improves long-horizon agent performance across computer-use and command-line benchmarks while producing model- and task-dependent cost changes. Its gains are strongest when tasks require preserving, verifying, and revising dependent environment states over extended trajectories.

  • Main Results: 80.7% WeaveBench PassRate, up from 51.8%, with Qwen 3.7-Plus and Claude Code under a matched comparison.Mean task score also increases from 0.702 to 0.835, with gains across all eight domains.
  • Main Results: 8.3% OSWorld 2.0 binary completion, up from 2.8%, while partial score rises from 21.5% to 35.2% with Qwen 3.7-Plus.The results indicate greater progress toward both partial and fully completed workflows.
  • Main Results: 35.3% binary completion and 66.9% partial score on a Claude Opus 4.7 OSWorld subset, up from 20.6% and 55.8%, respectively.Improvements with both Qwen 3.7-Plus and Claude Opus 4.7 indicate complementary contributions from the backbone and harness.
  • Main Results: 77.2% Terminal-Bench 2.1 success rate, up from 69.7%, showing gains when tasks are performed entirely through the command line.The framework also reaches 83.1% with GPT-5.6 Luna using Codex as the executor backend.
  • Cost and Computation: Auditing accounts for 19.4%, 24.8%, and 38.1% of tokens across WeaveBench, OSWorld 2.0, and Terminal-Bench 2.1, while managers account for only 2.8%, 2.0%, and 8.1%.Total cost varies by benchmark: 2.3× baseline tokens on WeaveBench, 3.6× baseline output tokens on OSWorld, and 24% fewer tokens on Terminal-Bench 2.1.
  • Task-Dependent Effectiveness: The largest gains occur in task categories requiring agents to preserve, inspect, and revise multiple dependent environment states over extended trajectories.Benefits are smaller when performance is dominated by individual capabilities such as visual perception, mathematical reasoning, coding, or algorithm design.
  • Agent Capability: LongHorizon-Harness raises Qwen’s mean WeaveBench Games score from 0.524 to 0.733 and recovers six near-total failures to scores between 0.30 and 0.92.The framework primarily raises the failure floor by recovering trajectories that would otherwise end in near-total failure.
  • Mechanisms and Case Studies: Independent auditing verifies environment state before updates, enabling recovery from stalled interactions and preventing plausible but non-compliant completion claims from entering persistent task state.Examples include escaping a repeated failed GUI interaction and verifying that all 15 headings have the required underlying style.

4 Conclusion

LongHorizon-Harness separates task-state management from environment interaction through a Manage–Execute–Audit loop. Across multiple benchmarks, interaction domains, and model backbones, it improves long-horizon task completion and supports the view that harness organization contributes to agent capability.

  • 4 Conclusion: LongHorizon-Harness maintains progress as explicit, audited task state while executing each subtask in a fresh context.Only independently verified outcomes persist across rounds.
  • 4 Conclusion: Experiments show consistent improvements across hybrid GUI–CLI workflows, professional desktop tasks, pure command-line environments, and different model backbones.
  • 4 Conclusion: Long-horizon agent capability depends not only on the underlying model but also on the harness that organizes, verifies, and converts local capabilities into end-to-end completion.

A Detailed Experimental Setup

The experimental setup evaluates Qwen 3.7-Plus through Claude Code on WeaveBench’s mixed-interface long-horizon tasks. It preserves the benchmark’s standard environment while adding restricted evidence preservation for auditing.

  • A Detailed Experimental Setup: WeaveBench contains 114 long-horizon computer-use tasks spanning eight domains and requiring GUI and CLI interactions within the same workflow.
  • A Detailed Experimental Setup: Qwen 3.7-Plus is evaluated with the Claude Code runtime, with each task allowed up to 25 Manage-Execute-Audit rounds.The executor timeout is 1800 seconds per round, while manager and verifier timeouts are 300 seconds.
  • A Detailed Experimental Setup: The setup preserves WeaveBench’s screenshot and desktop-actuation tools while adding save_screenshot for restricted evidence preservation.save_screenshot saves the currently visible environment without adding environment-control capability or exposing additional information.
  • A Detailed Experimental Setup: Screenshots are uploaded through an image URL proxy so multiple visual inputs do not create oversized base64-encoded request bodies.

A.2 OSWorld 2.0

The OSWorld 2.0 setup evaluates desktop workflows in the official Docker-based environment using GUI and CLI tools, with Binary and Partial Accuracy as complementary metrics. Results are averaged over three independent runs per task.

  • A.2 OSWorld 2.0: OSWorld 2.0 contains 108 professional desktop workflow tasks evaluated in the official Docker-based Ubuntu environment.The benchmark reports Binary Accuracy and Partial Accuracy.
  • A.2 OSWorld 2.0: LongHorizon-Harness uses a hybrid GUI+CLI tool pool, combining desktop operations with shell commands for filesystem inspection, scripting, and long-context processing.
  • A.2 OSWorld 2.0: Binary Accuracy counts only tasks with final score 1, whereas Partial Accuracy averages fine-grained scores across all tasks.
  • A.2 OSWorld 2.0: The evaluation uses three independent trials per task and reports each task’s average score against the Claude Code baseline and official leaderboard results.

B.1 Detailed Results on the OSWorld 2.0 Opus 4.7 Subset

On a 34-task OSWorld 2.0 subset with Claude Opus 4.7, LongHorizon-Harness outperforms the standard single-action GUI baseline on both partial and binary performance.

  • B.1 Detailed Results on the OSWorld 2.0 Opus 4.7 Subset: 35.3% binary accuracy versus 20.6% is achieved by LongHorizon-Harness on the 34-task Claude Opus 4.7 subset.Partial score also rises from 55.83% to 66.86%.
  • B.1 Detailed Results on the OSWorld 2.0 Opus 4.7 Subset: 66.86% partial score versus 55.83% is achieved by LongHorizon-Harness on the same OSWorld 2.0 subset.The baseline uses the standard single-action GUI setting, while the harness uses a hybrid GUI+CLI tool pool.
  • B.1 Detailed Results on the OSWorld 2.0 Opus 4.7 Subset: The comparison contrasts the standard single-action GUI baseline with LongHorizon-Harness using a hybrid GUI+CLI tool pool.

B.2 Fine-Grained Breakdown by Benchmark and Task Type

The fine-grained breakdown shows that LongHorizon-Harness helps across WeaveBench domains, especially when progress is represented by verifiable environment state, while remaining failures involve difficult-to-close conditions.

  • Verifiable state: The largest WeaveBench gains appear when progress can be represented as verifiable environment state.Examples include screenshots with defined metadata, application configuration, structured files, and cross-checked evidence from multiple interfaces.
  • Residual limitations: Remaining failures concentrate on hidden performance thresholds, embodied visual precision, temporal video evidence, and ambiguous task semantics.These conditions are difficult to close under the available verifier.
  • WeaveBench domain-level results: LH-Harness improves the mean score in seven of eight WeaveBench domains and increases pass rate in every domain.The largest gains occur in Design, Spatial/3D Applications, and Games; Desktop mean score slightly decreases despite a higher pass rate.
  • Games: Games improve most on tasks where the baseline often fails to establish a stable state representation.Mines, Stockfish puzzle analysis, Quadrapassel autoplay, PokerTH equity play, and SuperTux level repair move from near-zero or zero baseline scores to nontrivial completion.

B.2.2 OSWorld 2.0

On OSWorld 2.0, LongHorizon-Harness improves every capability-tag group, with the strongest gains where user information or task progress can be converted into durable, verifiable state.

  • Capability-tag results: LH-Harness improves all six OSWorld 2.0 capability-tag groups, although gains and residual failures differ substantially.The tags overlap, so one task may contribute to multiple rows.
  • Strongest gains: Streaming interaction tasks rise from a baseline score of zero to 0.500 on average.Human-in-the-loop tasks also improve strongly when user-provided information becomes an explicit acceptance constraint.
  • Task-property differences: Symbolic tutorials are easier to audit than embodied tutorials involving video timelines, CAD geometry, or fine mouse operations.Commands, tables, file names, and web fields can be decomposed into auditable steps, whereas spatial and motor-level requirements remain difficult.
  • Task-property differences: Visual-task gains are strongest when visual requirements can be rewritten as file, code, metadata, or search problems.Purely spatial or motor-level requirements remain limited by visual localization and cursor control.
  • Verification boundary: Implicit state inference works best when an authoritative closure exists in a file, database entry, application setting, or evaluator-visible artifact.Without such closure, plausible evidence may be mistaken for completed evidence.

B.2.3 Terminal-Bench 2.1

Terminal-Bench results indicate that explicit task contracts and independent auditing are most valuable for stateful, difficult tasks with hidden acceptance conditions.

  • Category-level results: 0.889 versus 0.593: LH-Harness produces the largest Terminal-Bench category gain in system administration.The category includes installation, service configuration, PATH management, build outputs, and persistent environment side effects.
  • Difficulty-level results: LH-Harness gains more on hard Terminal-Bench tasks than on medium tasks.The marginal value of explicit state management is smaller when a baseline can finish within one trajectory.
  • Tag-level results: The strongest positive tags are images, version-control, system, and sys-admin.These tags share correctness conditions that can be checked against durable artifacts such as hashes, repository history, executables, service state, logs, or filesystem layout.
  • Residual limitations: Negative tags expose limits when correctness depends on hidden thresholds, ranking semantics, temporal localization, or unrecoverable evaluation conventions.In such cases, careful execution can still yield a confidently verified wrong answer if the contract is misinterpreted.
  • Mechanism: The case studies frame the same mechanism as explicit task state, fresh-context execution, and independent verification rather than benchmark-specific heuristics.They show recurring failure modes being handled through the Manage-Execute-Audit structure.

C.1 Cross-Domain WeaveBench Cases

Across WeaveBench cases, LongHorizon-Harness converts GUI, document, game, web, data, operations, CAD, and design work into auditable state transitions supported by artifacts and cross-interface evidence.

  • Desktop workflow: Desktop workflows preserve completed note artifacts, recover from failed GUI interactions, and verify rendering bugs with screenshot evidence.Failed shortcuts and imprecise clicks become unresolved task-state changes rather than being lost in interaction history.
  • Document processing: Document processing verifies semantic heading normalization through ODT XML rather than relying on visually plausible formatting.The workflow inspects the application state, applies a reproducible macro, and audits the resulting structure.
  • Game analysis: Game analysis cross-validates an illegal PGN transition by combining GUI replay, the stopped board state, and file-level report checks.Neither GUI observation nor file inspection alone is sufficient for the complete artifact.
  • Web diagnostics: Web diagnostics combine browser charts, tooltip values, and Wireshark packet inspection into one audited evidence chain.Each observation is recorded as a verified fact that guides the next step.
  • Data analysis: Data-analysis audits reject screenshots whose filenames do not match the captured UI state and assign focused repair subtasks.Artifact presence becomes progress only after the auditor confirms artifact validity.
  • Operations, CAD, and design: DevOps and CAD workflows link symptoms or semantic layers to repairs across UI, CLI, file, rendering, and screenshot evidence.Design cases similarly ground visual comparisons in actual GIMP rendering state, histograms, and side-by-side outputs.
Loading 2608.01964v1…