PaperlayerView exact revision on arXiv

Source-linked AI summary

StateM: Reaching 95.3% Raw Accuracy, or a \$15 Frontier Run, on Terminal-Bench 2.1 via Harness Scaling

Ziheng Qin, Yaxin Lu, Zhangyang Atlas Wang, Kai Wang

arXiv:2608.15089v1cs.AI

TL;DR

Long-horizon agents can fail despite solving individual steps because they lose state, skip procedures, or stop without verifying completion. StateM adds durable states, phase-local context, checked transitions, and recoverable runbooks, reaching 95.3% raw accuracy on Terminal-Bench 2.1 with GPT-5.6 Sol xhigh.

  • Problem

    Long-horizon agents can fail despite knowing how to solve constituent steps because they lose mutable state, skip checks, or stop before verifiably completing the task.

  • Method

    StateM is an agent-native control layer that externalizes execution state, refreshes phase-local context, and enforces checked transitions through durable runbooks and procedural practices.

  • Results

    95.3% raw accuracy was achieved across 445 Terminal-Bench 2.1 trials with GPT-5.6 Sol xhigh, including at least one success on every task.

  • Takeaways & Limitations

    StateM supports harness scaling as a capability axis complementary to model scaling, with quality and cost frontiers demonstrated across agent models and tasks.

  • Takeaways & Limitations

    The GPT-5.5 reference was a public model-matched result rather than a newly rerun, agent-version-matched A/B control, and the Sol Ultra value used a different high-compute configuration.

Abstract

from arXiv · show

Long-horizon agents can fail even when their underlying models can solve the constituent steps. They may lose track of mutable state, fail to reactivate lessons from earlier executions, skip known procedures, or stop prematurely. We bet on harness scaling to improve the execution system around an agent without changing its model weights. We introduce StateM, an agent-native runtime that organizes execution around durable states, phase-local context, checked transitions, recoverable runbooks, and versioned procedural practices that agents and users can inspect together. On Terminal-Bench 2.1, StateM raises GPT-5.5 xhigh to 92.1\%, versus 83.1\% reference and GPT-5.6 Sol Ultra at 91.9\%. The runbook transfers unchanged to GPT-5.6. With GPT-5.6 Sol xhigh, StateM reaches 95.3\% raw accuracy across 445 trials and succeeds on all 89 tasks at least once. The frozen profile raises GPT-5.6 Luna from 76.7 to 85.4\%, above the 84.9\% Sol xhigh reference. Using the same runtime, runbook structure, and golden rules, less than \$38 of adaptation raises DeepSeek-V4 Flash from 82.7 to 88.1\% under standard timeouts and to 89.1\% on an 88-task common core. Extending only the remaining latency-sensitive task matches the reported 88.8\% GPT-5.6 Sol max result. Final-score API usage is about \$15 versus \$574.68 for the GPT reference; total DeepSeek expenditure is \$52.22. On BusinessBench, family-specific runbooks built on development sets yield held-out gains of 0.55 macro and 1.34 micro points; two mechanism-matched families improve by 10.04 points. Concrete rules generalize when tasks share execution structure, while the control methodology applies broadly. StateM turns selected postmortem findings into persistent, executable preconditions and practices, making learned controls explicit and enforceable through stateful controls. Code at github.com/henryqin1997/statem.

1 Introduction

StateM frames harness scaling as a complement to model scaling: an agent-native runtime externalizes procedural control through durable states, checked transitions, and shared runbooks. Across model versions, providers, and task families, the system improves completed-work reliability while concrete practices transfer locally or with inexpensive adaptation.

  • Motivation: StateM targets long-horizon failures in which capable models lose state, skip checks, repeat unproductive actions, or stop before verifiable completion.These failures matter because workflows and benchmarks evaluate completed jobs, not partial knowledge of constituent steps.
  • StateM runtime: StateM uses human-readable YAML runbooks with durable states, valid transitions, state-local instructions, hooks, checks, and recovery rules.Agents and users can inspect, audit, and revise the shared machine-operable artifact through the same execution environment.
  • StateM runtime: Each state refreshes phase-specific instructions and durable task information, while explicit exit conditions require executable checks or auditable semantic attestations.This design addresses control-signal dilution and mutable-state ambiguity by externalizing procedural state and checking evidence.
  • Results and scope: 95.28% raw accuracy was achieved by GPT-5.6 Sol xhigh on Terminal-Bench 2.1, with 424 successful trials out of 445 and at least one success on every task.GPT-5.5 xhigh with StateM reached 92.1% versus the 83.1% GPT-5.5 reference, while the frozen runbook raised GPT-5.6 Luna from 76.7% to 85.4%.
  • Results and scope: 88.09% was reached on the full 89-task benchmark by the adapted DeepSeek system after less than $38 of API expenditure, rising to 89.09% on the disclosed 88-task common core.The experiments separately test within-model improvement, frozen cross-generation transfer, adapted cross-provider transfer, and held-out task generalization; concrete practices transfer locally or require inexpensive adaptation.

2 Related Work

StateM combines long-horizon planning, stateful orchestration, agent memory, and harness adaptation through an agent-native, jointly editable execution representation. Its distinctive position is a shared runbook that preserves broad agent autonomy while enforcing auditable state transitions and supporting direct user intervention.

  • Positioning StateM: StateM’s novelty lies in combining four workstreams through an agent-native, jointly editable execution representation rather than introducing any component in isolation.The four workstreams are long-horizon planning, stateful agent orchestration, long-lived agent memory, and automated harness adaptation.
  • Stateful and graph-based orchestration: StateM’s lightweight control representation makes state boundaries both refreshed control contexts and auditable transition contracts without decomposing the primary agent into a sequence of separately prompted nodes.This design responds to evidence that heavy orchestration can fragment reasoning and introduce routing errors.
  • Stateful and graph-based orchestration: StateFlow and durable graph runtimes provide explicit states, transitions, persistence, checkpointing, interruption, and recovery, but conventionally keep the primary control artifact external to the executing agent.StateM instead exposes its control layer within a general-purpose CLI agent’s ordinary tool environment.
  • CLI agents, soft plans, and user control: StateM organizes plans, rules, checks, hooks, recovery paths, and history into a shared runbook that both the agent and user can inspect and edit.User intervention therefore operates through the same control artifact as runtime execution rather than only through additional natural-language corrections.
  • Positioning StateM: StateM targets the combination of broad autonomy, enforceable state transitions, and a jointly visible and editable control artifact, while addressing within-run procedural execution rather than longitudinal aging mechanisms.The paper describes lifespan diagnostics and StateM runbook adaptation as complementary future directions, and states that StateM does not itself solve compression, retrieval, or maintenance aging.

3 StateM: Agent-Native Control for Long-Horizon Execution

StateM is an agent-native execution substrate that preserves within-phase autonomy while making cross-phase progress explicit, persistent, and checkable. Its reusable runtime externalizes state, transitions, auditing, and recovery, while runbooks encode workflow-specific procedural knowledge.

  • Runtime and runbooks: The runtime provides reusable persistence, transition validation, hooks, history, and recovery, while versioned runbooks specify workflow-specific phases, instructions, checks, and repair policies.Agents and users share the same CLI-accessible runbook artifact, and empirical gains reflect the combined runtime and control profile rather than the state-machine abstraction alone.
  • Phase-level control: StateM organizes work as phase-level states, allowing agents to reason and use tools freely within phases while enforcing explicit transitions between them.States can represent planning, implementation, checking, review, repair, and handoff rather than individual model calls or tool actions.
  • Context and persistence: At state entry, hooks refresh the authoritative phase, valid transitions, local instructions, and relevant durable progress, reducing reliance on inferred context from prior execution.Entry procedures may inject prompts, run setup code, initialize files, or load compact progress records.
  • Checked transitions: Before a handoff, exit hooks and checks evaluate configured conditions, persist progress or artifacts, and block transitions when work is incomplete, verification fails, or external blockers remain.Checks range from host-evaluated commands and predicates to manual, checklist, message, and non-deterministic LLM reviews.
  • Recovery limits: StateM is checked, logged, and recoverable rather than fully transactional: it delays runtime-state commits until pre-commit operations succeed but cannot roll back arbitrary external side effects.Recovery applies to the execution record and configured repair procedure, while hook safety, compensation, retries, and permissions remain host responsibilities.

4 Harness Scaling in Practice: Two Frontiers and a Hierarchy of Transfer · 4.1 Evaluation Regimes and Evidence Boundaries · 4.2 Same Model, Better Harness: A Model-Generation-Sized Gain

StateM evaluates harness scaling across fixed-model lift, frozen transfer, adapted transfer, and held-out generalization, with transfer becoming more abstract as model or task distance increases. On Terminal-Bench 2.1, the harness substantially outperforms matched references, producing gains larger than the observed model-generation shift.

  • 4 Harness Scaling in Practice: Two Frontiers and a Hierarchy of Transfer: StateM combines the base agent, generic runtime, and control profile, so reported results measure the complete execution system rather than isolating the state-machine runtime.The measured system includes evolved prompts, routing, checks, and practices.
  • 4 Harness Scaling in Practice: Two Frontiers and a Hierarchy of Transfer: The evaluation spans fixed-model lift, frozen within-family transfer, adapted cross-provider transfer, and held-out task generalization.Exact profiles transfer to nearby models; across providers, runtime and runbook structure remain reusable while profiles require adaptation.
  • 4.1 Evaluation Regimes and Evidence Boundaries: Terminal-Bench 2.1 contains 89 tasks evaluated over five trials each, with trial-level success rate and five-trial task coverage as complementary metrics.Coverage counts tasks solved at least once across five observed trials and is also reported as Pass@5.
  • 4.1 Evaluation Regimes and Evidence Boundaries: The GPT-5.6 Sol and Luna evaluations freeze the versioned profile and activation policy, preventing target-model outcomes from revising prompts, routing, checks, defaults, or repair policies.Golden rules require minimal reusable control, routing from visible task semantics, and separation of development feedback from frozen evaluation.
  • 4.2 Same Model, Better Harness: A Model-Generation-Sized Gain: 92.1% is GPT-5.5 xhigh with StateM versus 83.1% for the GPT-5.5 Codex reference, while StateM solves 88 of 89 tasks at least once.The result is a nine-point difference and 98.9% five-trial task coverage without changing model weights.
  • 4.2 Same Model, Better Harness: A Model-Generation-Sized Gain: 95.3% raw accuracy is GPT-5.6 Sol xhigh with StateM versus 84.9% for the GPT-5.6 Sol xhigh reference, and the system solves all 89 tasks at least once.The difference is 10.38 points, or 10.4 points at one-decimal precision; 95.28% is the raw public-submission result.
  • 4.2 Same Model, Better Harness: A Model-Generation-Sized Gain: 1.8 points is the GPT-5.5-to-GPT-5.6 Sol reference shift, compared with +9.0 points for StateM on GPT-5.5 and +10.4 points on GPT-5.6 Sol.The section concludes that harness scaling changes completed-task performance more than the observed generation-to-generation model shift.

4.3 Transfer Follows Model Distance · 4.4 A $15 Frontier Run: Moving the Cost–Accuracy Curve

StateM transfers unchanged across nearby GPT generations but requires provider-specific adaptation across the GPT–DeepSeek boundary. That adaptation raises DeepSeek-V4-Flash to 88.09% under standard timeouts and 89.09% on the common core, while final-score evidence costs $15.20.

  • 4.4 A $15 Frontier Run: Moving the Cost–Accuracy Curve: Reusable runbook structure and development principles make provider adaptation cheap enough to move a lower-cost model upward on the quality–cost frontier.Exact controls do not survive the provider boundary unchanged, but the harness remains economically reusable.
  • 4.3 Transfer Follows Model Distance: 85.4% raises GPT-5.6 Luna from 76.7%, a gain of 8.7 points, exceeding the 84.9% Sol xhigh reference.The transferred profile uses the lower-cost model tier without GPT-5.6-specific control changes.
  • 4.3 Transfer Follows Model Distance: 82.0% under direct frozen transfer falls below DeepSeek-V4-Flash’s 82.7% baseline, showing that the exact GPT-developed profile fails across the provider boundary.Provider-specific behavior remains consequential even with the task set and control interface fixed.
  • 4.4 A $15 Frontier Run: Moving the Cost–Accuracy Curve: Provider-specific adaptation raises DeepSeek-V4-Flash to 88.09% on the full 89-task standard-timeout evaluation from an 82.7% baseline.The adaptation retains the generic runtime, runbook structure, applicable controls, golden rules, and failure-analysis loop.
  • 4.4 A $15 Frontier Run: Moving the Cost–Accuracy Curve: 89.09% results on the 88-task latency-stable common core using the same 392 successes, excluding only gpt2-codegolf.The common-core denominator is 440 trials.
  • 4.4 A $15 Frontier Run: Moving the Cost–Accuracy Curve: 88.76% is the descriptive full-suite aggregate after evaluating gpt2-codegolf with an extended timeout, matching the separately reported GPT-5.6 Sol max score at one-decimal precision.gpt2-codegolf is 0/5 under the standard timeout but 3/5 with the extended per-task timeout.
  • 4.4 A $15 Frontier Run: Moving the Cost–Accuracy Curve: $15.20 in realized API charges covers the complete DeepSeek final-score evidence, while adaptation costs $37.02 and total recorded expenditure reaches $52.22.The DeepSeek evidence uses 2.65% of the public GPT-5.6 Sol max submission’s recorded $574.68 model cost.

4.5 Task Generalization Follows the Control Boundary

StateM generalizes on BusinessBench when its controls match a family’s execution mechanism and completion boundary, rather than when one workflow is applied universally. Matched families improve substantially, while mismatched profiles can cause negative transfer until refined around the relevant obligations or invariants.

  • Held-out transfer: 0.55 points: frozen StateM transfer raises equal-family held-out macro from 84.67% to 85.22%, while micro rises 1.34 points from 84.44% to 85.78%.The profiles are developed on one split and evaluated on untouched held-out instances from the same family.
  • Mechanism-matched gains: 10.04 points: structurally matched Budget Approval and Machine Operating families raise equal-family held-out macro from 71.91% to 81.94%.Budget Approval improves from 62.91% to 75.12% (+12.21), while Machine Operating improves from 90.79% to 100.00% (+9.21).
  • Negative transfer: −2.78 and −3.70: RefactorBench falls from 80.56% to 77.78%, and WooCommerce Stock falls from 92.59% to 88.89% under mismatched frozen profiles.Both failures indicate that the control was attached to the wrong execution boundary, rather than simply being too weak.
  • Selective refinement: 79.17%: a thinner RefactorBench profile raises matched-rerun overall performance from 76.39% to 79.17% by closing direct code-migration obligations.The revised profile tracks visible obligations, verifies signature and call-shape changes, and closes stale repository references.
  • Control boundary: 90.12%: replacing WooCommerce’s generic procedure with entity, destination, and irreversible-side-effect invariants raises matched-rerun overall performance from 86.42% to 90.12%.WebArena remains largely tied under mutation-oriented controls, while WebTest is nearly saturated at 99.75% overall.

4.6 Failure-Boundary Evidence: Where StateM Intervenes

StateM targets three long-run failure gaps with phase-local knowledge activation, checked procedural handoffs, and versioned lessons. Task-level evidence shows its strongest interventions occur at consequential boundaries, including a configure-git-webserver case where verification gates convert a 0/5 baseline into a repairable, evidence-driven run.

  • Failure gaps and controls: StateM addresses epistemic, procedural-compliance, and procedural-memory gaps with distinct controls.In_hooks reactivate phase-relevant knowledge; checked transitions and evidence requirements block incomplete handoffs; versioned practices carry lessons into later runs.
  • Boundary interventions: The largest task-level gains cluster at service readiness, contract satisfaction, preservation, and evidence-closure boundaries.Table 3 associates each representative gain with the active StateM control, but does not constitute a component ablation.
  • Configure-git-webserver: 0/5 baseline performance on configure-git-webserver reflects unreliable preservation and validation of the required end-to-end live state.The baseline agent can configure Git, SSH, hooks, and an HTTP server, but does not reliably preserve and validate the live state.
  • Configure-git-webserver: Fresh consumer-facing evidence gates final handoff by requiring a clone–commit–push–curl path before verification can complete.If verification perturbs the environment, StateM keeps the run repairable until final-state consistency is restored.
  • Persistent control: StateM preserves phase-level agent freedom while adding persistent controls where recoverable errors might otherwise become final.Epistemic hooks activate knowledge, checked transitions enforce procedure completion, and versioned practices preserve lessons across runs.

4.7 Harness Learning as Selective Procedural Memory · 4.8 Operational Endurance: A 22-Hour Development Run

StateM treats harness learning as selective, versioned procedural memory: failures and feedback become executable controls only after abstraction and validation. Its durable state also sustains a 22-hour development run across context changes without implying unbounded execution.

  • 4.7 Harness Learning as Selective Procedural Memory: Practices encode reusable interventions—instructions, checks, constraints, activation conditions, or verification actions—in versioned profiles that persist across executions.The runbook functions as both a runtime control surface and selective memory, making lessons inspectable and executable.
  • 4.7 Harness Learning as Selective Procedural Memory: Learning loops from execution and feedback through cause attribution, reusable abstraction, profile changes, and revalidation.The same agent can act as executor and maintainer, modifying prompts, evidence checks, transitions, or repair paths.
  • 4.7 Harness Learning as Selective Procedural Memory: Family profiles require stronger-agent review to assess whether lessons from development instances generalize and remain compatible before shared adoption.This abstraction step is tested in frozen one-shot evaluation.
  • 4.7 Harness Learning as Selective Procedural Memory: 89 heterogeneous tasks require sparse routing through a thin shared runbook, semantic routing, and late state-local checks to limit control drift.Task-by-task repair can add unnecessary procedure or damage other tasks.
  • 4.7 Harness Learning as Selective Procedural Memory: BusinessBench shows that selective memory beats procedure accumulation: effective controls can be smaller, invariant-specific, lighter, or absent depending on task structure.Examples include obligation-and-closure for RefactorBench, cross-system invariants for WooCommerce, a lighter query-state boundary for WebArena, and no StateM workflow for Attendance.
  • 4.7 Harness Learning as Selective Procedural Memory: Failure-driven memory can preserve post hoc conventions as universal practices, including target-frame precision defaults and verifier-selected DNA insertion boundaries absent from task descriptions.Such behavior may agree with evaluators while deriving semantics from benchmark feedback rather than the stated contract.
  • 4.7 Harness Learning as Selective Procedural Memory: Valid failures can still yield incorrect abstractions, so stronger-agent or human review must filter experience before durable control is created.The design objective is to remember consequential boundaries rather than every failure trace.
  • 4.8 Operational Endurance: A 22-Hour Development Run: 22 hours of development were sustained across long history, context refresh or compaction, and stop-hook continuation using durable phase, transition, obligation, and recovery records.Slowdown came from unfolded terminal output, not loss of StateM control state.

5 Discussion: Runtime Design for Harness Scaling

StateM’s harness-scaling design requires an agent-native, enforceable runtime that is also cheap to modify as control policies evolve. The present work focuses on single-agent execution and leaves role-isolated multi-agent runtimes for future work because they raise additional coordination, state-sharing, and permission-design questions.

  • Runtime design: Harness scaling requires a runtime that is agent-native, enforceable, and cheap to modify as control policies evolve.Conventional state-machine and graph-based runtimes offer explicit orchestration and strong control but externalize reasoning into manually constructed nodes, edges, and handlers.
  • Scope: The framework focuses on controlling a single autonomous agent rather than extending the runtime to multi-agent execution.The same separation of state, context, and permissions motivates multi-agent runtimes, but those extensions are still being explored.
  • Limitations: Role-isolated multi-agent runtimes are deferred because they introduce separate coordination, state-sharing, and permission-design questions.The present work therefore restricts its analysis to single-agent control flows.

6 Conclusion

StateM presents harness scaling as a complementary capability axis that makes execution state, procedural control, and consequential transitions explicit and inspectable. Its results show strong quality gains, reusable runtime methodology, and model- and workflow-dependent practices rather than universal transfer.

  • Core contribution: StateM externalizes mutable execution state, refreshes phase-local context, and checks consequential transitions within a unified CLI-agent reasoning loop.These agent-native, enforceable controls make harness scaling feasible in efficiency and cost.
  • Empirical frontiers: 95.3% raw accuracy: GPT-5.6 Sol xhigh succeeds on 424 of 445 Terminal-Bench 2.1 trials and reaches at least one success on every task.The result uses a runbook frozen from GPT-5.5 and remains a public, pre-adjudication submission rather than a finalized leaderboard result.
  • Transfer and generalization: Procedural control transfers unchanged across GPT-5.5 and GPT-5.6, while DeepSeek-V4 Flash requires adapted model-dependent practices.The runtime and failure-driven development methodology remain reusable across providers.
  • Transfer and generalization: BusinessBench shows positive but modest aggregate generalization, with the largest gains where control is sparse and matched to an explicit execution boundary.This supports applying concrete rules when tasks share execution structure rather than assuming universal workflow transfer.
  • Implications and limitations: StateM targets three operational failures: unavailable knowledge, unactivated prior lessons, and incomplete procedures, using state-local context, versioned practices, and checked transitions.Its value is making failures inspectable and selected lessons reusable, not making every model universally stronger or fitting one runbook to every workflow.

A Runbook example

The coding-agent runbook organizes work into durable, checked phases: load context, plan, execute, review, and hand off. It also supports scoped dynamic checks and a controlled session-refresh loop when later execution needs cleaner context.

  • Initialization: The runbook begins by loading durable project context and using StateM state controls to execute the current step without relying on normal /clear.Agents read the task description, progress.md, StateM history, architecture notes, and project rules before planning, then use session_refresh and safe compaction when needed.
  • Planning: Planning must produce a concrete, reviewed implementation plan before transfer to execution.The plan is generated from task context, current progress, architecture notes, and project rules, with a predicate requiring that it be concrete enough to execute.
  • Execution: Execution keeps edits scoped, updates progress.md, runs verification, and can register task-specific current-entry checks through StateM dynamic commands.The runbook checks plan alignment, verification, and unrelated-file scope before leaving execution.
  • Review and handoff: Review compares the implementation with plans, documentation, constraints, and golden rules, returning to execution for fixable gaps or handing off documented risks and blockers.The handoff summarizes changes, verification, current state, known risks, and the next command, grounded in progress.md and StateM history.

B Our Experiment Equipment and Budget

The experiments primarily used a personal Codex Pro plan and a 2025 MacBook Pro with an M4 chip, staying below the 200$ budget. Formal submissions instead used AWS m7i.4xlarge because Daytona and the MacBook encountered compatibility or timeout issues.

  • Primary resources and budget: Less than 125$ of the 200$ budget was actually used through the personal Codex Pro plan.The main local computation resource was a 2025 MacBook Pro with an M4 chip.
  • Formal-run infrastructure: AWS m7i.4xlarge supported formal submissions because Daytona had frequent sandbox, verifier, network, and PyTorch-install timeouts.The personal MacBook Pro was incompatible with the required tune-mjcf kernel.

C Additional Figures and Tables

The additional figures illustrate how long execution sequences dilute critical plan tokens and how append-only state maintenance becomes ambiguous as tasks involve loops and branches. The accompanying tables summarize system-level results and generalized workflow controls distilled into StateM.

  • Attention allocation: Critical plan tokens accumulate sparsely but receive diluted attention among many execution tokens in long runs.The plan operates at a higher abstraction level, while execution tokens dominate the sequence.
  • State maintenance: Simple linear tasks remain recoverable because few appended state updates make each item’s latest version easy to locate.The latest value can be recovered without ambiguity when state updates are sparse.
  • State maintenance: Tasks with loops and branches make current state ambiguous because repeated iterations append many interleaved versions of the same item.Append-only context prevents updates from being written back in place, making reconstruction difficult as complexity increases.
  • System-level results: Table 4 reports Terminal-Bench 2.1 system-level results across reference, StateM, and transfer regimes.Its descriptive DeepSeek aggregate replaces only five gpt2-codegolf trials with the disclosed extended-timeout evaluation.
  • Workflow controls: Table 5 distills generalized Businessbench workflow controls into StateM, targeting reusable failure modes rather than benchmark-specific answers.Hard gates apply to evidence-bearing invariants, while potentially conflicting preferences remain nonbinding.
Loading 2608.15089v1…