Source-linked AI summary
StateAct: Program State, before Pixels, for Long-Horizon Computer-Use Agents
Yan Yang, Xiangru Jian, Ziyang Luo, Zirui Zhao, Yutong Dai, Ziji Shi, Hanshu Yan, Jun Hao Liew, Silvio Savarese, Junnan Li
TL;DR
Long-horizon computer-use agents rely heavily on screenshots, although pixels can hide the underlying program state and whether results are correctly saved. StateAct makes program state the primary interface while retaining GUI interaction and independent verification, raising Claude Opus 4.8’s binary success from 20.6% to 26.9% at roughly ninefold lower cost.
Problem
Screenshots can hide task-critical program state and provide limited evidence that long-horizon results are correctly completed and saved.
Method
StateAct uses code-based state-grounded action, a dedicated GUI subagent, an independent finish gate, and context management for long-horizon execution.
Results
26.9% binary success, up from 20.6%, with approximately 9× lower cost than the same-model reference harness.
Takeaways & Limitations
State-grounding shifts the main bottleneck in long-horizon computer use from perception toward reasoning.
Takeaways & Limitations
State-grounding offers no leverage for render-only tasks whose targets or interactions exist only in the visual output.
Abstract
from arXiv · showhide
Computer-use agents are usually improved by strengthening perception: better models for reading a screenshot and choosing where to click. Yet a screenshot is only a lossy rendering of the underlying program state, e.g., the files, application backends, and DOM that hold the task data. Different states can produce the same pixels, while code can inspect and modify that state directly. StateAct is a code-first, multi-agent harness built around this distinction. Its main agent works directly with program state by using code, while a dedicated GUI subagent handles screenshot-and-click interaction on the few subgoals that need it, just 28 of 108 tasks and 1.1% of main-agent steps. The same direct access to program state also supports verification: an independent finish gate double-checks the saved result for structural failures, e.g., output that is missing, unsaved, or written to the wrong path. To stay on track over hundreds of steps, the main agent hands subgoals to fresh subagents, keeping its own context focused. On OSWorld 2.0, StateAct lifts Claude Opus 4.8 from 20.6% to 26.9% on binary success, and from 54.8% to 61.6% on partial success, at ~ 9x lower cost per task than the same model driven by screenshots alone; a code-only variant with no GUI subagent reaches only 45.9% partial, below that screenshot-based baseline's 54.8%. In general, grounding action, verification, and memory in state, what we call state-grounding, shifts the main bottleneck from perception toward reasoning: failures depend more on what the agent thinks than on what it sees.
1 INTRODUCTION
StateAct argues that computer-use agents should act on and verify underlying program state rather than rely solely on lossy screenshots. Its state-grounded harness combines direct artifact manipulation with selective GUI interaction and improves Claude Opus 4.8 performance over the reference harness.
- Motivation: StateAct grounds action in program state because screenshots can conceal task-critical distinctions, such as formulas versus literals or hidden rows.Direct state access exposes and edits the values behind rendered pixels.
- Architecture: StateAct combines state-grounded action, verification, and long-horizon execution in a single agent loop.The main agent reads and writes real artifacts through persistent bash, Python, and a file editor.
- Results: 45.9% partial success with only bash falls below the 54.8% vision baseline, showing that code-only interaction is insufficient.StateAct retains GUI support because not every application exposes comprehensive or otherwise accessible state through code.
- Architecture: 1.1% of main-agent steps and 28 of 108 tasks use StateAct’s dedicated GUI subagent for visual interaction.The architecture primarily observes and acts on program state while retaining GUI support for visual subgoals.
- Results: 20.6% → 26.9% binary success and 54.8% → 61.6% partial success are the gains StateAct reports for Claude Opus 4.8 over the reference harness.The harness exceeds every public entry on a benchmark described in the contribution passage.
2 RELATED WORK
Prior work improves computer-use agents through perception, code or hybrid action spaces, and state-based verification. StateAct instead routes code/state operations through the main agent, delegates GUI interaction, and extends state inspection to its finish gate.
- Grounding-based computer-use agents: StateAct complements perceptual grounding by grounding the main agent in program state and delegating direct screen interaction as a fallback.Related approaches improve screen reading through GUI action models, test-time selection, or easier-to-ground observations.
- Code and hybrid action spaces: Prior systems establish executable code, hybrid API/GUI actions, unified GUI–API layers, and purpose-built interfaces as viable alternatives to UI-shaped action schemas.Examples include CodeAct, OSWorld, CoAct-1, UltraCUA, ComputerRL, UFO2, and SWE-agent.
- Code and hybrid action spaces: StateAct’s architectural distinction is default routing: code/state operations remain with the main agent, while a subagent controls the GUI.The same state inspection is extended to the verification gate.
- Self-verification for agents: State-based verification is established through application-specific verifiers, proactive interaction with hidden state, and backend instrumentation.StateAct’s finish gate is described as deliberately weaker on value, with the supplied passage ending before the contrasting property is specified.
3 THE STATE-GROUNDING PRINCIPLE
State-grounding favors querying and modifying program state because screenshots are lossy and task success is determined by the underlying state. Its advantage applies to state-addressable subgoals, not outcomes or interactions that exist only visually.
- The State-Grounding Principle: Rendering is non-injective: distinct program states can produce indistinguishable screenshots, so perception alone cannot generally recover the underlying state.The state channel can recover exact values such as formulas over the substate a task touches.
- The State-Grounding Principle: The loss of pixel-based state estimates compounds across hundreds of steps, causing state drift as task horizons lengthen.StateAct targets this long-horizon regime, where the state channel’s advantage grows with task length.
- The State-Grounding Principle: The deliverable is a change to program state, and success is therefore evaluated by whether the saved file or application settings satisfy the goal.For formulas, hidden rows, off-screen data, and backend state, success cannot be recovered from rendered pixels alone.
- Boundary: where the state channel does not help: State-grounding provides no leverage for visual outcomes or subgoals expressible only as rendered interactions.Examples include image editing, layout, chart appearance, canvas dragging, non-scriptable modals, and values existing only on-screen.
4 STATEACT
StateAct combines code-based action on program state, independent artifact verification, and context management for long-horizon computer-use tasks. The main agent delegates only irreducibly visual subgoals to a GUI specialist while preserving a state-grounded workflow.
- Architecture: StateAct has three components: a code-driven main agent, an independent finish gate that re-reads artifacts, and context management for hundreds of steps.These components jointly structure action, verification, and long-horizon execution.
- Main-agent action: The main agent operates through code and structured tools, with no live mouse or keyboard actuation exposed.Its tools include persistent bash, file editing, image viewing, planning, finishing, and delegation.
- GUI delegation: 1.1% of main-agent steps use the GUI subagent, while 28 of 108 tasks invoke it at least once for irreducibly visual subgoals.Delegation applies when probing finds no file, backend, or DOM path, or when the effect exists only as rendered interaction.
- Finish verification: The finish gate independently inspects the machine after a finish call, seeing only the task and machine rather than the agent’s history, plan, rationale, or expected values.Editor mutations are blocked, and the finish action requires at least three preceding non-finish steps.
- Context management: Up to 200 main-agent turns and a mean of 4.2 delegations per task are managed through fresh specialist contexts, auto-compaction, and an externalized checklist.Delegated loops are each limited to 50 turns, while the checklist survives compaction and is re-injected each turn.
5 EXPERIMENTS
On OSWorld 2.0, StateAct improves Claude Opus 4.8’s success and partial credit while reducing cost, with state-grounded action contributing most to performance. Ablations show that verification, context management, and flat delegation provide additional benefits, while code action alone is insufficient.
- Main results: 26.9% / 61.6% makes StateAct the best-performing entry, raising binary success by +6.3 points and mean partial by +6.8 on the identical backbone.Output tokens fall from 224K to 100K, while cost declines from ∼$72 to ∼$7.8 per task, a ∼9× reduction.
- Cost–accuracy frontier: 8.7 binary points separate StateAct from Opus-4.7, while 13.9 separate it from GPT-5.5, at ∼$7.8 per task versus ∼$33.6 and ∼$25.5.StateAct adds ≥12 partial points over both public entries and costs far less than the same-model reference at ∼$72.
- Component sensitivity: 61.6% →51.3% is the largest ablation drop when act-on-state is removed, falling below the reference’s 54.8% partial score.Disabling the finish gate yields 57.5%, while removing compaction and planning yields 58.7%.
- Delegation depth: 61.6 vs. 57.6 / 57.9 favors flat delegation over worker recursion and nested self-recursion on mean partial credit.The recursive branch fired on only 7 of 108 tasks and never nested, limiting attribution of differences to delegation depth.
- Additional designs: 45.9% partial from the bash-only configuration falls below the reference’s 54.8%, showing that code action alone is insufficient without the scaffold.On Claude Sonnet 4.6, the same harness raises binary success from 8.3% to 11.1%.
6 DISCUSSION
StateAct gains most where program state is machine-checkable, while reasoning errors, verifier limits, and modality bottlenecks remain the main failure sources. Because state carries most of the task, a compact GUI specialist performs comparably to Claude’s computer-use model on most benchmarks.
- Where the harness pays off: StateAct’s largest margins occur on machine-checkable capabilities, including multi-item state, cross-source reasoning, and conflict disambiguation.Human-in-the-loop and multimodal editing remain weaker because they require interaction or visual capabilities.
- Failure modes: 38 tasks had agent-reasoning errors, while 14 were verifier-weak or wrong-path finishes that the finish gate wrongly accepted.Together, these 52 tasks are described as plausibly addressable through better reasoning or verification.
- Verifier ceiling: 68/76 (≈90%) of non-perfect tasks reaching the finish gate were wrongly passed, because the gate checks structure but cannot adjudicate value correctness.The gate correctly rejected only 8 of 76 reached tasks, as re-deriving from the same source can reproduce the agent’s wrong answer.
- GUI fallback: 1.1% of main-agent steps invoked the GUI subagent, covering 28 of 108 tasks and allowing the main agent to rely primarily on program state.The experiment tested whether this rare visual fallback required a frontier GUI model.
- GUI fallback: On four of five benchmarks, replacing Claude’s computer-use model with SFR-CUA barely changed mean-partial scores: 81.1 vs. 81.9, 51.2 vs. 50.6, 84.1 vs. 81.9, and 68.4 vs. 70.1.Despite weaker standalone scores—66.9 vs. 80.9 on OSWorld-Verified and 7.6 vs. 54.8 on OSWorld 2.0—the compact specialist sufficed once the main agent carried tasks on state.
7 CONCLUSION
StateAct makes program state the main agent’s primary interface while retaining a GUI subagent for visual interaction. On a standard long-horizon GUI benchmark, it improves Claude Opus 4.8’s success at approximately 9× lower cost without changing the model.
- StateAct uses program state as the main agent’s primary interface and retains a dedicated GUI subagent for visual interaction.
- 26.9% binary success rises from 20.6%, while 61.6% partial success rises from 54.8% for Claude Opus 4.8.
- ∼9× lower cost accompanies these gains without any change to the model itself.
- The ablation and diagnostic analyses attribute the gain to what the agent observes: state rather than screenshots.