Source-linked AI summary
Step-level Optimization for Efficient Computer-use Agents
Jinbiao Wei, Kangqi Ni, Yilun Zhao, Guo Gan, Arman Cohan
TL;DR
Computer-use agents are costly and slow because they typically invoke large multimodal models at every interaction step, despite heterogeneous task difficulty. This paper introduces an event-driven cascade that uses a small policy by default and escalates when stuck or milestone monitors detect risk, achieving comparable success while reducing inference cost by up to 74.6% and latency by up to 45.8%.
Problem
Computer-use agents allocate expensive frontier-model inference uniformly across GUI steps, although many steps are routine and failures concentrate at fewer high-risk moments.
Method
An event-driven cascade runs a small GUI policy by default and escalates to a stronger model when stuck or milestone monitors detect elevated risk.
Results
Comparable task success is achieved while reducing inference cost by up to 74.6% and latency by up to 45.8% across realistic computer-use benchmarks.
Takeaways & Limitations
Selective escalation recovers most of always-large inference’s performance without invoking the stronger model at every step.
Abstract
from arXiv · showhide
Computer-use agents provide a promising path toward general software automation because they can interact directly with arbitrary graphical user interfaces instead of relying on brittle, application-specific integrations. Despite recent advances in benchmark performance, strong computer-use agents remain expensive and slow in practice, since most systems invoke large multimodal models at nearly every interaction step. We argue that this uniform allocation of compute is fundamentally inefficient for long-horizon GUI tasks. Such trajectories are highly heterogeneous: many steps are routine and can be handled reliably by smaller, cheaper policies, while errors tend to concentrate at a relatively small number of high-risk moments. Across computer-use benchmarks, these failures repeatedly take two forms: progress stalls, where the agent loops, repeats ineffective actions, or fails to make meaningful progress, and silent semantic drift, where the agent continues taking locally plausible actions after already deviating from the user's true goal. To address this inefficiency, we propose an event-driven, step-level cascade for computer-use agents that runs a small policy by default and escalates to a stronger model only when lightweight learned monitors detect elevated risk. Our framework combines two complementary signals: a Stuck Monitor that detects degraded progress from recent reasoning-action history and triggers recovery, and a Milestone Monitor that identifies semantically meaningful checkpoints where sparse verification is most informative for catching drift. This design turns always-on frontier-model inference into adaptive, on-demand compute allocation over the course of an evolving interaction. The framework is modular and deployment-oriented: it can be layered on top of existing computer-use agents without changing the underlying agent architecture or retraining the large model.
1 Introduction
Computer-use agents can automate diverse GUI workflows, but uniform frontier-model inference wastes compute because routine steps dominate while failures concentrate at high-risk moments. The paper introduces a modular, event-driven cascade that uses lightweight monitors to escalate selectively, achieving comparable success with substantially lower cost and latency.
- GUI-based agents automate workflows across diverse applications without brittle per-application integrations.This enables end-to-end execution on arbitrary software, including long-tail internal tools.
- Uniform frontier-model inference is inefficient because routine steps are often manageable by smaller models, while failures concentrate at high-risk moments.The paper identifies progress stalls and silent semantic drift as recurring failure forms.
- The proposed cascade runs a small GUI policy by default and escalates to a stronger model when learned monitors predict degraded execution.Its controller uses short windows of recent interaction history and combines Stuck and Milestone monitors.
- The framework combines complementary routing signals: stuck detection supports recovery from local failures, while milestone detection helps verify semantic progress.The design is modular and can be trained from logged trajectories without modifying the underlying agent.
- 74.6% maximum inference-cost reduction and 45.8% maximum latency reduction are achieved while maintaining comparable task success to always-large agents.These results establish the cascade’s cost–quality advantage on realistic multi-step GUI benchmarks.
2 Related Work
Related work studies adaptive model selection for balancing quality, latency, and cost, while recent computer-use-agent research emphasizes deployment efficiency alongside task success.
- Routing and cascading for cost–quality trade-offs: Routing selects one model per query, whereas cascading escalates to stronger models only when needed using quality estimators or stopping criteria.Examples include FrugalGPT, RouteLLM, and Hybrid LLM.
- Routing and cascading for cost–quality trade-offs: Recent work extends model-selection methods beyond query-level routing.
- Efficiency and practicality of computer-use agents: Computer-use-agent research increasingly treats latency and monetary cost as deployment bottlenecks, not merely task success.
- Efficiency and practicality of computer-use agents: OSWorld-Human provides a focused benchmark and temporal analysis for computer-use-agent efficiency.
- Efficiency and practicality of computer-use agents: SCUBA emphasizes enterprise realism and reports time and monetary cost alongside success.The passage also states that demonstration augmentation can improve success, but the supplied excerpt is truncated before specifying the result.
3 Failure Modes and Motivation for Step-Level Optimization
Long-horizon GUI agents face concentrated failures that arise as either stalled progress or silent semantic drift. Because interaction changes an evolving environment step by step, adaptive step-level cascading is more suitable than single query-level routing.
- Failure modes: Failures commonly take two forms: progress stalls with ineffective repeated actions, and silent semantic drift after deviation from user intent.In silent drift, subsequent actions can remain locally plausible while the trajectory is ultimately doomed.
- Motivation: Failures are not uniformly distributed across long closed-loop trajectories, where each step consumes compute and irreversibly changes interface state.The trajectory structure makes localized monitoring and intervention valuable for avoiding wasted computation during extended failure regions.
- Failure modes: 2.8× more steps than successes: EvoCUA-8B failed episodes were substantially longer than successful episodes.Qwen3-VL-8B showed a similarly large 2.5× gap, indicating extended stuck regions rather than only isolated bad decisions.
- Why query-level cascades are not enough: Unlike query-level cascades, computer-use agents can route computation at each step while interacting with an evolving environment.Step-level routing is more complex than standard query-level routing because decisions occur throughout the trajectory rather than once per query.
4 Methodology
The methodology introduces an event-driven cascade that uses a small GUI policy by default and allocates large-model computation when lightweight monitors detect elevated risk. A Stuck Monitor triggers recovery for degraded progress, while a Milestone Monitor enables sparse, goal-conditioned verification of intermediate outcomes.
- Event-driven cascade: The controller routes decisions to πsmall by default and escalates to πlarge when the Stuck Monitor detects trajectory stalls.The Stuck Monitor produces a stuck score from recent interaction history and compares it with threshold θs.
- Setup and notation: At each step, the selected model generates both a textual rationale and an executable GUI action from the current observation and interaction history.Episodes have length at most T, with observations such as screenshots and optional accessibility or DOM signals.
- Monitor inputs: The Stuck Monitor uses a compact local output window, whereas the Milestone Monitor also uses the task description because milestone significance depends on the goal.The window captures behavioral patterns such as repeated rationales, oscillating plans, and action loops; both monitors use lightweight ModernBERT encoders.
- Milestone detection and verification: When milestone confidence crosses θm, the system packages the task, rationale–action trace, and before-and-after screenshots for stronger-model verification.The verifier assesses whether the local trajectory advances toward the goal and whether the current UI state remains consistent with user intent.
- Training data via LLM supervision: The monitors are trained on πsmall trajectories labeled by a stronger LLM for stuckness and milestone completion, retaining only high-consensus annotations.Labels use the local window for stuckness and the task description plus window for milestone completion; training uses cross-entropy with class balancing.
5 Experiments
Experiments on OSWorld and WebArena compare standalone and cascaded computer-use agents under matched task settings. Across both benchmarks, selective escalation recovers much of always-large performance at lower cost and latency, with the large model chiefly determining the performance ceiling.
- Benchmarks: The framework is evaluated on OSWorld desktop tasks and WebArena web tasks, covering long-horizon GUI interaction across operating-system and online environments.OSWorld spans browsers, office tools, coding environments, and system utilities, while WebArena evaluates realistic web-based agents.
- Evaluation: Small models act as default policies, while cascaded configurations invoke stronger models only when the controller escalates under matched benchmark task settings.The evaluation measures whether step-level routing preserves stronger-model effectiveness while reducing inference cost.
- Results: Cascading consistently improves the performance–efficiency frontier: small models are cheaper but weaker, whereas cascades approach always-large success without invoking stronger models at every step.This pattern holds across both OSWorld and WebArena.
- OSWorld: 59.3% success is achieved by Qwen3-VL-8B + Kimi K2.5 versus 60.1% for standalone Kimi K2.5, while EvoCUA-8B + Kimi K2.5 reaches 58.2% success at $0.051 per task, reducing cost by 61.4%.Both Kimi-based cascades match or slightly exceed the standalone Claude Sonnet 4.5 baseline.
- WebArena: On WebArena, GPT-5.2-based cascades remain close to standalone GPT-5.2, outperform their standalone small-model counterparts, and reduce cost and latency.Standalone GPT-5.2 performs best overall, while event-driven escalation remains effective in shorter web trajectories.
- Model effects: For a fixed large model, changing the small policy usually affects accuracy modestly, whereas upgrading the large model yields a substantially larger gain.The results indicate that the large model sets the attainable performance ceiling, while the small model reduces expensive escalation frequency.
6 Analysis
The analysis shows that stuck and milestone detectors provide complementary routing signals: the full cascade performs best by responding to local failure loops while checking semantically meaningful progress. Event-driven verification also better matches trajectory structure than fixed-interval checks, and the learned milestone detector reaches 94.1% accuracy and 62.0% F1.
- Detector ablations: Across OSWorld and WebArena, either detector improves over no detection, while enabling both produces the best overall performance.The results attribute cascading gains to combining complementary escalation signals rather than relying on a single trigger.
- Detector ablations: The stuck detector targets repetition, oscillation, and short progress loops, whereas the milestone detector verifies meaningful checkpoints to catch silent semantic drift.Policies with stronger progress-stall behavior benefit more from stuck detection, while policies with more semantic drift benefit more from milestone verification.
- Detector ablations: The full system delivers the strongest and most consistent gains across settings and benchmarks because the two detectors provide distinct local and high-level oversight.Explicit stuck detection improves responsiveness to immediate failures, while milestone detection monitors meaningful progress.
- Event-driven versus periodic verification: Event-driven verification outperforms fixed-interval checking because periodic checks every 3, 5, or 7 steps can miss milestones or verify too far from the final outcome.The event-driven framework adapts verification to the trajectory’s semantic structure, which is especially useful for short, high-variance web tasks.
- Monitor evaluation: 94.1% accuracy and 62.0% F1: the milestone detector performs well on a held-out 20% EvoCUA-8B split.F1 is emphasized because milestone events are sparse and somewhat ambiguous.
7 Conclusion
The paper introduces an event-driven, step-level cascade that allocates expensive large-model inference only when lightweight monitors detect elevated risk. It exploits heterogeneous GUI trajectories by using a small default policy for routine steps while focusing resources on critical failures.
- 7 Conclusion: The framework allocates expensive large-model inference only when lightweight monitors detect elevated risk.This event-driven, step-level cascade avoids treating every interaction step as equally difficult.
- 7 Conclusion: Long-horizon GUI trajectories are highly heterogeneous, with many routine steps suitable for a small default policy.The approach is based on varying step difficulty across trajectories.
- 7 Conclusion: Failures tend to concentrate at a limited number of critical moments, motivating targeted compute allocation.
A Details of Benchmarks · A.1 OSWorld · A.2 WebArena
The paper evaluates its computer-use agent framework on two reproducible long-horizon benchmarks: OSWorld for desktop operating systems and WebArena-Verified for browser-based tasks. OSWorld uses controlled virtual machines and final-state scripts, while WebArena-Verified strengthens web evaluation through audited tasks, reference answers, deterministic evaluators, and version-controlled data.
- A.1 OSWorld: OSWorld evaluates agents on real desktop operating systems, primarily Ubuntu, inside controlled virtual-machine environments.Each task begins from a reproducible VM snapshot containing the initial desktop state.
- A.1 OSWorld: OSWorld tasks specify a natural-language user goal, a reproducible initial VM snapshot, and an execution script that returns a scalar reward.The snapshot records open applications, files, and window layout, while the script inspects the final VM state.
- A.2 WebArena: WebArena-Verified is a verified, reproducible release of the original WebArena benchmark for long-horizon web-agent execution on realistic browser tasks.The release preserves the original benchmark setting while strengthening its evaluation pipeline.
- A.2 WebArena: WebArena-Verified improves evaluation stability through audited tasks, reference answers, deterministic evaluators, and version-controlled benchmark data.These components make the benchmark substantially more stable and reproducible than the original evaluation pipeline.
- A.2 WebArena: Experiments use the ServiceNow WebArena-Verified release, where agents perform standard browser actions including clicking, typing, and scrolling.Performance is measured by the benchmark’s verified task evaluator.
- A.2 WebArena: The WebArena-Verified results report both overall success rate and per-domain breakdowns in the main results.The evaluation protocol is intended to support reliable and reproducible web-agent measurements.
B ModernBERT Fine-tuning Details … Stuck Example 2: Repetitive cell-selection loop in Calc
The paper fine-tunes ModernBERT-base classifiers for stuck and milestone detection, then illustrates stuck behavior through repetitive GUI actions in Steam and Calc. The examples show agents recognizing failed progress yet continuing ineffective actions instead of advancing the task.
- B ModernBERT Fine-tuning Details: ModernBERT-base is a 149M-parameter, 22-layer encoder-only transformer fine-tuned for binary stuck-detection and milestone-detection tasks.A linear classification head is added to the [CLS] representation.
- B ModernBERT Fine-tuning Details: Training uses 300 GUI-agent trajectories labeled by GPT-5.2 across five runs, retaining positives found in at least three runs and negatives found in none.Steps identified in only one or two runs are excluded to reduce label noise.
- B ModernBERT Fine-tuning Details: Both detectors train for 5 epochs with AdamW at 5 × 10−5, batch size 8, sequence length 2,048, bf16 precision, and an 80/20 split.Training also uses inverse-frequency class weighting and selects models by best test-split F1.
- Stuck Example 1: Dropdown-closing loop on the Steam page: In the Steam example, the agent repeatedly tries to dismiss a review-options dropdown while failing to resume the purchase flow for Artifact Foundry.Across steps 40–45, it switches pages, scrolls, and repeats Escape or click-away actions without locating the add-to-cart control.
- C Detector examples: The detector examples show stuck behavior as repeated actions that fail to return the agent to the intended task flow or produce meaningful progress.The Steam and Calc traces provide concrete instances of this pattern.
- Stuck Example 2: Repetitive cell-selection loop in Calc: In Calc, the agent repeatedly clicks cells A4 and A5 without entering year values or advancing the spreadsheet.Steps 44–49 explicitly acknowledge repeated failure but continue oscillating between the same cell targets.
- Stuck Example 2: Repetitive cell-selection loop in Calc: The Calc trace is flagged as a stuck loop because the reasoning recognizes failure while the behavior continues repeating ineffective cell selections.The agent does not enter values or advance the sheet despite proposing alternative clicks.
Milestone Example 1: Locating the insertion point in the report · Milestone Example 2: Reaching the final alignment action on slide 3
The examples illustrate milestone detection at meaningful intermediate states: locating the report’s insertion section and reaching the exact alignment command for a slide-specific edit. These checkpoints establish clear task progress before the broader edits are complete.
- Milestone Example 1: Locating the insertion point in the report: The task is to extract GPT-4 experiment results from “˜/Documents/awesome-desktop/expe-results.xlsx” and insert them into the report’s “Main Results” section.
- Milestone Example 1: Locating the insertion point in the report: The agent progressively scrolls through the LibreOffice Writer document because the target “Main Results” section is initially not visible.
- Milestone Example 1: Locating the insertion point in the report: The agent reaches section “5.2 Main Results” and prepares to position the cursor for table insertion.
- Milestone Example 1: Locating the insertion point in the report: This location qualifies as a milestone because it is the correct insertion point and a necessary intermediate state before opening the spreadsheet and inserting the GPT-4 table.
- Milestone Example 2: Reaching the final alignment action on slide 3: The second task requires aligning the first textbox right on slide 3, center on slide 4, and left on slide 5.
- Milestone Example 2: Reaching the final alignment action on slide 3: After stabilizing the interface, the agent navigates to slide 3, selects the “HELLO THERE!” textbox, and opens Format to access alignment options.
- Milestone Example 2: Reaching the final alignment action on slide 3: Reaching the exact alignment command marks clear progress toward the multi-slide assignment without completing all slide edits.
D Prompts
The prompts define milestone and stuck-step labeling for GUI-agent trajectories, then verify milestone success by comparing intended progress against before-and-after screenshots. They enforce evidence-grounded reasoning and structured JSON outputs throughout.
- D.1 Prompt Used for Milestone Data Labeling: Milestone labeling identifies steps where meaningful, verifiable progress is achieved.The prompt frames milestone detection as trajectory labeling for GUI agents.
- D.1 Prompt Used for Milestone Data Labeling: Short trajectories may yield only a final milestone, while longer trajectories allow spaced milestones and exclude stuck-region steps.Multiple milestones should be limited and at least three steps apart.
- D.1 Prompt Used for Milestone Data Labeling: Milestones must be meaningful and verifiable from the step text, favor higher-level progress, avoid unsupported UI details, and include reasoning.The required evidence includes action, response, done, or fail text.
- D.1 Prompt Used for Milestone Data Labeling: The milestone-labeling prompt supplies task ID, total steps, and trajectory data, and requires JSON containing each milestone step number and reasoning.The output schema is explicitly constrained to JSON.
- D.2 Prompt Used for Stuck-Step Labeling: Stuck-step labeling marks trajectories stuck when repeated actions, error or infinite loops, or several steps without meaningful progress occur.The evaluator analyzes computer-use-agent trajectories for execution stalls.
- D.2 Prompt Used for Stuck-Step Labeling: Stuck analysis returns structured fields for status, affected steps, reasons, severity, and summary, with a separate successful-completion response when the agent is not stuck.The prompt also includes task ID, total steps, trajectory data, and a JSON-format requirement.
- D.3 Prompt Used for Milestone Verification: Milestone verification infers the intended milestone, compares before-and-after screenshots with recent actions, and marks success only when meaningful progress is clearly achieved.The input includes the task description, actions since the previous milestone, and both screenshots; output includes the inferred milestone, success, and reasoning.