PaperlayerView exact revision on arXiv

Source-linked AI summary

GUI vs. CLI: Execution Bottlenecks in Screen-Only and Skill-Mediated Computer-Use Agents

Xiao Zhou, Siyue Zhang, Yilun Zhao, Jinbiao Wei, Tingyu Song, Arman Cohan, Chen Zhao

arXiv:2606.24551v1cs.AI

TL;DR

Existing evaluations conflate interaction modality with task setup, making GUI–CLI comparisons difficult to interpret. This paper introduces a matched benchmark and finds that GUI outperforms original-skill CLI, while verifier-guided skill augmentation makes CLI strongest.

  • Problem

    Existing evaluations vary tasks, states, verifiers, and action spaces simultaneously, limiting evidence about performance differences attributable to interaction modality.

  • Method

    The paper benchmarks 440 desktop tasks across 18 applications with identical goals, states, and verifiers while restricting agents to native GUI or CLI actions.

  • Results

    59.1% GUI full pass rate exceeds 48.2% for original-skill CLI, while verifier-guided skill augmentation raises CLI success to 69.3%.

  • Takeaways & Limitations

    GUI agents face visual-grounding and long-workflow bottlenecks, whereas CLI agents depend on skill-interface coverage and reliable exposure of verifier-relevant state.

  • Takeaways & Limitations

    The patched-skill experiment is diagnostic and does not establish that repaired skills generalize to unseen tasks, applications, or verifier designs.

Abstract

from arXiv · show

Computer-use agents can execute software tasks through either graphical interfaces or programmatic command interfaces, but existing evaluations confound interaction modality with differences in tasks, initial states, verifiers, and permitted actions. We introduce a matched execution-layer benchmark of 440 desktop tasks across 18 applications and 12 workflow categories, where screen-only GUI agents and skill-mediated CLI agents receive identical goals, states, and final-state verifiers while being restricted to modality-native actions. In this controlled setting, the strongest GUI agent reaches a 59.1% full pass rate, outperforming the strongest original-skill CLI agent at 48.2%; however, verifier-guided skill augmentation raises CLI success to 69.3%, showing that much of the CLI deficit comes from incomplete skill coverage rather than model capability alone. These results suggest that GUI and CLI expose different execution bottlenecks: GUI agents are limited by reliable grounded interaction over long-horizon workflows, whereas CLI agents are limited by the coverage and scalability of their skill interfaces.

1 Introduction

The paper introduces a matched benchmark that isolates GUI versus skill-mediated CLI execution by fixing goals, states, and verifiers while varying modality-native actions. Results show stronger GUI performance under the original skill layer, with GUI limited by visual grounding and long workflows and CLI by skill coverage gaps.

  • Motivation: Existing evaluations confound applications, initial states and final-state verifiers, and permitted action spaces, preventing outcomes from isolating interaction modality.When these factors vary simultaneously, success cannot identify the source of performance differences.
  • Matched benchmark: 440 desktop tasks span 18 applications and 12 workflow categories, with identical goals, initial states, and executable final-state verifiers across modalities.Instructions are modality-neutral, while each agent uses only its modality-native action space.
  • Benchmark results: 59.1% full pass rate is achieved by the strongest GUI agent, GPT-5.4, versus 48.2% for the strongest CLI agent, Codex GPT-5.5, under the original CLI-Anything skill layer.The benchmark reports different workflow preferences across the two modalities.
  • Execution bottlenecks: GUI agents are bottlenecked by visual grounding and long workflow execution, whereas CLI agents are bottlenecked by skill coverage gaps and implicit-default reconstruction.The results frame the comparison as where execution logic is engineered: the visible interface or a constructed skill layer.
  • CLI diagnosis: 37.6% of verifier checkpoints are satisfied by the original skills, identifying skill coverage as a CLI bottleneck.The contribution includes a diagnostic analysis of skill coverage as a limiting factor for CLI agents.

2 Related Work

Related work has expanded GUI-agent training and benchmarks across web, enterprise, and desktop environments, while CLI-agent studies evaluate programmatic execution under interfaces with differing action exposure. GUI research repeatedly identifies perceptual grounding, long action chains, and recovery from layout changes as dominant failure modes.

  • GUI Agents: GUI-agent research spans training methods and benchmarks for web, enterprise, and full desktop operating-system environments.The cited work includes recent training methods and benchmarks across these graphical-interface settings.
  • GUI Agents: Perceptual grounding, long action chains, and recovery from layout changes recur as dominant GUI-agent failure modes.These failure modes appear across the cited web, enterprise, and desktop settings.
  • CLI Agents and Skills: CLI-agent benchmarks study execution through APIs, command-line environments, or direct filesystem access, but interfaces differ in what actions they expose.The passage contrasts unrestricted shells, which afford arbitrary code, with curated skill layers that constrain the agent.

3 Benchmark

The benchmark compares GUI and CLI agents at the execution layer using matched goals, initial states, and final-state verification. It spans 440 tasks across 18 applications and 12 workflow categories, enabling aggregate and per-workflow analysis of modality-specific execution.

  • The benchmark evaluates GUI and CLI interaction modalities under matched task goals, initial states, and final-state verification.
  • 440 tasks span 18 real-world applications and 12 workflow categories, organized by workflow to support aggregate and per-workflow comparisons.
  • Workflow diversity covers tasks requiring visual layout and interface navigation alongside structured artifacts or application-level operations.
  • Tasks are selected where CLI-Anything skills are available, rewritten into modality-agnostic outcome descriptions, curated for balance, and manually validated for identical execution and verification.Each task must be solvable through either modality without modality-specific procedural cues and must specify a precise target state evaluated by the same verifier.

4 Experiment Setup

The experiment compares GUI and CLI agents under matched instructions, initial states, and executable final-state verification while restricting each modality to native interaction mechanisms. It evaluates success by final application state and reports full pass rate and average task time.

  • Both modalities use the same benchmark, instructions, initial states, and executable final-state verifiers.
  • Action Space: Agents cannot use unrestricted programming or direct artifact manipulation to bypass the evaluated interface.GUI agents interact through a sandboxed Ubuntu desktop; CLI shell use is limited to skill discovery, execution management, and read-only inspection or verification.
  • Models: The study evaluates six GUI agents and four CLI baselines, including GPT-5.4, Claude-Sonnet-4.6, and Claude-Opus-4.7.The GUI set also includes EvoCUA-32B-20260105, Qwen3.5-27B, and Kimi-K2.6; CLI baselines include GPT-5.4, GPT-5.5, Claude-Sonnet-4.6, and Claude-Opus-4.7.
  • Evaluation Metrics: Task success is determined by final application state through executable verifier checkpoints, not reference action trajectories.The reported metrics are full pass rate, requiring every verifier check to pass, and average task time as execution cost.

5 Main Results

Under matched tasks, initial states, and verifiers, GUI agents outperform the strongest original-skill CLI agent overall, while modality changes both robustness and which workflow structures are favored. GUI performance is more stable across categories, whereas CLI performance depends more heavily on skill coverage and representation.

  • Overall performance: 59.1% full pass rate is achieved by GPT-5.4, versus 48.2% for Codex GPT-5.5, under identical tasks, initial states, and verifiers.The comparison reflects model capability and interaction design rather than task variation.
  • Overall performance: Interaction design can narrow, and sometimes reverse, performance gaps between underlying models.
  • Robustness across workflows: 42.9%–88.2% is GPT-5.4’s workflow-category range, compared with 9.8%–100.0% for Codex GPT-5.5.GUI agents show more stable performance across categories, while CLI execution depends more heavily on skill coverage and workflow demands.
  • Workflow structures: GUI agents perform better on Audio, Presentations, Communication, and Web workflows where interfaces expose the intended operation sequence.The per-workflow results show that modality advantages are not uniform across the benchmark.
  • Workflow structures: Visual Design tasks can favor programmatic operations when structured properties such as pages, shapes, labels, and connectors matter.Thus, modality advantage depends less on application category than on the representation each interface provides for execution.

6 Error Analysis

Error analysis shows that CLI failures are dominated by skill-interface limitations, while GUI failures arise from navigation and long-horizon workflow execution. The taxonomy identifies distinct modality-specific mechanisms, including hidden application semantics for CLI and incomplete verification for GUI.

  • Failure distribution: 93.8% of CLI failures were classified as skill coverage and contract gaps, compared with GUI failures split between workflow execution and UI navigation/control discovery.The analysis manually annotated 80 randomly sampled failed trajectories per modality from Codex GPT-5.5 in CLI and GPT-5.4 in GUI.
  • CLI failure mechanisms: CLI failures occur when skill interfaces omit required operations, diverge from implementations, or fail to preserve verifier-relevant application state.Agents can follow prescribed workflows yet still fail final verification when the skill contract does not expose the needed executable action boundary.
  • CLI failure mechanisms: CLI agents also fail by misreconstructing implicit defaults and hallucinating application semantics unavailable through commands, scripts, or skills.These errors involve identity conventions such as default names and identifier rules, producing structurally correct outputs that violate implicit semantics.
  • GUI failure mechanisms: GUI failures arise when agents cannot discover controls or execute ordered sequences involving menus, dialogs, confirmations, and state transitions.The analysis links these failures to interaction-heavy, stateful execution through visual interfaces and longer action trajectories.
  • GUI failure mechanisms: GUI agents may declare success after plausible interactions without checking that required artifacts or persisted state changes were actually produced.Examples include failing to inspect exported files, confirm saved state, or validate application outputs against task requirements.

7 Further Analysis

Further analysis identifies incomplete CLI skill coverage and GUI procedural grounding as distinct execution bottlenecks. Verifier-guided skill patching substantially improves CLI performance, while procedure-guided descriptions modestly improve GUI completion and reduce runtime.

  • CLI skill coverage: 100% coverage in the patched-skill experiment reflects verifier-observed completion, not evidence that repaired CLI skills generalize to unseen tasks.The diagnostic is counterfactual and uses verifier information rather than establishing a fair deployed-agent baseline.
  • CLI skill coverage: Verifier-observed skill incompleteness explains a large fraction of the original CLI performance gap, although CLI still underperforms GUI in Spreadsheet and Web workflows.Skill patching does not uniformly eliminate the remaining GUI–CLI gap.
  • CLI skill coverage: Skill coverage is a major CLI bottleneck, but modality-specific differences persist even after verifier-guided patching.The results indicate that interaction modality cannot be fully reduced to the availability of skill operations.
  • GUI procedural grounding: Procedure-guided descriptions modestly improve GUI performance and reduce interaction cost, suggesting more direct action paths and less unnecessary exploration.Completion gains remain modest despite the reduction in execution time.
  • GUI procedural grounding: 59.7% to 60.2% full pass rate followed procedure-guided descriptions, while average reward rose from 0.7401 to 0.7576 and average runtime fell from 397.0s to 314.8s.Procedure-guided descriptions preserve the same verifier and target state while adding explicit workflow cues.

8 Conclusion

The benchmark isolates how native action interfaces shape computer-use performance by fixing task goals, initial states, and final-state verifiers. It shows that GUI agents perform best and most stably when the interface exposes the workflow.

  • 8 Conclusion: The benchmark fixes task goals, initial states, and final-state verifiers while systematically varying the native action interface.This design makes interface effects visible within matched evaluations.
  • 8 Conclusion: Agents do not simply execute identical tasks through different controls; the action interface provides different representations of what can be done.The conclusion distinguishes interface-mediated affordances from control modality alone.
  • 8 Conclusion: GUI performance is strongest and more stable when the interface itself exposes the workflow.The passage identifies workflow visibility as a condition associated with stronger, more stable GUI performance.

Limitations

The evaluation isolates modality under matched constraints rather than estimating unconstrained production-agent performance. Its patched-skill analysis and failure taxonomy are diagnostic tools, not fully deployable or perfectly separable accounts of agent behavior.

  • Evaluation scope: The benchmark restricts agents to modality-native actions, so results do not estimate unconstrained systems combining screenshots, terminals, scripts, file edits, databases, or APIs.Hybrid strategies may be effective but obscure whether success comes from GUI interaction, programmatic control, or bypassing the application interface.
  • Skill augmentation: Patched-skill results diagnose verifier-readable skill coverage, but do not guarantee autonomous selection and composition of the right skill calls.Repairs use verifier-observed requirements and validate against the same checkpoints, making this setting unsuitable as a deployable baseline.
  • Failure analysis: Failure-taxonomy proportions are coarse because GUI failures often combine control discovery and workflow execution despite assigning each sampled failure one primary category.An agent may miss a menu because it lacks workflow knowledge, or fail the workflow because it never discovers the necessary control.

A Appendix … A.5 Prompts

The appendix examines modality-specific execution through divergent case studies, verifier-coverage auditing, a structural visual-design example, grounded failure examples, and prompts that enforce native action spaces. It shows that CLI performance depends on skill coverage, while GUI failures can arise from long visual construction chains.

  • A.1 Case Studies: CLI skills provide compact, structured information, but their effectiveness depends strongly on the underlying skill layer’s quality and coverage.The appendix identifies a substantial gap between generated skills and robust real-world execution, raising scalability concerns.
  • A.1 Case Studies: CLI agents complete the four-page draw.io task through direct page, shape, connector, and structure operations, while GUI agents exhaust their step budget during the long visual construction chain.Observed GUI trajectories leave required pages, labels, or connectors missing from the saved artifact.
  • A.1 Case Studies: GUI agents complete Audacity’s label-track task, whereas CLI agents fail to reliably create verifier-visible label tracks and entries in the persisted project state.Failed CLI runs often preserve the original project and wavetrack while leaving label-track checks unsatisfied.
  • A.2 Verifier Coverage Pipeline: 37.6% original-skill coverage rises to 100% improved-skill coverage after repairing and validating Partial and Fail verifier checkpoints.Coverage is computed over verifier checkpoints, with the improved result using the same denominator.
  • A.2 Verifier Coverage Pipeline: The four-phase pipeline maps verifiers to skills, repairs and tests incomplete implementations, reports coverage, and updates documentation to match executable capabilities.The process requires manual inspection and approval before advancing, followed by an app-level comprehensive test before an application counts as fully covered.
  • A.3 Visual Design Example: The Kubernetes diagram task is visually presented but chiefly verified through 12 vertices, specified labels, and at least 9 directed edges encoding the cluster hierarchy.CLI operations can add labeled shapes and connect source-target pairs directly, while GUI execution must place, label, and connect shapes on screen.
  • A.4 Grounded Examples for the Error Taxonomy: Tables 5 and 6 provide representative CLI and GUI failure cases grounded in execution trajectories and verifier outputs.These examples support the error taxonomy in the main text.
  • A.5 Prompts: The benchmark prompts restrict agents to skill-based CLI actions or screen-level GUI interactions, preventing cross-modality leakage.The prompts also specify an Ubuntu x86_64 environment with internet access and instruct agents to act without requesting clarification.
Loading 2606.24551v1…