PaperlayerView exact revision on arXiv

Source-linked AI summary

Efficient GUI Agents: A Systems Survey of Observation, Memory, Action, and Runtime Optimization

Bizhe Bai, Jiakang Yuan, Hongming Wu, Xinyue Wang, Jie Ren, Siyao Chen, Yuchen Ya, Fan Bai, Pai Peng, Huafeng Qin, Tao Chen

arXiv:2609.02309v1cs.CL

TL;DR

GUI-agent research often emphasizes task success despite substantial context, computation, action, and runtime costs. This survey organizes efficient GUI agents across four system axes and synthesizes mechanisms, efficiency signals, and introduced overheads. Across the literature, progress centers on selective observation, compact memory, verified or abstracted action, and hybrid execution, while evaluation remains difficult because efficiency accounting is incomplete.

  • Problem

    GUI-agent progress is commonly reported through task success even though practical deployment also depends on the context, computation, action budget, and runtime overhead consumed while succeeding.

  • Method

    The survey uses an end-to-end systems taxonomy spanning observation, context and memory, action, and planner-side/system efficiency, expanded through targeted search and citation chaining.

  • Results

    The literature converges on selective reading, global-to-local visual allocation, recoverable memory, verification-aware control, and runtimes that can switch between GUI and non-GUI execution.

  • Takeaways & Limitations

    Efficient GUI agents require coordinated optimization across observation, memory, action, and runtime layers rather than success-only evaluation.

  • Takeaways & Limitations

    The field lacks shared efficiency accounting, leaving verifier calls, parser overhead, and multi-agent orchestration without a common pricing framework.

Abstract

from arXiv · show

GUI agents increasingly operate across websites, mobile apps, and desktop environments, yet the field still reports progress primarily through task success. We argue that practical deployment depends equally on efficiency: how much context, computation, action budget, and runtime overhead an agent consumes while succeeding. This survey studies efficient GUI agents through an end-to-end systems lens that preserves the current technical axes of observation efficiency, context and memory efficiency, action efficiency, and planner-side/system efficiency. For each subsection, we expand the seed literature through targeted search plus backward and forward citation chaining, then synthesize the dominant mechanisms, reported efficiency signals, and new overheads they introduce. Across the literature, recent progress converges on a small set of recurring ideas: selective reading instead of full-context ingestion, global-to-local visual allocation, recoverable memory rather than raw history replay, verification-aware control, and hybrid runtimes that can switch between GUI and non-GUI execution. We conclude by identifying the main open problems, including honest accounting of verifier cost, cross-benchmark comparability, and co-design of observation, memory, and execution layers under real latency and privacy constraints.

1 Introduction

GUI agents now handle increasingly realistic, multi-step computer-use tasks, but their practical deployment is constrained by substantial interaction and runtime costs. This survey therefore organizes efficiency across observation, memory, action, and planner/system layers.

  • Motivation: GUI agents operate across websites, mobile applications, and desktop systems while completing increasingly realistic multi-step workflows.These capabilities are evaluated by benchmarks covering web navigation, mobile-app control, and open-ended operating-system tasks.
  • Efficiency Gap: Many agents incur high costs through repeated observation–reasoning–action cycles, redundant operations, repeated model calls, and long waits.OSWorld-Human reports that planning, judging, and reflection substantially contribute to end-to-end latency.
  • Why GUI Efficiency Matters: GUI agents require multimodal observations and must manage visual grounding, partial observability, long-horizon state tracking, and noisy interface representations.Their observations and actions may involve screenshots, DOM or HTML, accessibility trees, window metadata, and interface-level operations.
  • Why GUI Efficiency Matters: Efficiency also affects deployability and privacy because practical systems may need to minimize exposed interface content, cloud traffic, and runtime overhead.These concerns extend beyond task completion to the resources and information consumed during interaction.
  • Survey Scope: The survey contributes an efficiency-centered taxonomy and a synthesis of cross-layer tradeoffs, including savings offset by parser, verifier, retriever, or orchestration costs.The taxonomy spans observation, context and memory, action, and planner-side/system optimization.

2 Preliminaries

The survey models GUI agents as closed loops that observe interfaces, retain context, plan, act, and verify outcomes. Its efficiency framework distinguishes resource costs across observation, memory, action, and planner/system execution.

  • From Agents to GUI Agents: GUI agents receive a user goal, observe a graphical interface, reason about the next step, and execute an interface-level action.Observations can include screenshots, DOM or HTML, accessibility trees, and related metadata.
  • System View: A GUI-agent system is a loop of observation, context and memory, planning, grounding and action execution, and verification.Figure 1 maps the survey’s four efficiency axes onto these stages.
  • Functional Pipeline: Most GUI agents construct an actionable interface representation, retain relevant context, reason about a subgoal, ground and execute an action, then verify its effect.The pipeline may use screenshots, DOM, accessibility trees, or hybrid parsing outputs.
  • Efficiency Axes: Observation efficiency limits redundant interface representation, memory efficiency compresses historical information, and action efficiency reduces the cost of reaching the goal.Planner-side and system efficiency covers reasoning depth, orchestration, backend routing, and end-to-end runtime.
  • Efficiency Metrics: Objective evaluation separates effectiveness, which measures task quality, from efficiency, which measures the resources spent to obtain that quality.The survey defines latency, observation, and context-and-memory metrics for this separation.

3 Observation Efficiency

Observation efficiency seeks decision-sufficient interface representations with less redundancy and ambiguity. The literature reduces textual interface cost through selective retrieval and pruning, while screenshot methods allocate visual resolution toward actionable regions.

  • Overview: Observation efficiency targets decision-sufficient representations while minimizing redundancy, grounding ambiguity, and downstream reasoning cost.Inefficiency can arise from long textual structures, cluttered screenshots, or weakly structured multimodal inputs.
  • Structured Interface Representations: Raw web representations can be extremely large, with DOM inputs reaching 800k tokens and typical tasks requiring 150–220 seconds and about 25 LLM calls.Hierarchical distillation and denoising are used to make these representations more manageable.
  • Structured Interface Representations: DOM and accessibility-tree methods reduce large textual observations through downsampling, selective retrieval, and executable filtering.FocusAgent reports more than 50% average AxTree reduction, while Prune4Web can reduce over 500 candidates to fewer than 20.
  • Region-Focused Visual Perception: Screenshot-based efficiency is primarily a visual-allocation problem because actionable targets may be small amid repeated widgets and irrelevant background regions.Region-focused methods use sub-image partitioning, zoomed proposals, and region-aware grounding objectives.
  • Actionable Observations: Set-of-Mark and ScreenAI add referential or textual annotations to screen elements to make screenshot observations more actionable.ScreenAI also illustrates the scale of screen-understanding backbones.

4 Context and Memory Efficiency

Context and memory efficiency replaces costly raw trajectory replay with compact, recoverable representations and runtime compression. Methods range from progress summaries and diff histories to GUI-specific KV-cache policies and continuous embeddings.

  • Historical Context: Context and memory efficiency preserves, accesses, and updates historical information without letting context growth dominate runtime and memory consumption.Long GUI trajectories make replaying screenshots, actions, and reasoning traces costly and noisy.
  • Historical Context: Agent-S, ColorBrowserAgent, and GUI-Rise replace raw trajectory replay with task-progress memories, progressive summaries, or progress-aware representations.Diff-based history can be more token-efficient than replaying full observations, especially under short look-back settings.
  • Runtime-Level Representation Compression: GUI-KV exploits spatial and temporal redundancy in GUI trajectories to reduce serving cost, reporting 38.9% fewer MFLOPs per decoded token at five screenshots.It uses 5–20% cache budgets, while five screenshots can exceed 80GB of GPU memory.
  • Runtime-Level Representation Compression: Runtime-level representation compression improves scalability even when the high-level memory policy remains unchanged.This makes serving efficiency a distinct optimization target from memory selection or retention policy.
  • Dense Memory Representations: Continuous-memory methods compress trajectories into fixed-length embeddings, while other approaches distill prior screenshots and actions into concise semantic context.Auto-scaling Continuous Memory compresses trajectories into eight embeddings even when raw trajectories exceed 15k tokens.

5 Action Efficiency

Action efficiency reduces GUI-agent cost by abstracting repeated operations, pruning or validating candidate actions, and making long-horizon execution recoverable. The literature also explores structured backtracking and constrained exploration to reduce wasted interaction.

  • 5.1 Action Abstraction: Action abstraction compiles repeated GUI operations into reusable skills, routines, or executable programs that reduce interaction cost.SkillWeaver and PolySkill target reusable procedures, while ActionEngine compiles frequent interaction patterns into state-machine memory and programs.
  • 5.1 Action Abstraction: $0.71 to $0.06 cost, 237.5 to 118.3 seconds latency, 62.3k to 8.1k input tokens, and 10.2 to 1.8 model calls were reported after ActionEngine compilation.These figures compare repeated reactive planning with state-machine memory and executable programs.
  • 5.1 Action Abstraction: Average steps fell to 10.15 with CoAct-1, versus 15.22 for GTA-1 and 14.90 for UI-TARS, by routing subtasks between GUI operation and code execution.The hybrid modality targets subtasks such as file manipulation or data processing that can be inefficient as GUI-only trajectories.
  • 5.2 Candidate Reduction and Verification: Early DOM pruning can shrink more than 500 elements to fewer than 20 candidates, reducing observation size and downstream action search.Candidate scoring and verification methods further assess actions before expensive reasoning or irreversible execution.
  • 5.3 Recovery and Exploration: Backtracking, rollback, verification, judging, and reflection help contain cascading failures in long-horizon GUI trajectories.LongHorizonUI evaluates rollback-oriented execution over average trajectories of 24.6 steps and maximum trajectories of 37 steps.
  • 5.3 Recovery and Exploration: Explicit backtracking and compact intent representations constrain exploration and make recovery from off-trajectory decisions more systematic than unrestricted trial and error.LASER models interaction as state-space exploration, while Auto-Intent guides self-exploration with higher-level latent goals.

6 Planner-Side and System Efficiency

Planner-side efficiency targets reasoning and reflection overhead inside the decision loop, while system efficiency addresses memory reuse, backend routing, privacy-aware deployment, and orchestration. Reported approaches selectively invoke expensive reasoning and combine heterogeneous execution paths.

  • 6 Planner-Side and System Efficiency: Planner-side efficiency concerns decision-loop computation, whereas system efficiency includes memory reuse, privacy-aware routing, and orchestration across components.This distinction separates internal planning burden from broader runtime overhead.
  • 6.1 Planner-Side Efficiency: Agents required 2.7×–4.3× more steps than human-derived trajectories, and reflection accounted for 76%–96% of task latency on OSWorld-Human.These measurements motivate treating planner-side efficiency as a first-class systems objective.
  • 6.1 Planner-Side Efficiency: UI-R1 used 136 training samples and about 8 hours on 8 RTX 4090 GPUs, while high-pixel settings exposed out-of-memory risk.The result illustrates that compact training procedures can still encounter hardware constraints.
  • 6.1 Planner-Side Efficiency: Selective reasoning reduced unnecessary reasoning tokens by 40% in AdaGUI-R1 but added 23.5% FLOPs under its harder-example schedule.Think Twice, Click Once similarly reports processing time increasing from 2.6 to 5.4 seconds when slow thinking is invoked.
  • 6.1 Planner-Side Efficiency: On-demand hierarchical reflection reduced MobileUse reflection overhead to 10%, although exploration could cost up to 100 steps per app at 19.5 seconds per step.Other systems factorize planning or integrate reasoning and reflection within compact generalist models.
  • 6.2 System Efficiency: ActionEngine amortizes online reasoning through program synthesis, while CoAct-1 routes subtasks between GUI and code backends when their costs differ.CORE and GUIGuard also treat local/cloud deployment topology and privacy recognition as runtime design choices.

7 Open Challenges and Future Directions

The survey identifies missing standardized accounting for efficiency and insufficiently detailed benchmarks as central obstacles to comparing GUI agents and judging deployability. It calls for reporting resource costs beyond success rate, tokens, and steps.

  • 7 Open Challenges and Future Directions: Many papers report local savings in tokens, actions, or module latency, but benchmarks often stop at success rate or aggregate completion.This makes cross-paper efficiency comparison difficult, especially when verifier, parser, and orchestration costs are omitted.
  • 7 Open Challenges and Future Directions: Verifier calls, parser overhead, and multi-agent orchestration still lack a shared pricing framework for efficiency accounting.OSWorld-Human and MMBench-GUI provide more direct profiling but do not resolve this accounting gap.
  • 7 Open Challenges and Future Directions: Future benchmarks should report peak GPU memory, prefill and decode latency, MFLOPs per decoded token, training or search GPU-hours, and success-normalized GPU cost.These measurements are intended to assess deployability under realistic serving constraints rather than efficiency only in tokens or steps.

A GUI-agent

The survey models a GUI-agent episode as a system-level trajectory linking latent state, observations, retained memory, actions, and verification signals. Its four efficiency axes correspond to reducing observation, memory, action, and planner/runtime costs over that trajectory.

  • A GUI-agent: A GUI-agent episode is represented as a trajectory of latent states, GUI observations, retained memories, executed actions, and verification signals.The trajectory abstraction captures deployment cost across the full interaction rather than only a single next-action prediction.
  • A GUI-agent: The observation module may use screenshots, DOM, HTML, accessibility trees, parsers, or hybrid perception, while memory supports summarization, retrieval, compression, or KV-cache reuse.The planner or orchestrator selects actions, the environment transitions, and verification provides reflection or progress feedback.
  • A GUI-agent: The trajectory outcome is binary success or a graded completion score, and Sections 3–6 reduce its cost through cheaper observations, memory updates, actions, or planner/runtime overhead.This frames the survey taxonomy as four ways of optimizing one end-to-end interaction process.
  • A GUI-agent: The tree-structured taxonomy overview and section-wise paper summary tables are collected in the appendix for main-text readability.The appendix serves as the location for the survey’s detailed organizational material.

B Literature Expansion Protocol

The survey expands literature subsection by subsection rather than treating search as a single flat retrieval problem. It combines seed citations, targeted searches, and backward and forward citation chaining, retaining work that directly addresses efficiency in each subsection.

  • The protocol treats literature search as subsection-specific rather than as one flat retrieval task.
  • Literature expansion starts from papers already cited in each subsection, then adds targeted searches and backward and forward citation chaining.Searches covered arXiv, ACL Anthology, and OpenReview.
  • Papers are retained only when they directly address observation, memory, action, or planner/runtime efficiency in the relevant subsection.The protocol excludes generic GUI grounding, pure benchmark-generation, and unrelated long-horizon agent work unless directly relevant.

C Added Works by Subsection

Table 1 organizes additional works integrated beyond the seed draft according to the existing subsection taxonomy.

  • Additional works are grouped by the existing subsection taxonomy.

D Taxonomy Figure and Section Summary Tables

The appendix organizes efficiency-oriented GUI-agent work across observation, memory, action, and planner/system concerns, while recording reported metrics and unreported overheads. Its tables function as an evidence ledger rather than a direct meta-analysis.

  • Reporting conventions: The appendix warns that reported percentages, speedups, token counts, costs, and step counts are copied from author baselines and are not directly comparable meta-analytic estimates.NR denotes that a source paper does not report a comparable quantitative efficiency metric.
  • Observation efficiency: Observation-efficiency methods reduce interface burden through DOM filtering, visual-token pruning, region allocation, adaptive views, and parser-based representations.Reported examples include shrinking DOMs from more than 500 elements to fewer than 20, removing 33% of redundant visual tokens, and using up to 8122 input resolution.
  • Context and memory efficiency: Context- and memory-efficiency methods compress histories through cache pruning, fixed-length latent memories, bounded context, and semantic summaries.GUI-KV reports 38.9% fewer MFLOPs per decoded token at five screenshots, ST-Lite reports 2.45× decoding and 1.40× end-to-end speedups, and continuous memory maps trajectories to eight embeddings.
Loading 2609.02309v1…