PaperlayerView exact revision on arXiv

Source-linked AI summary

ASSISTGUI: Task-Oriented Desktop Graphical User Interface Automation

Difei Gao, Lei Ji, Zechen Bai, Mingyu Ouyang, Peiran Li, Dongxing Mao, Qinchen Wu, Weichen Zhang, Peiyi Wang, Xiangwu Guo, Hengxu Wang, Luowei Zhou, Mike Zheng Shou

arXiv:2312.13108v2cs.CV

TL;DR

Desktop GUI automation for productivity software remains distinct from web and Android tasks because it involves dense interfaces, complex operations, and long procedures. The paper introduces ASSISTGUI, a 100-task benchmark, and the ACE agent framework. Experiments indicate promising potential while also highlighting limitations in current models and the complexity of the task.

  • Problem

    Prior GUI automation work focused mainly on web or Android tasks, while desktop productivity tasks involve dense interfaces, complex operations, and long procedures.

    Source: §1 · p. 2

  • Method

    The paper constructs ASSISTGUI with 100 application tasks and presents ACE, an embodied agent combining GUI parsing with hierarchical reasoning.

    Source: §1 · p. 2

  • Results

    Experiments show ACE's promising potential while underscoring task complexity and limitations in current LLMs, LMMs, and LLM-based agents.

    Source: §1 · p. 2, §7 · p. 12

  • Takeaways & Limitations

    ASSISTGUI provides a benchmark and baseline for assessing desktop GUI automation, while the experiments identify challenges for existing models.

    Source: §1 · p. 2

    Planner predictions can include redundant operations, and the model still lacks the capability to perceive the middle case accurately.

    Source: §5.2 · pp. 8, 11

Abstract

from arXiv · show

Graphical User Interface (GUI) automation holds significant promise for assisting users with complex tasks, thereby boosting human productivity. Existing works leveraging Large Language Model (LLM) or LLM-based AI agents have shown capabilities in automating tasks on Android and Web platforms. However, these tasks are primarily aimed at simple device usage and entertainment operations. This paper presents a novel benchmark, AssistGUI, to evaluate whether models are capable of manipulating the mouse and keyboard on the Windows platform in response to user-requested tasks. We carefully collected a set of 100 tasks from nine widely-used software applications, such as, After Effects and MS Word, each accompanied by the necessary project files for better evaluation. Moreover, we propose an advanced Actor-Critic Embodied Agent framework, which incorporates a sophisticated GUI parser driven by an LLM-agent and an enhanced reasoning mechanism adept at handling lengthy procedural tasks. Our experimental results reveal that our GUI Parser and Reasoning mechanism outshine existing methods in performance. Nevertheless, the potential remains substantial, with the best model attaining only a 46% success rate on our benchmark. We conclude with a thorough analysis of the current methods' limitations, setting the stage for future breakthroughs in this domain.

1. Introduction

The paper targets desktop GUI automation for productivity software, where dense interfaces, complex operations, and long procedures pose challenges beyond many web and Android tasks. It introduces ASSISTGUI and the ACE baseline to study this setting.

1. Introduction

  • Desktop productivity tasks require dense GUI understanding, complex operations, and long procedures, unlike the simpler interactions common in web and Android automation.The paper frames this as an important but largely unexplored domain for evaluating model performance on productivity software.Source: §1 · p. 2
  • ASSISTGUI supplies instructional videos alongside user queries so models can adapt demonstrated techniques, including customized functions in applications such as After Effects.The model is expected to modify the software to fulfill the query; videos are intended to help it acquire usage techniques.Source: §1 · p. 2
  • ASSISTGUI pairs 100 tasks across nine applications with queries, instructional videos, and project files, while ACE provides a baseline with GUI parsing and hierarchical reasoning.The paper also reports extensive experiments assessing the approach and highlighting challenges for existing models.Source: §1 · p. 2
  • ACE combines an LLM-driven GUI parser with hierarchical task decomposition and stepwise evaluation to adjust future actions.The framework draws on LLM-based agents and the Actor-Critic concept; experiments and ablations also expose limitations in current models and agents.Source: §1 · p. 2

2. Related Work

Prior UI automation research spans mobile and web environments, terminal tasks, and LLM-based agents, but desktop GUI automation brings complex graphical interaction and extended procedures. The paper positions its approach as a specialized agent for that setting.

2. Related Work

  • Earlier UI automation work developed interactive Android environments and benchmarks with increasingly diverse tasks [40] [5] [16] [26] [42].Simulated web environments and terminal benchmarks such as NL2Bash and AgentBench provide other interaction settings [6, 25, 33, 50, 54] [17, 19].Source: §2 · pp. 2–3
  • LLM-agent research combines reasoning with APIs, tools, or visual models, while this paper targets a specialized LMM-based agent for desktop GUI automation [24] [28] [34] [41] [51] [7] [20] [32] [36] [45] [49].Prior examples include CoT, ReAct, visual question answering, video summarization, and image editing [45].Source: §2 · p. 3
  • End-to-end UI methods use reinforcement or imitation learning [8] [10], while two-stage methods parse interface elements before generating executable steps [1] [3] [9] [14] [11] [15] [50] [54].The paper identifies limited desktop-GUI coverage and difficulty with long processes as challenges for these approaches, motivating diverse parsing tools and hierarchical planning with a critic.Source: §2 · p. 3

3. ASSISTGUI Benchmark

ASSISTGUI is a real-world interactive benchmark for broad desktop tasks with goal-oriented evaluation. It defines tasks around a query, instructional video, and application, then provides a diverse collection and task-specific success measures.

3. ASSISTGUI Benchmark

  • ASSISTGUI provides a real-world interactive environment, a dataset spanning broad tasks, and goal-oriented evaluation.The benchmark is designed for desktop task automation.Source: §3 · p. 3

3.1. Task Formulation

  • A task maps a user query, instructional video, and relevant application to a sequence of UI actions that fulfills the request.The environment observation combines operating-system metadata, which mainly exposes panel and pop-up layout, with screenshots that provide broader visual context; metadata may miss specific text and buttons.Source: §3.1 · p. 3

3.2. Data Collection

  • The benchmark collects 100 tasks from nine applications, using selected instructional videos and manually written queries that may add customized requirements beyond the demonstrated steps.Project files support reproducible starting states for editing tasks, and annotators check videos and files for correctness and functionality.Source: §3.2 · pp. 4–5

3.3. Evaluation

  • Evaluation uses task-specific success metrics because outputs range from videos and documents to software states, system settings, and folder structures.Design and Office outputs are compared with ground truth at pixel granularity; widget screenshots are checked in the display region, while scripts assess system settings and folder structure.Source: §3.3 · p. 5

4. Method

ACE uses an LLM-based agent to perceive the desktop environment, plan actions, and execute tasks through a high-level plan and collaborative modules. The Planner structures the task, while the GUI Parser, Critic, and Actor support sequential execution and adjustment.

4. Method

  • ACE first creates a high-level plan from a query and video, then uses the GUI Parser, Critic, and Actor to observe, assess, and execute subtasks.The Actor adjusts the plan based on the Critic’s assessment and generates code to control the desktop.Source: §4 · p. 5

4.1. Planner

  • The Planner derives milestones and subtasks from video subtitles, refines them against the user’s query, and traverses leaf subtasks in order.The resulting hierarchical task tree pairs each milestone with a list of subtasks.Source: §4.1 · p. 5, Figure 5 · p. 6

4.2. GUI Parser

  • The GUI Parser converts screenshots into a panel-by-panel textual representation by combining system metadata, OCR, icon matching, and vision tools.It localizes objects in footage and records UI-element meanings and positions.Source: §4.2 · p. 6

4.3. Critic

  • The Critic evaluates an action by comparing screenshots before and after execution, then sends success and finish flags with explanations to the Actor.The flags indicate whether the action succeeded and whether the current subtask is complete.Source: §4.3 · p. 6

4.4. Actor

  • The Actor keeps or advances the current subtask according to the Critic’s finish flag, then generates a Python action using the current observation, subtask, milestone, and feedback.An output may be a single action or a sequence of actions.Source: §4.4 · pp. 6–7

5. Experiments

Experiments compare planning and module variants on ASSISTGUI, then examine qualitative examples. The results show an advantage over prior planning methods alongside persistent limitations in GUI perception and reasoning.

5.1. Quantitative Results

  • The model significantly surpasses previous planning methods on ASSISTGUI, although the benchmark remains challenging for complex productivity tools.Table 1 reports success rates for planning methods; its caption also identifies Human* as a reference from three non-expert humans.Source: §5.1 · p. 7, Table 1 · p. 7
  • Planner and Critic effects were clearer on complex Office and Design tasks; Critic gains were smaller than expected, while instructional videos helped with complex software.The Critic’s judgments were not always accurate on complex tasks, and instructional videos were especially helpful for After Effects and Premiere Pro.Source: §5.1 · pp. 7–8, Table 2 · p. 7
  • Removing OCR had the largest parser impact, icons mattered in Design and Office, and panel layout and other elements had little effect.The authors attribute the limited effect of other elements partly to the model’s difficulty completing complex footage operations.Source: §5.1 · p. 8, Table 3 · p. 8
  • GPT-3.5-turbo and Llama2-7B performed unsatisfactorily, with output-format failures and hallucinated action code identified as key issues.The authors suggest that lightweight models may need fine-tuning on specific datasets for these customized functions.Source: §5.1 · p. 8, Table 4 · p. 8

5.2. Qualitative Results

  • The GUI parser detects most elements but misses timestamp details, timeline bars, curves, and figures; GPT-4V-SoM performs poorly on desktop segmentation.The examples also show GPT-4V reading text but failing to provide button positions.Source: §5.2 · pp. 8, 10
  • Examples show continuous action sequences and simple-click judgments, but planners add redundant video steps and the Critic struggles with complex crop-completion judgments.Other reported errors include intricate footage operations, blurred or ambiguous regions, and selecting specific words within dense text.Source: §5.2 · pp. 8, 10, 11

6. Comparison with Previous Benchmarks

AssistGUI differs from prior Web and smartphone benchmarks by targeting desktop productivity software, whose dense interfaces and varied operations pose distinct challenges. It provides project files for its tasks so methods can start from the same state.

Platform

  • Desktop GUIs have denser information, more diverse visual elements, and more varied panel layouts than Web and smartphone interfaces [40] [42] [50].AssistGUI focuses on desktop operations rather than those platforms.Source: §6 · p. 11

Task Focus

  • Prior benchmarks [40] [42] [50] often cover games, Web navigation, or basic smartphone operations with relatively limited operation types.Game tasks often have clear rewards, while Web navigation mainly involves planning purchases.Source: §6 · p. 11
  • Productivity software may require new operation types, such as drawing on After Effects materials, challenging GUI understanding and action generation.This differs from the relatively limited operations in many game, Web, and smartphone tasks.Source: §6 · p. 11

Dataset Scale and Annotation

  • AssistGUI's roughly 100 tasks match the scale of previous benchmarks [40] [42] [50], but include project files to standardize initial states.The comparison table also identifies project files as a distinguishing feature of AssistGUI.Source: §6 · p. 11, Table 5 · p. 12

7. Conclusion

The paper introduces AssistGUI and an Actor-Critic Embodied Agent for desktop GUI automation. The framework outperforms existing methods, though substantial challenges remain.

7. Conclusion

  • The Actor-Critic Embodied Agent notably outperforms existing methods on GUI automation, while the results also highlight substantial remaining challenges.The conclusion does not state a numerical success rate.Source: §7 · p. 12
  • AssistGUI evaluates Windows mouse-and-keyboard task completion through 100 tasks spanning nine widely used applications, each with project files.Source: §7 · p. 12
  • The proposed framework combines an LLM-agent-driven GUI parser with an enhanced reasoning mechanism for complex, lengthy procedures in professional software.Source: §7 · p. 12

Related papers

Loading 2312.13108v2…