PaperlayerView exact revision on arXiv

Source-linked AI summary

AppAgentX: Evolving GUI Agents as Proficient Smartphone Users

Wenjia Jiang, Yangyang Zhuang, Chenxi Song, Xu Yang, Joey Tianyi Zhou, Chi Zhang

arXiv:2503.02268v3cs.AI

TL;DR

LLM-based GUI agents are flexible but inefficient when routine tasks require repeated step-by-step reasoning, while rule-based automation lacks adaptability. AppAgentX records execution history, evolves shortcut actions for repetitive sequences, and reports higher efficiency and accuracy than existing methods across benchmark tasks.

  • Problem

    LLM-based GUI agents can require inefficient step-by-step reasoning for routine tasks, while rule-based systems lack flexibility for novel scenarios.

    Source: §1 · p. 2

  • Method

    AppAgentX records execution history in a chain and abstracts repetitive low-level action sequences into high-level shortcut actions.

    Source: §1 · p. 2

  • Results

    AppAgentX outperforms baseline and state-of-the-art methods in efficiency and accuracy across several benchmark tasks.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    The framework balances intelligent reasoning with efficient execution by compressing repetitive operations into evolved high-level actions.

    Source: §1 · p. 2, §6 · p. 9

    Time comparisons include only tasks successfully executed by all methods, so failures from early termination or excessive retries are excluded.

    Source: §5.1 · p. 6

Abstract

from arXiv · show

Recent advancements in Large Language Models (LLMs) have led to the development of intelligent LLM-based agents capable of interacting with graphical user interfaces (GUIs). These agents demonstrate strong reasoning and adaptability, enabling them to perform complex tasks that traditionally required predefined rules. However, the reliance on step-by-step reasoning in LLM-based agents often results in inefficiencies, particularly for routine tasks. In contrast, traditional rule-based systems excel in efficiency but lack the intelligence and flexibility to adapt to novel scenarios. To address this challenge, we propose a novel evolutionary framework for GUI agents that enhances operational efficiency while retaining intelligence and flexibility. Our approach incorporates a memory mechanism that records the agent's task execution history. By analyzing this history, the agent identifies repetitive action sequences and evolves high-level actions that act as shortcuts, replacing these low-level operations and improving efficiency. This allows the agent to focus on tasks requiring more complex reasoning, while simplifying routine actions. Experimental results on multiple benchmark tasks demonstrate that our approach significantly outperforms existing methods in both efficiency and accuracy. The code will be open-sourced to support further research.

1 Introduction

LLM-based GUI agents offer flexibility and reasoning, but step-by-step operation makes routine tasks inefficient. AppAgentX addresses this trade-off by learning high-level shortcuts from execution history while retaining adaptability, and reports improved efficiency and accuracy.

Motivation

  • Step-by-step reasoning improves action selection but introduces delays for routine GUI tasks, whereas rule-based automation is efficient yet inflexible.Search operations may require separately clicking, typing, and submitting even when advanced reasoning is unnecessary.Source: §1 · p. 2

Approach

  • AppAgentX evolves high-level actions by identifying repetitive low-level sequences in execution history and replacing them with compact shortcuts.For example, repeated search interactions can become a single high-level search action.Source: §1 · p. 2
  • A chain-based knowledge framework records task execution history to support abstraction, behavior evolution, and continued strategy refinement.The framework is designed to let the agent focus reasoning on tasks requiring more intelligent judgments while simplifying repetitive operations.Source: §1 · p. 2

Results

  • Experiments across benchmark tasks report that the visual-only approach outperforms baseline and state-of-the-art methods in both efficiency and accuracy.The approach does not require backend access or APIs.Source: §1 · p. 2

Contributions

  • The paper proposes an evolutionary mechanism for learning from task execution history and abstracting repetitive operations.The code is intended to be open-sourced for further research.Source: §1 · p. 2

2 Related Works

Prior work shows that LLM agents can generalize across tasks and interfaces, including GUI automation, but most existing agents lack continuous adaptation. AppAgentX addresses this gap with task-trajectory memory for improved GUI efficiency and adaptability.

LLMs and agents

  • LLM-powered agents support complex task execution and generalization across diverse applications and dynamic interfaces, unlike explicitly predefined rule-based systems.The related work covers autonomous task decomposition, multimodal interaction, and domain-specific agents.Source: §2 · pp. 2–3

LLMs as GUI Agents

  • Existing GUI agents improve interface interaction, perception, precision, and cross-platform operation through multimodal and collaborative approaches.Examples include WebVoyager, AppAgent, MobileAgent, OmniParser, Ferret-UI, TinyClick, and CogAgent.Source: §2 · p. 3

Research gap

  • Most existing agents rely on static training data and lack continuous adaptation, motivating AppAgentX’s use of task-trajectory memory.The proposed direction targets efficiency and adaptability in GUI automation.Source: §2 · p. 3

3 Preliminary

The baseline controls smartphones through iterative screen perception and low-level action execution. The LLM interprets annotated interfaces, selects actions from an app-agnostic action space, and repeats this cycle until task completion.

Screen perception

  • The baseline first captures and annotates screenshots with interactive elements before passing them to the LLM for action planning.OmniParser detects elements such as buttons and text boxes and overlays tagged bounding boxes on the screenshot.Source: §3 · p. 3

Action execution

  • The basic action space contains low-level smartphone interactions including tapping, long-pressing, swiping, text input, and back navigation.These actions form an app-agnostic set of typical human interface interactions.Source: §3 · p. 3
  • Abasic denotes the set of atomic actions available to the agent.The formal action-space definition is introduced for the baseline execution stage.Source: §3 · p. 3

Execution cycle

  • The agent iteratively observes the current UI, reasons about the next step, invokes a corresponding action, and continues until successful completion.This observation–reasoning–function-calling cycle is illustrated in Figure 2.Source: §3 · p. 3

4 Methodology

The methodology combines trajectory memory, shortcut evolution, and dynamic execution to replace repeated reasoning with reusable high-level actions. It preserves basic-action fallback when shortcut conditions or execution outcomes are unsuitable.

4.1 Memory mechanism

  • The memory mechanism records page transitions, UI elements, actions, and outcomes in a chain so the agent can learn from past interactions.Page and element nodes store descriptions, detected-element information, interaction details, and logging attributes.Source: §4.1 · p. 4
  • Trajectory execution is decomposed into overlapping source-page, action-element, and target-page triples, whose LLM-generated descriptions are merged into unified page records.This creates context-aware descriptions and coherent chain attributes for the agent’s interaction history.Source: §4.1 · p. 4

4.2 Evolutionary Mechanism

  • The evolutionary mechanism identifies repetitive patterns, generates shortcut-node descriptions and invocation conditions, and adds the resulting high-level actions to the action space.Shortcut nodes replace inefficient low-level sequences and reduce unnecessary reasoning.Source: §4.2 · p. 5

4.3.1 Execution Process

  • During execution, visual embedding matches connect current UI elements to shortcut nodes, after which the LLM checks task context and generates executable low-level templates.When multiple matches exist, execution follows the matched elements’ sequence order.Source: §4.3.1 · p. 6

4.3.2 Fallback Strategy

  • If shortcut conditions are unmet or execution errors occur, the agent reverts to the basic action space as a fallback.This strategy is intended to preserve operation when high-level abstractions cannot be applied.Source: §4.3.2 · p. 6
  • High-level paths convert repeated LLM reasoning into page matching and retrieval of predetermined action sequences, improving execution efficiency.The resulting process can complete partially repetitive operations or entire tasks more efficiently.Source: §4.3.2 · p. 6, §4.3.1 · p. 6

5 Experiments

The experiments evaluate AppAgentX across multiple benchmarks using success, efficiency, and token metrics, finding that memory and evolved high-level actions improve performance. AppAgentX outperforms baselines across task lengths, benchmarks, LLM backends, and action-space variants.

5.1 Experimental Setup

  • The experimental setup reports average steps, success rate, task time, step time, and LLM token consumption, averaging five repetitions and restricting time comparisons to jointly successful tasks.The benchmarks span 50 AppAgent tasks, 158 DroidTask tasks, 116 AndroidWorld tasks, and 201 A3 tasks.Source: §5.1 · pp. 6–7

5.2 Experimental Analysis

  • 71.4% average success rate accompanies reductions in average steps from 9.1 to 5.7, step time from 23 to 16 seconds, and token consumption from 9.26k to 4.94k.The component analysis attributes success-rate gains to memory and efficiency gains to high-level actions.Source: §5.2 · pp. 7–8, Table 1 · p. 7
  • AppAgentX improves efficiency and accuracy over baselines and state-of-the-art methods across several benchmark tasks.The evaluation covers AppAgent, DroidTask, AndroidWorld, and Android Agent Arena, with comparisons emphasizing execution time and success rates.Source: §5.1 · p. 7, §5.2 · p. 8
  • Per-step execution is faster than previous state-of-the-art and AppAgent across GPT-4o, Claude 3.5 Sonnet, and Gemini 1.5 Pro backends.The comparison focuses on execution time as a practical and reproducible efficiency metric.Source: §5.2 · p. 8, Figure 4 · p. 8
  • AppAgentX consistently reduces task completion time versus the baseline for short, medium, and long tasks, with all paired-test p-values below 0.05.The advantage becomes clearer as task complexity increases.Source: §5.2 · p. 8, Figure 5 · p. 8
  • The full evolved action space significantly reduces execution time while improving success rates compared with the basic action space, including when screen perceptrons are replaced.The ablation compares OmniParser and Ferret-UI with basic and full action spaces.Source: §5.2 · p. 9, Table 3 · p. 9
  • Qualitative Gmail executions use shortcuts to reduce per-step LLM reasoning while completing tasks across different interfaces and actions.The workflow includes task retrieval, shortcut matching, and high-level action execution.Source: §5.2 · p. 9, Figure 6 · p. 9

6 Conclusion

The conclusion presents AppAgentX as an evolving GUI-agent framework that abstracts high-level actions from execution history. Its chain-based knowledge framework supports behavior refinement, while experiments report improved accuracy and efficiency over existing methods.

6 Conclusion

  • AppAgentX compresses repetitive operations into high-level actions, balancing intelligent reasoning with efficient execution.The framework learns from execution history rather than relying solely on rule-based automation.Source: §6 · p. 9
  • A chain-based knowledge framework enables continuous behavior refinement, reducing redundancy and improving adaptability.The conclusion links this refinement to the framework’s reported efficiency and intelligence.Source: §6 · p. 9
  • Experiments show that the framework outperforms existing methods in accuracy and efficiency.Source: §6 · p. 9

A Design and Execution Details

This section introduces the detailed design of the chain structure and the system’s overall task-execution workflow.

A Design and Execution Details

A.1 Chain Design Details

The chain design models pages, elements, shortcuts, and transitions through typed relationships that support memory organization and action evolution.

A.1 Chain Design Details

  • Page-HAS_ELEMENT→Element represents that a page contains a specific UI element.Source: §A.1 · p. 11
  • Shortcut-COMPOSED_OF→Element represents a composite action composed of ordered element operations and their atomic-action parameters.The stored properties include execution order, atomic action type, and JSON-formatted action parameters.Source: §A.1 · p. 11
  • Element-LEADS_TO→Page links an element to the page reached through the composite action.Source: §A.1 · pp. 11–12
  • Together, these typed edges form the memory store and support analysis of evolving actions across page transitions and containment relationships.The overall workflow adds a shortcut route between task input and final execution result.Source: §A.1 · p. 12, Figure 7 · p. 12

A.2 Execution Process Details

The workflow combines page analysis, element comparison, matching, and memory storage within a hierarchical action framework.

  • The workflow includes page analysis, element comparison, matching, and memory storage.Source: §A.2 · p. 12
  • High-level nodes are paired with corresponding basic action spaces.Source: §A.2 · p. 12
  • When high-level matching fails, the system transitions to the corresponding basic action space.Source: §A.2 · p. 12

B Detailed Numerical Results

The numerical analysis compares per-step execution times across frameworks and LLM backends, then tests whether AppAgentX reduces task completion times.

C Additional Quantitative Analysis

A qualitative Apple Music example shows AppAgentX executing a task through existing shortcuts, reducing per-step LLM reasoning and improving efficiency.

Loading 2503.02268v3…