PaperlayerView exact revision on arXiv

Source-linked AI summary

AppAgent v2: Advanced Agent for Flexible Mobile Interactions

Yanda Li, Chi Zhang, Wenjia Jiang, Wanqi Yang, Bin Fu, Pei Cheng, Xin Chen, Ling Chen, Yunchao Wei

arXiv:2408.11824v4cs.HCcs.AI

TL;DR

Mobile agents can struggle with GUI recognition and unfamiliar, changing app interfaces. AppAgent combines parser and visual features with exploration-built UI knowledge retrieved and updated during deployment. Experiments report improved GUI recognition and task execution across diverse application environments.

  • Problem

    Mobile agents face GUI-recognition errors and difficulty handling unfamiliar apps and rapidly changing interfaces.

    Source: §1 · p. 1

  • Method

    AppAgent combines parser and visual features with exploration-built UI documentation stored in a structured knowledge base and retrieved or updated through RAG during deployment.

    Source: §1 · p. 2

  • Results

    Experiments report improved GUI recognition and task execution across diverse application environments.

    Source: §5 · p. 8

  • Takeaways & Limitations

    The framework’s parser-and-visual action space and two-phase knowledge workflow are presented as ways to manage dynamic mobile interfaces and diverse applications.

    Source: §5 · p. 8

    For hidden UI elements such as a video acceleration button, the agent may lack prior knowledge needed to detect and use them.

    Source: §6 · p. 9

Abstract

from arXiv · show

With the advancement of Multimodal Large Language Models (MLLM), LLM-driven visual agents are increasingly impacting software interfaces, particularly those with graphical user interfaces. This work introduces a novel LLM-based multimodal agent framework for mobile devices. This framework, capable of navigating mobile devices, emulates human-like interactions. Our agent constructs a flexible action space that enhances adaptability across various applications including parser, text and vision descriptions. The agent operates through two main phases: exploration and deployment. During the exploration phase, functionalities of user interface elements are documented either through agent-driven or manual explorations into a customized structured knowledge base. In the deployment phase, RAG technology enables efficient retrieval and update from this knowledge base, thereby empowering the agent to perform tasks effectively and accurately. This includes performing complex, multi-step operations across various applications, thereby demonstrating the framework's adaptability and precision in handling customized task workflows. Our experimental results across various benchmarks demonstrate the framework's superior performance, confirming its effectiveness in real-world scenarios. Our code will be open source soon.

1 Introduction

The introduction identifies inaccurate GUI recognition and unfamiliar, changing interfaces as barriers for mobile agents. AppAgent combines parser and visual inputs with a structured knowledge base and exploration/deployment phases; the paper reports effectiveness across smartphone applications.

1 Introduction

  • Mobile agents face GUI-recognition errors and difficulty adapting to unfamiliar apps and rapidly updated interfaces.The introduction notes that even GPT-4 struggles with lesser-known apps and unfamiliar visual elements.Source: §1 · p. 1
  • AppAgent combines parser information with OCR, detection, text, and visual features to support actions on a wider range of interface elements.Its structured storage system records parser details, text, and visual descriptions for retrieval in novel scenarios.Source: §1 · p. 2
  • The agent documents UI functionality through agent-driven or manual exploration, then uses RAG to retrieve and update that knowledge during deployment.The introduction describes exploration and deployment as the two operating phases.Source: §1 · p. 2
  • Empirical testing across smartphone applications demonstrates the agent’s effectiveness, adaptability, user-friendliness, and efficiency in real-world scenarios.Source: §1 · p. 2

2 Method

The method organizes mobile interaction into exploration and deployment: UI knowledge is documented, stored in a structured format, and retrieved or updated during task execution. It also describes visual fallback, human oversight for sensitive actions, and cross-app task handling.

2 Method

  • The agent analyzes each GUI against task requirements and carries observations, thoughts, actions, and summaries forward as execution memory.Source: §2 · p. 2

2.3 Exploration Phase

  • Agent-driven exploration compares screenshots before and after actions, reflects on relevance, and documents effective UI functions; manual exploration lets humans guide operations when the agent misjudges unfamiliar interfaces.Source: §2.3.1 · pp. 3–4, §2.3.2 · p. 4

2.4 Development Phase

  • During deployment, a self-query retriever uses resource IDs or OCR-derived information to find relevant documents, which inform GUI- and task-conditioned actions and are updated with action outcomes.Source: §2.4 · p. 4

2.5 Document Generation

  • The structured document stores UI identifiers, labels, text, visual features, coordinates, and interpreted functions, and can be dynamically updated as new elements appear.Source: §2.5 · p. 4

2.6.1 Visual Features Decision-Making

  • When a desired control lacks a numerical tag or tagged controls cannot complete the task, the agent switches to a visual layout using OCR and detection to annotate text and icons.Source: §2.6.1 · p. 5

2.6.2 Safety Check

  • For sensitive actions such as passwords or payments, the agent switches to manual mode and resumes deployment after the user signals “finish.”Source: §2.6.2 · p. 5

2.6.3 Cross-Apps Task

  • For cross-app tasks, the agent checks whether actions in one application are complete before navigating away, then proceeds with commands for another application.Source: §2.6.3 · p. 5

3 Experiments

The experiments evaluate the agent on DroidTask, AppAgent, and Mobile-Eval, alongside a qualitative user study and analysis of UI parsing. Results cover benchmark comparisons and the use of structured data and visual features for mobile interfaces.

3.1.1 DroidTask

  • On DroidTask, direct deployment outperformed GPT-4 and memory-augmented AutoDroid on a benchmark of 158 tasks across 13 apps.Completion Rate measures the probability of accurately completing every action in a task sequence.Source: §3.1.1 · pp. 5–6, Figure 2 · p. 6

3.1.2 AppAgent

  • On the ten-app AppAgent benchmark, agent-driven exploration was comparable to demo watching, while adding manually explored documents significantly improved performance.The primary metric is application-level success rate.Source: Table 1 · p. 6, §3.1.2 · p. 6

3.2 User study

  • The user study reports strong performance on a complex task involving cross-app activities, long-term multi-step execution, and multi-step memory storage.The study presents eight core steps of the process.Source: §3.2 · pp. 6–7

3.1.3 Mobile-Eval

  • 100% success across all instructions in 10 Mobile-Eval categories; average Process Score exceeded 90% across three instruction sets.The results are presented against Mobile-Agent scores and human performance.Source: §3.1.3 · p. 6, Table 2 · p. 7

3.3 Analysis of UI Interface Parsing

  • XML structured data provides widget interactivity details and suits most generic apps, while visual features help recognize text and icons in custom-developed apps.Structured data cannot parse custom Android icons.Source: §3.3 · p. 7
  • Because visual features alone cannot determine icon operability, they are used only when no XML-based icon can perform the required task.Visual-only recognition may lead to redundant attempts on non-interactive elements.Source: §3.3 · p. 7

4 Related works

The related-work section surveys LLM-based agents across general and multimodal domains, then describes systems developed for mobile-device interaction.

4.1 LLM-based agents

  • The survey covers LLM agents in general domains and multimodal areas including video understanding, embodied AI, and visual generation.It also describes multi-agent cooperative systems with distinct roles.Source: §4.1 · pp. 7–8
  • Figure 3 is identified as qualitative results for a cross-app task.Source: Figure 3 · p. 8

4.2 Agent for mobile devices

5 Conclusion

The paper presents a multimodal mobile-agent framework combining parser and visual features with a structured knowledge base and exploration/deployment phases. Experiments across applications are reported to improve GUI recognition and task execution; future work targets cross-application functionality and decision-making.

5 Conclusion

  • Experiments across applications demonstrate improved GUI recognition and task execution, with adaptation to diverse application environments.The conclusion describes these results as confirming the framework’s effectiveness.Source: §5 · pp. 8–9
  • The framework combines parsers with visual features for a flexible action space and uses a structured knowledge base to store diverse elements.Exploration and deployment phases are used to manage dynamic mobile interfaces.Source: §5 · p. 8
  • The authors characterize the advances as incremental in precision and adaptability within complex mobile environments.Source: §5 · p. 8
  • Future work will enhance cross-application functionality and refine decision-making to improve efficiency and user experience.Source: §5 · p. 9

6 Limitations

The paper identifies two limitations: numerical-tag recognition can confuse interface numbers with element tags, and hidden controls may be undetectable without prior knowledge.

6 Limitations

  • Numerical tags used to identify UI elements can cause confusion when the interface element itself contains numbers.Source: §6 · p. 9
  • Preliminary manual exploration and documentation can mitigate errors caused by number-containing interface elements.Source: §6 · p. 9
  • Without prior knowledge, the agent cannot detect hidden controls such as a video-acceleration button absent from the current UI.Source: §6 · p. 9

7 Ethics Statement

The ethics statement describes commitments to protecting user information, implementing safety checks, considering societal impacts, and monitoring the technology.

7 Ethics Statement

  • The authors state that they adhere to global privacy standards and implement security measures to protect user information.Source: §7 · p. 9
  • The authors state that they implement safety checks to support reliability in dynamic environments.Source: §7 · p. 9
  • The statement says the authors consider possible employment and environmental-sustainability effects.Source: §7 · p. 9
  • The authors commit to ongoing monitoring and refinement to address emerging challenges and incorporate user feedback.Source: §7 · p. 9

A Prompt Structure Description

The appendix explains how prompt annotations distinguish replaceable parameters, user-filled text, and explanatory annotations.

A Prompt Structure Description

B Explanation of DroidTask Results

The DroidTask comparison with AutoDroid is affected by differences in test devices and app versions, and some tasks could not be run directly. The authors report that performance would be higher under identical conditions.

B Explanation of DroidTask Results

  • Differences in testing devices and application versions precluded direct execution of some DroidTask tasks against AutoDroid.AutoDroid used a specific Android phone, while this agent used an emulator.Source: B Explanation of DroidTask Results · p. 11
  • For some unsupported tasks, the authors used alternatives that preserved the procedural flow, while tasks with no alternative were counted as errors.For example, sorting documents by initial letter replaced unavailable date sorting.Source: B Explanation of DroidTask Results · p. 11
  • The authors state that the agent’s performance would be higher under identical testing conditions.Source: B Explanation of DroidTask Results · p. 11

C Details of Action Space

The action space provides commands for selecting UI elements, entering text, performing gestures, navigating between screens, waiting, and ending an operation.

C Details of Action Space

D Case Study

The case studies show a safety-check example and qualitative results on multi-step tasks across applications, including setting an alarm and adding an item to a cart.

D Case Study

Loading 2408.11824v4…