PaperlayerView exact revision on arXiv

Source-linked AI summary

AppAgent v2: Advanced Agent for Flexible Mobile Interactions

Yanda Li, Chi Zhang, Wanqi Yang, Bin Fu, Pei Cheng, Xin Chen, Ling Chen, Yunchao Wei

arXiv:2408.11824v3cs.HCcs.AI

TL;DR

Mobile agents face difficulties recognizing changing interfaces and unfamiliar or non-standard UI elements. AppAgent v2 combines parser and visual features with exploration and a structured knowledge base used through RAG. The paper reports improved GUI recognition and task execution across varied applications, while noting limitations in number-tag interpretation and hidden-element detection.

  • Problem

    Mobile agents face GUI-recognition challenges in changing interfaces and unfamiliar apps, while parser dependence can limit handling of non-standard elements.

    Source: §1 · pp. 1–2

  • Method

    AppAgent v2 combines parser and visual features with exploration and deployment phases that use RAG to retrieve and update a structured UI knowledge base.

    Source: §1 · p. 2

  • Results

    Experiments across various applications are reported to show improved GUI recognition and task execution and effectiveness in diverse environments.

    Source: §1 · p. 2, §5 · p. 8

  • Takeaways & Limitations

    The framework brings parser and visual information together with an updateable knowledge base to handle diverse mobile interfaces and tasks.

    Source: §1 · p. 2, §5 · p. 8

    Numbered UI elements can be confused with numbers in their content, and hidden controls may be undetectable without prior knowledge.

    Source: §6 · p. 9

Abstract

from arXiv · show

With the advancement of Multimodal Large Language Models (MLLM), LLM-driven visual agents are increasingly impacting software interfaces, particularly those with graphical user interfaces. This work introduces a novel LLM-based multimodal agent framework for mobile devices. This framework, capable of navigating mobile devices, emulates human-like interactions. Our agent constructs a flexible action space that enhances adaptability across various applications including parser, text and vision descriptions. The agent operates through two main phases: exploration and deployment. During the exploration phase, functionalities of user interface elements are documented either through agent-driven or manual explorations into a customized structured knowledge base. In the deployment phase, RAG technology enables efficient retrieval and update from this knowledge base, thereby empowering the agent to perform tasks effectively and accurately. This includes performing complex, multi-step operations across various applications, thereby demonstrating the framework's adaptability and precision in handling customized task workflows. Our experimental results across various benchmarks demonstrate the framework's superior performance, confirming its effectiveness in real-world scenarios. Our code will be open source soon.

1 Introduction

Mobile agents must interpret changing graphical interfaces and unfamiliar elements, where visual recognition and parser dependence can limit performance. AppAgent v2 combines parser and visual information with a structured, updateable knowledge base, and reports effectiveness across smartphone applications.

1 Introduction

  • Changing mobile interfaces and unfamiliar visual elements challenge GUI recognition, while parser dependence can restrict interaction with non-standard components.The introduction notes that even sophisticated models can struggle with lesser-known apps and rapidly updated interfaces.Source: §1 · pp. 1–2
  • AppAgent v2 combines parser data with OCR- and detection-based visual features to support interaction with a wider variety of interface elements.Its structured knowledge base stores element attributes such as parser details, text, and visual descriptions.Source: §1 · p. 2
  • The framework uses exploration to document UI functionality and deployment with RAG to retrieve and update that knowledge during interactions.Exploration can be agent-driven or manual, and the documented information is organized in a structured knowledge base.Source: §1 · p. 2
  • Tests across three benchmarks and numerous applications, together with user studies, are reported as demonstrating effectiveness, superiority, and robustness.The introduction also describes adaptability, user-friendliness, and efficiency in real-world smartphone applications.Source: §1 · p. 2

2 Method

The method separates mobile interaction into exploration, which documents UI functionality, and deployment, which uses task context and stored information to guide actions. It adds flexible element handling, a structured updateable document, and safeguards and routines for visual fallback and cross-app tasks.

2.1 Agent Framework

  • The framework carries observations, thoughts, actions, and summaries across execution rounds, using each summary as memory for the next prompt.It is implemented in an Android 15 emulator and combines structured GUI parsing with OCR and detection from screenshots.Source: §2 · p. 2, §2.1 · p. 2, Figure 1 · p. 3

2.2 Agent Interactions

  • The agent supports taps, text entry, long presses, swipes, back, home, wait, and stop commands for Android interactions.A tap can target a numbered element or describe its visual features.Source: §2.2 · p. 3

2.3 Exploration Phase

  • Exploration documents UI-element functions through either agent-driven or manual operations, with screenshots used to compare interface changes.The exploration phase analyzes the GUI in relation to the current task.Source: §2.3 · p. 3, §2.3.1 · p. 3

2.3.1 Agent-Driven exploration

  • Agent-driven exploration reflects on outcomes, returns from irrelevant actions, and records effective task-aligned UI information for later retrieval.Irrelevant actions are added to a useless_list and fed back to the LLM.Source: §2.3.1 · pp. 3–4

2.3.2 Manual Exploration

  • Manual exploration lets GPT-4 observe human operations and compare before-and-after screenshots, helping address erroneous judgments from incomplete app understanding.OCR and detection models support UI analysis, while humans guide the action sequence and conclude exploration.Source: §2.3.2 · p. 4

2.4 Development Phase

  • During deployment, a self-query retriever finds relevant UI documents using resource IDs or OCR-derived information, and the agent combines them with screenshots and task requirements.After each action, prompts are updated with historical information and outcomes; the agent can also handle many common tasks without loading a document.Source: §2.4 · p. 4

2.5 Document Generation

  • The structured document stores diverse UI-element information and supports real-time updates as new elements appear during deployment.Stored attributes include Android ID, visible labels, text, visual features, coordinates, and interpreted functionality.Source: §2.5 · p. 4

2.6 Advanced Features

  • Advanced features switch to OCR and detection when numbered elements are ineffective, hand sensitive actions to the user without retaining their information, and track progress across apps.For cross-app tasks, the agent checks whether actions in one app are complete before proceeding to another.Source: §2.6.1 · p. 5, §2.6.2 · p. 5, §2.6.3 · p. 5

3 Experiments

The experiments evaluate the agent on three benchmarks, a cross-app user-study task, and UI parsing scenarios. Results report strong performance across benchmarks, while the parsing analysis identifies limits of visual-only recognition.

3 Experiments

  • Experiments used the Android Studio emulator and evaluated the agent on DroidTask, AppAgent, and Mobile-Eval; DroidTask has 158 high-level tasks from 13 apps and uses Completion Rate.Mobile-Eval lists Success and Process Score.Source: §3 · p. 5, §3.1 · p. 5, §3.1.1 · p. 5

3.1 Quantitative Results

  • Across the three benchmarks, the authors report superior performance in varied application scenarios.The quantitative evaluation includes complex-task performance, exploration-method comparisons, and comprehensive mobile-agent capabilities.Source: §3.1 · p. 5

3.1.1 DroidTask

  • The agent without exploration significantly outperformed GPT-4 and surpassed AutoDroid augmented with memory on DroidTask.Testing environments and app versions differed, and alternative methods were used for a small subset of tasks.Source: Figure 2 · p. 6, §3.1.1 · p. 6

3.1.2 AppAgent

  • Manual-exploration documents significantly improved performance over agent-driven exploration, whose results were comparable to AppAgent watching demos.The benchmark covers ten commonly used applications and evaluates task success rate.Source: §3.1.2 · p. 6, Table 1 · p. 6

3.1.3 Mobile-Eval

  • 100% success across all instructions in 10 task categories, with average Process Score above 90% across three instruction sets.The evaluation compares the agent with original Mobile-Agent benchmark scores and human performance.Source: §3.1.3 · p. 6, Table 2 · p. 7

3.2 User study

  • The user study presented eight core steps of a complex task involving cross-app activity, long-term multi-step execution, and multi-step memory storage.The authors report strong performance on the task.Source: §3.2 · pp. 6–7

3.3 Analysis of UI Interface Parsing

  • Structured XML parsing provides widget interactivity details and works well for most generic apps, but cannot parse custom Android icons.Visual features are used to recognize text and icons in those cases, but visual features alone cannot determine icon operability without interaction.Source: §3.3 · p. 7

4 Related works

The related-work section reviews LLM-based and multimodal agents, then surveys mobile agents that use GUI parsing, visual recognition, dynamic app analysis, or structured app memory.

4.1 LLM-based agents

  • LLM-based agents span decision-making and applications in multimodal understanding, embodied AI, visual generation, and multi-agent cooperation.The section names systems including MetaGPT, HuggingGPT, AssistGPT, and SeeClick.Source: §4.1 · pp. 7–8

4.2 Agent for mobile devices

  • Mobile-agent approaches include GUI-state prompting, XML component identification, OCR and icon detection, dynamic app analysis, and modular task memory.Examples include DroidBot-GPT, AppAgent, MobileAgent, AutoDroid, and MobileGPT.Source: §4.2 · p. 8

5 Conclusion

The framework combines parser and visual features with a structured knowledge base in exploration and deployment phases. Experiments across applications show improved GUI recognition and task execution, while cross-application functionality remains future work.

5 Conclusion

  • Experiments across various applications demonstrate improved GUI recognition and task execution, with adaptation to diverse application environments.Source: §5 · p. 8
  • The agent combines parsers and visual features in a flexible action space and uses a structured knowledge base for diverse interface elements.Source: §5 · p. 8
  • Future work targets cross-application functionality and refinement of decision-making to improve efficiency and user experience.Source: §5 · p. 9

6 Limitations

The agent has limitations in recognizing numerical UI tags and detecting hidden interface elements when it lacks prior knowledge.

6 Limitations

  • Recognizing numerical UI tags can confuse the agent when an interface element itself contains numbers.Source: §6 · p. 9
  • Preliminary manual exploration and documentation can mitigate errors by clarifying the context of numerical tags.Source: §6 · p. 9
  • Without prior knowledge, the agent cannot detect hidden controls such as a video acceleration button absent from the current UI.Source: §6 · p. 9

7 Ethics Statement

The ethics statement describes commitments to protect user information, support reliability and safety, consider broader societal impacts, and monitor the technology over time.

7 Ethics Statement

  • The authors state that they adhere to global privacy standards and use security measures to protect user information.Source: §7 · p. 9
  • They describe safety checks for reliability in dynamic environments and ongoing monitoring to refine the technology and integrate user feedback.Source: §7 · p. 9
  • The statement says the authors consider possible effects on employment and environmental sustainability.Source: §7 · p. 9

A Prompt Structure Description

The appendix describes prompt notation, distinguishing replaceable parameters, user-filled text, and annotations by formatting and color.

A Prompt Structure Description

B Explanation of DroidTask Results

DroidTask comparisons are affected by differences in devices and app versions, as well as tasks unsupported by the authors’ application. Alternative task procedures were used where possible, while unsupported tasks without alternatives counted as errors.

B Explanation of DroidTask Results

  • Device and application-version differences between AutoDroid’s real-device tests and the authors’ emulator tests prevented direct execution of some tasks.The authors note that these mismatches precluded a direct comparison for those tasks.Source: B Explanation of DroidTask Results · p. 11
  • For some tasks that could not be completed, the authors substituted alternatives that preserved the procedural flow but changed the final selection.Their example replaces sorting document names by date with sorting by initial letter.Source: B Explanation of DroidTask Results · p. 11
  • Tasks unsupported by the application and lacking an alternative were counted as errors, and the authors state performance would be higher under identical conditions.The stated higher performance is conditional on identical testing conditions.Source: B Explanation of DroidTask Results · p. 11

C Details of Action Space

The action space provides commands for interacting with interface elements, entering text, navigating between screens, waiting, and ending a task.

C Details of Action Space

  • TapButton selects an interface element by its numeric label or text, while Text enters a string into an input area.The examples include tapping element 5 or the element labeled “hat,” and typing “Hello, world!”Source: C Details of Action Space · p. 11
  • LongPress applies a prolonged press to a specified element, and Swipe moves across an element in a chosen direction and distance.The documented swipe parameters include direction and distance.Source: C Details of Action Space · pp. 11–12
  • Back returns to the previous interface state, whereas Home returns the agent to the main screen.The descriptions identify Back for navigation and Home for resetting the environment or starting new tasks.Source: C Details of Action Space · p. 12
  • Wait pauses operation for two seconds, and Stop ends the current operation to finalize task execution.Source: C Details of Action Space · p. 12

D Case Study

The case studies present qualitative results across varied applications and tasks, including safety checks during sensitive operations and multi-step task handling.

D Case Study

Related papers

Loading 2408.11824v3…