Source-linked AI summary
AppAgent: Multimodal Agents as Smartphone Users
Chi Zhang, Zhao Yang, Jiaxuan Liu, Yanda Li, Yucheng Han, Xin Chen, Zebiao Huang, Bin Fu, Gang Yu
TL;DR
Text-only agents are limited in perceiving and interacting with smartphone environments, while diverse app interfaces make broad adaptation difficult. The paper introduces a vision-enabled multimodal agent that operates apps through GUI actions, learns through exploration or demonstrations, and consults documented knowledge during execution. Experiments cover 50 tasks across 10 applications, supporting the framework’s ability to handle diverse high-level smartphone tasks.
Problem
Text-only agents have limited environmental perception, while diverse app interfaces make collecting demonstrations difficult and generalization to unseen apps uncertain.
Method
A multimodal agent operates smartphone apps through simplified GUI actions and learns app functionality through autonomous exploration or human demonstrations recorded in a reference document.
Results
50 tasks across 10 apps produced quantitative and user-study evidence supporting the framework’s adaptability, user-friendliness, and efficient learning and operating capabilities.
Takeaways & Limitations
The framework demonstrates potential as a versatile tool for operating smartphone applications and handling diverse high-level tasks.
Takeaways & Limitations
The simplified action space does not support advanced controls such as multi-touch and irregular gestures, restricting applicability in some challenging scenarios.
Abstract
from arXiv · showhide
Recent advancements in large language models (LLMs) have led to the creation of intelligent agents capable of performing complex tasks. This paper introduces a novel LLM-based multimodal agent framework designed to operate smartphone applications. Our framework enables the agent to operate smartphone applications through a simplified action space, mimicking human-like interactions such as tapping and swiping. This novel approach bypasses the need for system back-end access, thereby broadening its applicability across diverse apps. Central to our agent's functionality is its innovative learning method. The agent learns to navigate and use new apps either through autonomous exploration or by observing human demonstrations. This process generates a knowledge base that the agent refers to for executing complex tasks across different applications. To demonstrate the practicality of our agent, we conducted extensive testing over 50 tasks in 10 different applications, including social media, email, maps, shopping, and sophisticated image editing tools. The results affirm our agent's proficiency in handling a diverse array of high-level tasks.
1 Introduction
The paper develops a multimodal agent for operating smartphone apps through human-like GUI actions, addressing the limits of text-only agents and app-specific training. It learns app functionality through exploration or demonstrations, then uses documented knowledge to perform tasks across diverse applications.
- Vision-equipped LLM agents address the restricted perception and interaction of text-only agents by interpreting visual context and cues.
- The framework operates smartphone apps through tapping and swiping on their GUIs instead of system back-end access.This design is presented as broadly applicable across applications and adaptable to interface changes.
- Diverse app GUIs, icon meanings, and operational logic make collecting demonstrations difficult and leave generalization to unseen apps uncertain.
- The agent explores apps autonomously or through a few human demonstrations, documenting interactions for later task execution without per-app LLM parameter adaptation or extensive training data.
- 50 tasks across 10 apps evaluated the framework, covering social media, messaging, email, maps, shopping, and complex image editing.The reported results and user studies emphasize adaptability, user-friendliness, and efficient learning and operation.
- The paper contributes an open-source smartphone-operation framework, an exploration strategy for novel apps, and experiments validating its advantages.
2 Related Work
Related work traces the shift from language-only LLMs to multimodal agents that process visual data and perform complex interactive tasks. It identifies unfamiliar smartphone interfaces as a remaining challenge addressed by this work.
- 2.1 Large language models: GPT-4V extends LLM capabilities to visual understanding, tool use, API calls, coding, reasoning, and complex instruction following.
- 2.1 Large language models: Open-source vision-language models demonstrate capabilities similar to GPT-4V, but a performance gap remains between them.
- 2.2 LLMs as agents: LLM agents combine language and reasoning abilities with environmental interaction, decision-making, and learning from interactions.
- 2.2 LLMs as agents: Multimodal LLM agents broaden agent applications by processing text, images, audio, and video for complex tasks in physical and tool-mediated environments.
- 2.2 LLMs as agents: The framework is organized into exploration and deployment phases: app interaction produces a reference document, which later supports app operation.
3 Method
The method uses a simplified smartphone action space and multimodal observations to learn app functionality, either autonomously or from human demonstrations. During deployment, the agent consults dynamically updated interaction knowledge while executing tasks step by step.
- The framework comprises an experimental environment, an action space, an exploration phase, and a deployment phase for executing high-level tasks.
- Experimental Environment: The agent receives screenshots and XML descriptions of interactive elements, whose unique identifiers support accurate interaction.
- Action Space: Its action space mirrors smartphone use with taps, long presses, directional swipes, direct text input, Back, and Exit functions.
- Exploration Phase: Autonomous exploration uses trial and error, comparing screenshots before and after actions to document UI-element functions and action effects.
- Exploration Phase: Human demonstrations provide efficient examples for complex functionality by recording the elements and actions used.This narrows exploration and avoids irrelevant app pages compared with autonomous interaction.
- Deployment Phase: During deployment, each step combines the current screenshot, a dynamically generated document, observations, task reasoning, and function calls.
4 Experiments
The evaluation combines quantitative and qualitative experiments across diverse smartphone applications to assess AppAgent’s performance and design choices. Results indicate benefits from the simplified action space, exploration-based documents, and multimodal operation, including image editing.
- Experimental Setup: The benchmark covers 10 popular applications spanning maps, social media, messaging, video, music, reviews, email, shopping, time management, and image editing.The experiments assess adaptability across varied functions and interfaces.
- Qualitative Results: Qualitative examples across Google Maps, Gmail, and Lightroom illustrate AppAgent’s ability to perceive, reason, and execute tasks.The figure presents one qualitative task in each of the three applications.
- Experimental Setup: The evaluation reports Successful Rate, Reward, and Average Steps to measure task completion, progress toward objectives, and efficiency.Tasks unfinished within 10 steps count as failures for Successful Rate.
- Design and Analysis: 45 tasks across 9 apps are used for the main quantitative comparison, excluding Lightroom because task completion there is inherently ambiguous.The comparison evaluates raw versus simplified actions and alternative document-generation methods.
- Design and Analysis: The simplified action space improves GPT-4 baseline performance by avoiding the need to produce accurate x-y coordinates.Autonomous-exploration and human-demonstration documents outperform the GPT-4 baseline and approach human-written documents.
- Case Study: In Lightroom, document-equipped agents consistently outperform GPT-4 in user-ranked image editing, while demonstration-generated documents perform comparably to manually crafted ones.With documents, the agent tends to use more varied tools than the GPT-4 baseline.
5 Conclusion
The paper concludes that AppAgent uses multimodal perception and exploration-based learning to operate smartphone applications human-like across diverse high-level tasks. Its simplified action space does not support advanced multi-touch or irregular gestures, limiting some challenging scenarios.
- Conclusion: AppAgent leverages LLM vision capabilities to operate smartphone applications through human-like interactions without system back-end access.The framework is presented as adaptable and flexible across unfamiliar interfaces.
- Conclusion: Exploration-based learning helps the agent adapt to new applications with unfamiliar user interfaces and handle diverse high-level tasks.The conclusion emphasizes adaptability and learning efficiency across varied apps.
- Limitation: The simplified action space excludes advanced controls such as multi-touch and irregular gestures, restricting applicability in some challenging scenarios.The paper identifies support for these controls as future research.