Source-linked AI summary
Mobile-Agent: Autonomous Multi-Modal Mobile Device Agent with Visual Perception
Junyang Wang, Haiyang Xu, Jiabo Ye, Ming Yan, Weizhou Shen, Ji Zhang, Fei Huang, Jitao Sang
TL;DR
Mobile-agent systems need visual perception and semantic understanding, while existing localization approaches may depend on inaccessible interface files. Mobile-Agent uses screenshot-based visual tools, autonomous planning, stepwise navigation, and self-reflection, and is evaluated with Mobile-Eval. It achieves completion rates of 91%, 82%, and 82% across three instruction types and completes challenging Mobile-Eval instructions across varied apps.
Problem
Mobile agents require visual perception and semantic understanding, but GPT-4V and file-dependent localization methods have difficulty reliably locating operations when underlying files are inaccessible.
Method
Mobile-Agent combines screenshot-based text and icon localization with GPT-4V planning, stepwise operation execution, and self-reflection, and evaluates agents with Mobile-Eval.
Results
91%, 82%, and 82% completion rates were achieved across three instructions, while case studies showed successful completion across varied apps and challenging instructions.
Takeaways & Limitations
Mobile-Agent provides a vision-centric mobile-agent framework that operates without system code and can complete challenging instructions across diverse applications.
Abstract
from arXiv · showhide
Mobile device agent based on Multimodal Large Language Models (MLLM) is becoming a popular application. In this paper, we introduce Mobile-Agent, an autonomous multi-modal mobile device agent. Mobile-Agent first leverages visual perception tools to accurately identify and locate both the visual and textual elements within the app's front-end interface. Based on the perceived vision context, it then autonomously plans and decomposes the complex operation task, and navigates the mobile Apps through operations step by step. Different from previous solutions that rely on XML files of Apps or mobile system metadata, Mobile-Agent allows for greater adaptability across diverse mobile operating environments in a vision-centric way, thereby eliminating the necessity for system-specific customizations. To assess the performance of Mobile-Agent, we introduced Mobile-Eval, a benchmark for evaluating mobile device operations. Based on Mobile-Eval, we conducted a comprehensive evaluation of Mobile-Agent. The experimental results indicate that Mobile-Agent achieved remarkable accuracy and completion rates. Even with challenging instructions, such as multi-app operations, Mobile-Agent can still complete the requirements. Code and model will be open-sourced at https://github.com/X-PLUG/MobileAgent.
1 Introduction
Mobile-Agent addresses mobile-agent localization challenges by combining visual perception with autonomous planning and reflection, without relying on underlying system files. The paper also introduces Mobile-Eval to assess mobile device agents across apps and instruction difficulties.
- GPT-4V can generate effective operations but struggles to accurately locate their positions on mobile screens.
- Existing localization methods depend on Android XML or web HTML files, which may be inaccessible because permissions are unavailable.
- Mobile-Agent uses visual perception tools to locate operations from screenshots, then plans tasks and reflects on invalid or incomplete operations.
- Mobile-Agent operates solely from device screenshots without system code, providing a purely vision-based solution.
- Mobile-Eval contains 10 commonly used apps and instructions spanning three difficulty levels for evaluating mobile device agents.
- The paper conducts a comprehensive Mobile-Eval analysis and presents selected cases to examine Mobile-Agent’s capabilities.
2 Mobile-Agent
Mobile-Agent combines GPT-4V with text and icon localization tools to translate visual instructions into mobile operations. It iteratively plans actions from screenshots and history, while self-reflection helps detect and correct execution errors.
- Visual Perception: The framework combines GPT-4V with text detection for text localization and icon detection for icon localization.
- Visual Perception: OCR localizes requested text, clicking its center when one match exists and prompting reselection when matches are absent or overly numerous.
- Visual Perception: Icon localization uses Grounding DINO to detect icons and CLIP similarity to select the region best matching the requested icon description.
- Instruction Execution: Mobile-Agent defines eight operations, including opening apps, clicking text or icons, typing, scrolling, navigating back, exiting, and stopping.
- Instruction Execution: At each iteration, the agent receives the instruction, operation history, and current screenshot before producing the next action or ending the process.
- Instruction Execution: Self-reflection responds to unchanged screenshots or wrong pages, helping the agent recover from incorrect or invalid operations.
- Instruction Execution: The prompt format separates Observation, Thought, and Action, requiring the agent to describe context, reason about the next step, and select an operation with parameters.
3 Experiments
The experiments evaluate Mobile-Agent on Mobile-Eval using success, process accuracy, relative efficiency, and completion rate, then examine planning, reflection, multi-app, multilingual, and gameplay cases. Mobile-Agent achieves high completion and step accuracy while correcting invalid operations and handling varied app scenarios.
- Experimental Setup: Mobile-Eval contains 10 commonly used Apps and instructions requiring both single-App and simultaneous two-App usage.The benchmark uses three instruction difficulty levels for each App.
- Metrics: Success (Su), Process Score (PS), Relative Efficiency (RE), and Completion Rate (CR) measure completion, step accuracy, human-relative steps, and human-step completion.RE compares Mobile-Agent’s steps with human steps, while CR equals 1 when an instruction is completed.
- Quantitative Results: 91%, 82%, and 82% completion rates were achieved across the three instruction types, while PS was around 80% and RE indicated 80% human-optimal capability.The reported results collectively indicate effectiveness as a mobile device assistant.
- Case Study: Mobile-Agent corrected invalid or incorrect operations after detecting unchanged screenshots or wrong pages, and most such instructions were ultimately completed.The cases illustrate self-reflection and error correction during execution.
- Case Study: Mobile-Agent transferred information between Apps and generated reprocessed content in multi-App scenarios.The qualitative cases also covered multilingual interaction and gameplay after describing game rules.
- Case Study: Across varied interfaces, operations, and challenging instructions, Mobile-Agent successfully completed the demonstrated Mobile-Eval tasks.The examples include wholesale-cap searches, video commenting, App downloads, navigation, and music search and playback.
4 Related Work
Related work situates mobile-device agents within the broader rise of tool-using LLM agents and MLLMs. Existing mobile solutions acquire operable regions through Android XML files, exploration, demonstrations, or documents.
- LLM-based Agents: LLM-based agents use diverse tools to support task planning and reasoning across increasingly varied applications.The broader literature includes image and video editing, generation, visual question answering, and prediction tasks.
- Agents for Mobile Devices: AppAgent operates mobile Apps with GPT-4V by labeling manipulable UI regions through Android XML files.It also acquires operational capabilities through self-exploration, user video demonstrations, and user documents.
5 Conclusion
Mobile-Agent combines visual perception with autonomous planning and stepwise mobile-App navigation. The paper demonstrates this approach across varied Apps and challenging instructions, including multi-App operations.
- Conclusion: Mobile-Agent uses a unified visual perception framework to identify and locate visual and textual interface elements across mobile applications.The framework supports operation without relying on XML mobile system metadata.
- Conclusion: The demonstrated cases include searching videos, downloading Apps, navigation, and searching or playing music across different mobile applications.These examples are presented as cases from the broader evaluation.