Source-linked AI summary
GPT-4V in Wonderland: Large Multimodal Models for Zero-Shot Smartphone GUI Navigation
An Yan, Zhengyuan Yang, Wanrong Zhu, Kevin Lin, Linjie Li, Jianfeng Wang, Jianwei Yang, Yiwu Zhong, Julian McAuley, Jianfeng Gao, Zicheng Liu, Lijuan Wang
TL;DR
Smartphone GUI agents must interpret screens and execute natural-language instructions, but effective zero-shot navigation remains a research challenge. This paper presents MM-Navigator, a GPT-4V-based agent evaluated on iOS and Android navigation, achieving strong performance while still exhibiting errors in knowledge-dependent, localization, and complex scenarios.
Problem
Virtual assistants should carry out everyday tasks from natural-language instructions, motivating autonomous agents that can interact with computing devices.
Method
MM-Navigator is a GPT-4V-based smartphone GUI-navigation agent evaluated on a collected analytic iOS dataset and sampled Android benchmark episodes.
Results
MM-Navigator achieved 91% accuracy for intended action descriptions and 75% for localized action execution on single-step iOS instructions, while outperforming previous LLM-based approaches on Android navigation.
Takeaways & Limitations
The evaluations support GPT-4V’s effectiveness for zero-shot smartphone GUI navigation and provide an analytic benchmark for screen understanding and action execution.
Takeaways & Limitations
GPT-4V still makes errors when required knowledge is missing, locations are misselected, initial guesses fail in complex scenarios, or clickable areas are unmarked.
Abstract
from arXiv · showhide
We present MM-Navigator, a GPT-4V-based agent for the smartphone graphical user interface (GUI) navigation task. MM-Navigator can interact with a smartphone screen as human users, and determine subsequent actions to fulfill given instructions. Our findings demonstrate that large multimodal models (LMMs), specifically GPT-4V, excel in zero-shot GUI navigation through its advanced screen interpretation, action reasoning, and precise action localization capabilities. We first benchmark MM-Navigator on our collected iOS screen dataset. According to human assessments, the system exhibited a 91\% accuracy rate in generating reasonable action descriptions and a 75\% accuracy rate in executing the correct actions for single-step instructions on iOS. Additionally, we evaluate the model on a subset of an Android screen navigation dataset, where the model outperforms previous GUI navigators in a zero-shot fashion. Our benchmark and detailed analyses aim to lay a robust groundwork for future research into the GUI navigation task. The project page is at https://github.com/zzxslp/MM-Navigator.
1 Introduction
MM-Navigator uses GPT-4V to navigate smartphone GUIs by interpreting screens, reasoning about actions, and localizing executable interactions. Evaluations on collected iOS and public Android data show strong zero-shot performance, while highlighting reliability limitations.
- Motivation: Smartphone assistants must translate natural-language instructions into GUI actions, from simple tasks to multi-step activities.The motivation spans everyday assistance and complex operations such as finding a hotel.
- Research gap: Supervised GUI navigators can generalize poorly beyond their training screens and instructions, whereas LLM-based approaches generalize better but may rely on an error-prone intermediate text-conversion step.These limitations motivate direct multimodal navigation with large multimodal models.
- Approach: MM-Navigator is a GPT-4V agent that uses action histories and set-of-mark tags to generate precise executable smartphone actions.The system connects screen interpretation with localized interaction outputs.
- Benchmark: The benchmark separates GUI navigation into intended action description and localized action execution, evaluating both reasoning and action localization.The collected iOS dataset contains diverse screens and user instructions for these two challenges.
- Results: Evaluations on two datasets and detailed analyses indicate MM-Navigator is effective for zero-shot GUI navigation, while continued improvements are needed for reliability.The paper reports human and automatic evaluations on iOS and Android data.
2 Related Work
Related work frames autonomous GUI navigation as following instructions through interfaces, while multimodal agents extend language-model reasoning toward visual environments. Existing benchmarks assess either specific UI-element identification or broader task completion across GUI views.
- Autonomous GUI navigation: Autonomous GUI navigation requires following instructions through websites or applications to complete user-queried tasks.Benchmarks evaluate either specific UI-element identification or interaction across a sequence of GUI views.
- Autonomous GUI navigation: GUI navigation benchmarks use synthetic or real-world instructions to test element identification and overarching task completion.These represent different granularities of navigation evaluation.
- Multimodal agents: Large language models have motivated agent systems that combine reasoning logic with external tools for complex language tasks.This line of work has expanded from language agents toward multimodal agents.
- Multimodal agents: Multimodal-agent research applies language-model-based agents to settings that combine visual inputs with reasoning and interaction.The paper situates GUI navigation within this broader research direction.
3 MM-Navigator
MM-Navigator uses GPT-4V to map smartphone screenshots, instructions, and interaction history to actions. Set-of-Mark prompting supports localized execution, while multimodal self-summarization compresses prior interactions into a concise history.
- Task formulation: Each navigation episode begins with a natural-language instruction, and the agent selects actions from screenshots at successive time steps.An episode spans interactions from an initial state to a final state.
- Set-of-Mark prompting: GPT-4V accepts visual images and text but may understand the target region without reliably producing an executable screen action from coordinates alone.This motivates a structured interface between model outputs and screen interactions.
- Set-of-Mark prompting: Set-of-Mark prompting adds numeric tags to detected UI elements so GPT-4V can choose a marked element and produce a localized executable action.OCR and IconNet provide element detections with bounding boxes, text, or icon labels.
- History generation: The agent must use prior interactions together with the current screen, but feeding all historical screens or actions can be expensive and information-heavy.Screens may change rapidly, making a concise history representation desirable.
- History generation: Multimodal self-summarization converts historical actions and current-step information into a concise natural-language history for subsequent decisions.The summarized trace is generated autoregressively during episode rollout.
4 iOS Screen Navigation Experiment
The iOS experiments separate GUI navigation into understanding the intended action and executing it at a localized screen position. GPT-4V performs strongly on both single-step tasks and shows promise for multi-screen navigation, while retaining identifiable failure modes.
- Experimental setup: The dataset uses iOS screens with and without added marks to separately evaluate intended action description and localized action execution.The first task measures whether GPT-4V can describe the correct action; the second measures whether it can select a location that fulfills the instruction.
- Intended action description: 90.9% accuracy was achieved for generating correct intended action descriptions in human evaluation.Examples include identifying relevant settings, alerts, products, and interface controls from screenshots and instructions.
- Localized action execution: 74.5% accuracy was achieved for selecting locations that could lead to the desired outcome.GPT-4V selected marked controls across screens containing tabs, many clickable icons, complex imagery, and web links.
- Failure cases: GPT-4V still makes errors involving missing knowledge, incorrect locations, incorrect initial guesses, and unmarked clickable areas.The authors note that iterative exploration may correct many single-step failures at the episode level.
- Episode-level navigation: MM-Navigator extends from single screens to recursively processing screen-input episodes using the objective and previous action at each step.A qualitative shopping example shows multi-step reasoning toward completing an instruction.
5 Android Screen Navigation Experiment
The Android experiment evaluates GPT-4V zero-shot navigation against prior baselines and examines how visual inputs, text descriptions, history, tagging, prompts, and annotation quality affect performance. GPT-4V shows strong screen understanding, while errors arise from both benchmark annotation issues and genuine navigation mistakes.
- Experimental setup: The evaluation uses 300 AITW test episodes sampled from a large Android UI-control benchmark covering diverse multi-step tasks.AITW contains 715K episodes and 30K unique instructions.
- Experimental setup: The main metric is screen-wise partial action matching, requiring the predicted action type and gesture to match the user action under the benchmark’s click criteria.For clicks, correctness is determined by proximity to the gold gesture or overlap with its detected bounding box.
- Performance comparison: GPT-4V outperforms previous LLMs that receive ground-truth textual screen descriptions, while visual inputs preserve screen information more directly.Adding parsed screen descriptions and interaction history further improves GPT-4V performance.
- Performance comparison: The benchmark compares fine-tuned Llama 2, zero-shot and five-shot PaLM-2, five-shot ChatGPT, and GPT-4V zero-shot variants.The GPT-4V variants include image-only, +text, and +history settings.
- Ablation studies: The tagging ablation compares side, red, and center tags, with side tags potentially hurting performance when nearby icons are crowded.The study uses 50 episodes sampled from five categories.
- Error analysis: Some apparent false negatives reflect inaccurate Set-of-Mark parsing or incomplete ground-truth annotations rather than incorrect GPT-4V actions.Examples include over-segmented target regions and multiple valid ways to complete an instruction.
- Error analysis: True negative errors include clicking instead of scrolling, selecting a non-button, and interpreting an instruction too literally.These mistakes occur in zero-shot testing without demonstrative examples.
6 Discussion
The discussion identifies benchmark realism, error correction, and computational cost as important boundaries for smartphone GUI navigation. It proposes more dynamic environments, continual learning, broader device coverage, and distillation as future directions.
- Future benchmarks: Future device-control benchmarks should support dynamic exploration and returning to a previous state after a mistake.The discussion also proposes automatic success evaluation and datasets spanning devices such as personal computers and iPads.
- Error correction: Error correction remains an open issue when pretrained LMMs fail in novel settings because of data or algorithmic bias.Continual learning with ongoing interaction and feedback is proposed as one direction.
- Model distillation: Using GPT-4V for GUI navigation is costly, motivating model distillation toward smaller models with competitive navigation performance.The intended benefits are lower latency and higher efficiency.
7 Conclusion
MM-Navigator is a GPT-4V-based agent for GUI navigation, benchmarked across iOS and Android screen-navigation datasets. The evaluation highlights GPT-4V’s capabilities in understanding, reasoning, and planning over screen environments, while motivating simulator-based multi-step evaluation.
- MM-Navigator is a GPT-4V-based multimodal agent designed for the GUI navigation task.
- The system is benchmarked on both a collected iOS dataset and a public Android navigation dataset.
- The benchmarks reveal GPT-4V’s capabilities in understanding, reasoning, and planning over screen environments.
- A promising future direction is a simulator-based benchmark with multi-step and episode-level automatic evaluations.