Source-linked AI summary
DroidBot-GPT: GPT-powered UI Automation for Android
Hao Wen, Hongming Wang, Jiaxuan Liu, Yuanchun Li
TL;DR
Automating diverse applications with little development effort remains challenging. DroidBot-GPT turns Android GUI state and available actions into prompts for an LLM; it completed 13 of 33 evaluated tasks.
Problem
Supporting automation across diverse applications with little or no development effort remains a challenging problem.
Source: §1 · p. 1Method
DroidBot-GPT translates Android GUI state and available actions into natural-language prompts, asks an LLM to choose actions, and executes them on the smartphone.
Source: §1 · p. 1, §2.1 · p. 3Results
It completed straightforward tasks and some complex multi-jump tasks; overall, it completed 13 of 33 tasks.
Source: §1 · p. 1, §3.1 · p. 4Takeaways & Limitations
The authors view natural-language grounding as an opportunity for using LLMs in decision-making and robotic process automation.
Source: §1 · p. 2Failures were associated with unnamed GUI elements, obscure GUI relationships, and uncertainty about actions.
Source: §3.2 · pp. 5–6
Abstract
from arXiv · showhide
This paper introduces DroidBot-GPT, a tool that utilizes GPT-like large language models (LLMs) to automate the interactions with Android mobile applications. Given a natural language description of a desired task, DroidBot-GPT can automatically generate and execute actions that navigate the app to complete the task. It works by translating the app GUI state information and the available actions on the smartphone screen to natural language prompts and asking the LLM to make a choice of actions. Since the LLM is typically trained on a large amount of data including the how-to manuals of diverse software applications, it has the ability to make reasonable choices of actions based on the provided information. We evaluate DroidBot-GPT with a self-created dataset that contains 33 tasks collected from 17 Android applications spanning 10 categories. It can successfully complete 39.39% of the tasks, and the average partial completion progress is about 66.76%. Given the fact that our method is fully unsupervised (no modification required from both the app and the LLM), we believe there is great potential to enhance automation performance with better app development paradigms and/or custom model training.
1 Introduction
DroidBot-GPT applies an LLM to Android app automation by translating GUI information into prompts and executing the chosen actions. The introduction frames this as a way to address the challenge of automating diverse applications with little development effort.
1 Introduction
- Automating diverse applications with little or no development effort remains a challenging problem for natural-language-grounded UI automation.Existing assistants commonly extract intent and parameters before connecting an API to a specific command, while recent approaches also explore end-to-end methods.Source: §1 · p. 1
- DroidBot-GPT extracts Android GUI state, prompts an LLM to generate the next action, and executes that action on the smartphone.The system uses DroidBot to obtain structured GUI information and translates it into natural language.Source: §1 · p. 1
- The evaluation uses 33 manually designed tasks across 17 Android applications and includes straightforward tasks and some complex tasks requiring multiple GUI jumps and prior knowledge.One complex example asks for the first holiday of Bavaria in 2023.Source: §1 · p. 1
- The paper presents DroidBot-GPT as an Android GUI automation tool using an LLM, without modifying the app or model.The authors also introduce natural-language descriptions of tasks, states, and actions to ground the LLM’s action choices.Source: §1 · p. 2
2 Method
DroidBot-GPT fetches an Android app’s state, describes it in natural language, and prompts ChatGPT with the task and action history to select operations. Its method converts GUI elements and available actions into text that the model can process.
2 Method
- DroidBot-GPT combines the app state, task, and action history in a prompt, asks ChatGPT for an action, and sends the operation to the phone.It first fetches the app state and describes it in natural language.Source: §2 · p. 3
2.1 GUI description
- Directly prompting with a tree-form GUI description can produce thousands of words, so the system converts the interface into familiar natural-language sentences.It describes each user-visible element through its properties and combines the resulting prompts into a UI-state sentence.Source: §2.1 · p. 3
2.2 Action space
- The action space includes choices such as clicking and scrolling, which the model selects by number; editing text uses a separate prompt to obtain the text to enter.If a response is a sentence rather than a choice, the system extracts the first choice it contains.Source: §2.2 · p. 3
2.3 Prompt synthesis
- The synthesized prompt includes the task, GUI elements and their action choices, action history, and whether the output should be a choice or text to enter.The sleep-timer example asks the model to choose the “Sleep Timer” view and then “5m.”Source: §2.3 · p. 4, Figure 4 · p. 4
3 Evaluation
DroidBot-GPT was evaluated on 33 tasks across 17 Android apps, completing 13 tasks and achieving 66.76% average completion progress. Results vary by task category, and failures are linked to unnamed controls, obscure GUI relationships, and uncertain actions.
3 Evaluation
- The evaluation used 1–3 tasks per F-Droid app across 17 apps, with each task requiring 2–13 GUI steps.Table 1 gives an example of tasks designed for World Weather.Source: §3 · p. 4, Table 1 · p. 5
- A 20-word cap limits each GUI element’s generated text because longer prompts can exclude crucial information or later elements.The cap was set based on the authors’ experiments.Source: §3 · p. 4
3.1 Results
- 13 of 33 tasks were fully completed, with 66.76% average completion progress across all tasks.Straightforward tasks with 2–3 steps were easier to complete; the authors attribute this to the model not being fine-tuned to understand interface relations.Source: §3.1 · p. 4
- “Record” and “Life” tasks had relatively low average completion progress, often involving typing and missed input locations caused by ambiguous GUI relationships.Table 3 reports completion rate and total task count by category.Source: §3.1 · p. 4, Table 3 · p. 5
- Compared with systems built for narrower app or action ranges [10, 11, 1], DroidBot-GPT needs no additional GUI training data and generalizes across more categories.The authors also report that it completes some complex tasks with minimal instructions.Source: §3.1 · p. 5
3.2 Limitations
- Failures stemmed from unnamed GUI elements, obscure relationships between controls and screens, and probabilistic action order that can cause redundancy or failure.Unnamed controls can trigger early termination or incorrect clicks; obscure relationships can lead to redundant or incorrect operations.Source: §3.2 · pp. 5–6
- Figure 5 illustrates a missing unnamed checkmark needed to save operations and an unclear relationship between the weather display and settings controls.The authors propose fine-tuning the LLM and using machine learning to recognize unnamed elements as future remedies.Source: Figure 5 · p. 6, §3.2 · p. 6
4 Related work
Related work spans GUI test generation and digital task automation; DroidBot-GPT is positioned as an LLM-powered approach to interacting with application UIs, with claimed greater flexibility and scalability.
4 Related work
- DroidBot-GPT uses DroidBot to generate Android application input from LLM instructions, within a field spanning random, model-based, search-based, record/replay, and learning-based [19] testing.GUI test generation aims to create scripts that verify interface functionality and reduce manual testing effort.Source: §4 · p. 6
- Digital task automation spans web and mobile applications; WebGPT [21] fine-tunes GPT-3 for long-form question answering in a text-based browsing environment.Other web agents use natural-language task descriptions with NLP and reinforcement techniques [20].Source: §4 · p. 6
- Unlike prior mobile-UI work, including Li et al. [1], MOTIF [22], Lexi, and Screen Correspondence, DroidBot-GPT uses an LLM-powered UI interaction method claimed to offer greater flexibility and scalability.The cited works address action generation and datasets, task automation and feasibility, UI information extraction, and UI reasoning, respectively.Source: §4 · p. 7
5 Conclusion
The conclusion presents DroidBot-GPT as an LLM-guided Android app automator that translates GUI information into prompts and queries, and evaluates it across tasks of varying complexity and categories.
5 Conclusion
- DroidBot-GPT translates a GUI and its corresponding action space into prompts and queries so a language model can perform user-specified tasks.The paper describes it as the first Android app automator guided by a large language model.Source: §5 · p. 7
- The system is evaluated on a dataset of tasks spanning varying complexity and categories.The conclusion also says the paper identifies system limitations and proposes potential solutions.Source: §5 · p. 7
- The paper's stated goal is to enable LLM assistance with individual decision-making and task completion.The conclusion frames this as an ultimate goal rather than a demonstrated evaluation result.Source: §5 · p. 7