PaperlayerView exact revision on arXiv

Source-linked AI summary

AutoDroid: LLM-powered Task Automation in Android

Hao Wen, Yuanchun Li, Guohong Liu, Shanhui Zhao, Tao Yu, Toby Jia-Jun Li, Shiqi Jiang, Yunhao Liu, Yaqin Zhang, Yunxin Liu

arXiv:2308.15272v4cs.AIcs.SE

TL;DR

Existing mobile task automation requires substantial manual effort and struggles with scalable language understanding. AutoDroid combines LLMs with app-specific knowledge obtained through automated exploration, achieving strong Android task-automation performance while reducing query cost.

  • Problem

    Existing mobile task automation approaches suffer from poor scalability because they require substantial developer or user effort, while LLMs lack app-specific knowledge.

  • Method

    AutoDroid uses automated app exploration to obtain app-specific knowledge and combines it with LLM-based GUI understanding and task execution.

  • Results

    71.3% task completion and 90.9% action accuracy were achieved with GPT-4, with task completion improving 36.4% to 39.7% over off-the-shelf LLM baselines.

  • Takeaways & Limitations

    The results support combining LLM commonsense knowledge with domain-specific mobile-app knowledge for task automation without manual efforts.

  • Takeaways & Limitations

    Increased latency limits AutoDroid’s practical use, motivating collaborative smaller-model execution and instruction caching.

Abstract

from arXiv · show

Mobile task automation is an attractive technique that aims to enable voice-based hands-free user interaction with smartphones. However, existing approaches suffer from poor scalability due to the limited language understanding ability and the non-trivial manual efforts required from developers or end-users. The recent advance of large language models (LLMs) in language understanding and reasoning inspires us to rethink the problem from a model-centric perspective, where task preparation, comprehension, and execution are handled by a unified language model. In this work, we introduce AutoDroid, a mobile task automation system capable of handling arbitrary tasks on any Android application without manual efforts. The key insight is to combine the commonsense knowledge of LLMs and domain-specific knowledge of apps through automated dynamic analysis. The main components include a functionality-aware UI representation method that bridges the UI with the LLM, exploration-based memory injection techniques that augment the app-specific domain knowledge of LLM, and a multi-granularity query optimization module that reduces the cost of model inference. We integrate AutoDroid with off-the-shelf LLMs including online GPT-4/GPT-3.5 and on-device Vicuna, and evaluate its performance on a new benchmark for memory-augmented Android task automation with 158 common tasks. The results demonstrated that AutoDroid is able to precisely generate actions with an accuracy of 90.9%, and complete tasks with a success rate of 71.3%, outperforming the GPT-4-powered baselines by 36.4% and 39.7%. The demo, benchmark suites, and source code of AutoDroid will be released at url{https://autodroid-sys.github.io/}.

1 INTRODUCTION

AutoDroid frames Android task automation as a scalable, model-centric problem: LLMs provide language understanding while automated app analysis supplies domain knowledge. It introduces a benchmark and evaluates an end-to-end system across multiple LLMs.

  • Motivation: Existing mobile automation approaches scale poorly because they require substantial developer effort or human demonstrations and instructions.Developer-based systems require implementation and registration work, while learning- and demonstration-based methods require ad hoc or large-scale human input.
  • Approach: AutoDroid combines LLM capabilities with app-specific knowledge to automate arbitrary unseen tasks without manual efforts.Its design addresses GUI representation, knowledge integration, and LLM query cost.
  • Approach: AutoDroid represents GUIs as HTML-style text, explores target apps to extract UI transition graphs, and injects synthesized task knowledge into prompts.The system analyzes UI states and transitions with LLMs before using the resulting knowledge during task execution.
  • Evaluation: The benchmark contains 158 manually labeled tasks from 13 open-source mobile apps and includes executable environments for reproducible evaluation.Tasks cover PixelHelp how-to questions and common app functionalities.
  • Results: 71.3% task completion with GPT-4 and 90.9% action accuracy were achieved, while completion improved 36.4% to 39.7% over baselines.Average LLM-querying cost was reduced by 51.7%.

2 BACKGROUND AND MOTIVATION

Mobile task automation seeks to convert natural-language requests into smartphone actions, while LLMs offer reasoning and language abilities but face GUI, app-knowledge, and cost constraints. AutoDroid addresses these challenges by combining LLMs with dynamic app-specific knowledge.

  • Mobile Task Automation: Mobile task automation takes an arbitrary natural-language task and mobile app as input, producing a sequence of executable UI actions.Tasks are often multi-step functionality requests without explicit instructions.
  • Mobile Task Automation: Multi-step automation requires planning and knowledge of which UIs are essential, beyond summarizing or answering questions about individual mobile UIs.AutoDroid targets unsupervised automation of arbitrary tasks on black-box apps without human effort, assuming apps are available for automated analysis.
  • LLMs and Automation: LLMs provide instruction following, step-by-step reasoning, and zero-shot generalization, making them promising for scalable task automation.Their tool-use potential complements mobile automation, where apps often lack documented interfaces.
  • Challenges: App analysis helps connect semantically indirect actions such as opening “more options” and “settings” to deleting all calendar events.The example illustrates why app-specific transition knowledge matters for task planning.
  • Challenges: Android automation challenges include lengthy GUI representations, missing application-specific knowledge, and costly LLM inference.A UI state averages about 40k tokens, while task completion may use over 2000 tokens.

3 OUR APPROACH: AUTODROID

AutoDroid separates app exploration and task execution: it builds app-specific memory offline, then uses that memory to guide online LLM-driven actions. Privacy filtering and action verification mediate execution.

  • Offline Stage: AutoDroid traverses the explored UI elements and summarizes the tasks they can accomplish, creating app-specific knowledge for later prompts.This knowledge is synthesized as simulated tasks from the explored interface.
  • Online Stage: During online operation, the system continuously queries a memory-augmented LLM for next-action guidance and executes the suggested actions.Task completion follows the actions selected by the LLM.
  • Offline Stage: During offline preparation, AutoDroid randomly explores an app and records UI relations in a UI Transition Graph memory.The graph captures the results of clicking available screen buttons.
  • Execution: A privacy filter replaces sensitive information before prompting, while the task executor parses and security-checks the LLM’s proposed action.Potentially risky actions, such as deleting all events, trigger user confirmation.

3.1 Task-oriented UI Prompting

AutoDroid converts Android GUIs into structured HTML-style prompts so LLMs can interpret states and select valid UI actions. It enriches these prompts with task guidance, current UI information, and output constraints.

  • 3.1 Task-oriented UI Prompting: UI prompting represents underlying interface information as text injected into an LLM prompt.Its goal is to expose textual and structural UI content while restricting predictions to valid interactions.
  • 3.1 Task-oriented UI Prompting: Figure 3 presents AutoDroid’s workflow for converting GUI information into prompts and obtaining action guidance.
  • 3.1 Task-oriented UI Prompting: AutoDroid’s prompt supplies overall guidance, task representation, current UI state, and output requirements.The current UI state refers to the interface displayed in the prompt’s black-box area.
  • 3.1.1 Converting GUI to Simplified HTML Representation: The GUI parser converts interfaces into simplified HTML using tags for buttons, checkboxes, scrollers, inputs, and other views.These tags represent clickable, checkable, swipeable, editable, and miscellaneous UI elements.
  • 3.1.1 Converting GUI to Simplified HTML Representation: Each HTML element records properties including ID, label, onclick, text, direction, checked status, and input value.The onclick property provides hints about UI states reached after interaction.
  • 3.1.1 Converting GUI to Simplified HTML Representation: AutoDroid automatically scrolls through vertically scrollable interfaces to include otherwise hidden components in the current UI state.This provides more complete information and reduces the need for explicit scrolling instructions to the LLM.
  • 3.1.2 Restricting the Action Space with Selections: The system requires LLM outputs to follow a predetermined structure containing an element ID, action type, and value.

3.2 Exploration-based Memory Injection

AutoDroid explores apps to build app-specific memory, then injects relevant simulated-task knowledge into prompts. It also uses this memory to create app-specific training data for local LLMs.

  • 3.2 Exploration-based Memory Injection: Exploration-based memory injection addresses the inability of raw UI-transition data to directly provide task-automation knowledge.The UI Transition Graph must be transformed into information that LLMs can use for app understanding and decisions.
  • 3.2.1 Simulated Task Generation: AutoDroid analyzes UI Transition Graphs to summarize the functionality associated with UI elements and synthesize simulated tasks.The graph records connections between interfaces and the elements available on each screen.
  • 3.2.1 Simulated Task Generation: Each app-memory entry stores a simulated task together with the UI states and UI elements needed to complete it.The table records paths from the initial interface to the state where the functionality is available.
  • 3.2.2 Augmenting Prompts with App Memory: AutoDroid selects the most relevant simulated tasks by comparing their embeddings with the current user task.The resulting tasks are used to augment the prompt with app-specific guidance.
  • 3.2.2 Augmenting Prompts with App Memory: Algorithm 1 generates a guide from app memory and similar simulated tasks, then augments matching UI elements before each LLM query.The online loop reads the current GUI, generates a prompt, obtains an action, executes it, and records it in history.
  • 3.2.2 Augmenting Prompts with App Memory: The prompt’s onclick property describes the functionality reached after interacting with an element, using the most relevant simulated task information.This turns app-memory knowledge into actionable hints embedded in the HTML UI representation.
  • 3.2.3 Tuning Local LLM with App-specific Data: Local LLMs offer a cost-effective alternative to cloud models but show weaker reasoning and lower accuracy, motivating app-specific fine-tuning.
  • 3.2.3 Tuning Local LLM with App-specific Data: Naively generated fine-tuning answers contain only target element, action type, and value, lacking detailed context for action selection.AutoDroid therefore asks larger LLMs to generate step-by-step reasons for the selected actions.

3.3 Multi-granularity Query Optimization

AutoDroid reduces inference overhead by lowering both prompt size and the number of LLM queries. It prunes redundant UI content, merges GUI states, and directly executes sufficiently obvious actions from app memory.

  • 3.3 Multi-granularity Query Optimization: LLM queries are AutoDroid’s primary overhead, so reducing query frequency lowers task-execution cost.
  • 3.3.1 Reducing Prompt Length: Token pruning removes noninformative UI elements and merges functionally equivalent elements to shorten HTML prompts.
  • 3.3.2 Reducing Query Times by Shortcuts and GUI Merging: GUI merging combines scrolled interfaces so the LLM can select a target without separate scrolling and clicking queries.The merged prompt can produce one action such as scrolling directly to a target and touching it.
  • 3.3.2 Reducing Query Times by Shortcuts and GUI Merging: Shortcut execution uses embedding similarity to identify simulated tasks whose recorded action sequences can be performed without calling the LLM.A threshold γ controls how strictly similar tasks are selected.

4 IMPLEMENTATION

AutoDroid is implemented with Python and Java, fine-tunes Vicuna locally, and adds safeguards for risky actions and private information.

  • 4 IMPLEMENTATION: AutoDroid is implemented using Python and Java, with Vicuna fine-tuned through PyTorch.
  • 4 IMPLEMENTATION: Risky actions that may alter data or cannot be undone require user confirmation before execution.The paper illustrates this safeguard with calling a contact and verifying the number before dialing.
  • 4 IMPLEMENTATION: A privacy filter detects personal information in cloud-bound prompts, replaces it with non-private placeholders, and restores the mappings afterward.

5 BENCHMARK

DroidTask is a reproducible benchmark for end-to-end Android task automation, combining 158 tasks from 13 apps with action traces, exploration memory, and executable environments.

  • DroidTask contains 158 high-level tasks extracted from 13 popular Android apps.
  • The benchmark provides GUI action traces, exploration memory, and app environments for evaluating agents that require dynamic information.
  • All 13 apps are installed with necessary permissions and can reproduce the collected GUI action traces.
  • Annotators created natural-language tasks by interacting with selected open-source Android apps through a desktop-controlled smartphone interface.

6 EVALUATION

AutoDroid consistently improves Android action prediction and task completion over baselines, while memory injection and query optimizations address accuracy, failure modes, and runtime cost.

  • 6.2 Action Accuracy: AutoDroid outperforms baselines on every action type, achieving a 37.6% overall accuracy improvement.
  • 6.4.2 Query Optimization: UI pruning and merging reduce the average choices per GUI state from 36.4 to 13.2, while optimized prompts reduce inference latency by 21.3% on average.
  • 6.3 Task Completion Rate: AutoDroid improves task completion over LLM-framework by 40.5% with Vicuna-7B, 26.4% with GPT-3.5, and 39.7% with GPT-4.
  • 6.3 Task Completion Rate: As task length increases, completion rate decreases because stepwise errors accumulate and annotators may mark alternative valid approaches incorrect.
  • 6.4.1 Memory Injection.: Memory improves completion more than single-step accuracy by supporting crucial decisions, with smaller models benefiting more than larger models.
  • 6.4.1 Memory Injection.: Fine-tuned Vicuna-7B with generated app-memory Chain-of-Thought data and some MoTiF data achieves 57.7% action accuracy and 41.1% completion rate.Its input accuracy is 40.0%; without MoTiF data, action accuracy falls to 51.9% and inputting accuracy is 0%.
  • 6.4.2 Query Optimization: AutoDroid reduces LLM calls by 1.2 per task, yielding a 13.7% decrease in total calls; correct shortcuts save 38.02% of steps.
  • 6.4.2 Query Optimization: Offline preparation takes about 0.5–1 hour for UTG generation, 5–10 minutes for simulated-task synthesis, and about 10 seconds for embedding.This preparation is performed once per app and not at runtime.

7 RELATED WORK

Related work covers GUI understanding and augmented LLMs, while AutoDroid uses UI-transition memory to support complex multi-step mobile automation without custom APIs.

  • UI Understanding and Automation: Prior GUI research summarizes functions, interprets UI-element purposes, and answers questions about individual interfaces.
  • UI Understanding and Automation: AutoDroid instead leverages UI-transition memory to complete complex multi-step tasks and augment LLM reasoning and planning.
  • Augmented LLM: Augmented-LLM methods address limits in stored knowledge and context length by incorporating tools such as browsers, APIs, and other models.
  • Augmented LLM: Unlike approaches depending on public APIs, AutoDroid does not require custom APIs, which are uncommon in mobile applications.

8 DISCUSSION

AutoDroid’s practical deployment is constrained by latency, while its behavior is also sensitive to the LLM temperature setting. The discussion identifies smaller-model collaboration and instruction caching as possible ways to reduce repeated LLM use.

  • Randomness of LLMs: 2.1% accuracy reduction occurred when GPT-3.5 temperature changed from 0.25 to 0.Conversely, increasing temperature to 0.7 boosted action accuracy by 3.8%.
  • Limitations: Increased latency limits the practical use of AutoDroid.The authors propose using smaller models after one LLM-generated task guideline and caching common instructions to reduce repeated invocations.

9 CONCLUSION

AutoDroid is presented as an LLM-powered mobile task automation system supporting arbitrary tasks without manual efforts. The authors report effective task automation and argue that combining LLM commonsense knowledge with app-specific knowledge could support helpful personal assistants.

  • Conclusion: AutoDroid supports arbitrary mobile tasks without manual efforts.The system is described as an LLM-powered mobile task automation approach.
  • Conclusion: The method outperformed existing training-based and LLM-based baselines in experiments.The conclusion characterizes the reported results as demonstrating effective task automation.
  • Conclusion: Combining LLM commonsense knowledge with mobile-app domain knowledge could help realize intelligent personal assistants.The authors frame this synergy as a potential route toward helpful assistants.
Loading 2308.15272v4…