Source-linked AI summary
Explore, Select, Derive, and Recall: Augmenting LLM with Human-like Memory for Mobile Task Automation
Sunjae Lee, Junyoung Choi, Jungjae Lee, Munim Hasan Wasi, Hojun Choi, Steven Y. Ko, Sangeun Oh, Insik Shin
TL;DR
LLM-based mobile automation is powerful but unreliable, slow, and costly. MobileGPT addresses these limits with human-like app memory that decomposes tasks into reusable subtasks and recalls them across contexts. It achieves strong new-task accuracy and near-perfect adaptation while reducing repeated-task latency and cost.
Problem
LLM-based mobile task automators face unpredictable execution and high time and monetary costs, limiting reliable practical use.
Source: §1 · p. 1Method
MobileGPT combines Explore, Select, Derive, and Recall with hierarchical app memory, user repair, and adaptive retrieval of modular subtasks.
Source: §1 · p. 2, §5.1 · p. 7, §5.2 · p. 8Results
MobileGPT completes new tasks with 82.7% accuracy, adapts learned tasks with 98.75% accuracy, and reduces recall latency and LLM query cost by 62.5% and 68.8%.
Source: §1 · p. 2Takeaways & Limitations
The results support reusing modular task knowledge to improve mobile automation accuracy, contextual adaptability, and efficiency within the evaluated apps and tasks.
Source: §1 · p. 2, §10 · p. 14The current implementation does not support apps lacking text representations, including third-party UI engines and image-dominated screens.
Source: §9 · p. 14
Abstract
from arXiv · showhide
The advent of large language models (LLMs) has opened up new opportunities in the field of mobile task automation. Their superior language understanding and reasoning capabilities allow users to automate complex and repetitive tasks. However, due to the inherent unreliability and high operational cost of LLMs, their practical applicability is quite limited. To address these issues, this paper introduces MobileGPT, an innovative LLM-based mobile task automator equipped with a human-like app memory. MobileGPT emulates the cognitive process of humans interacting with a mobile app -- explore, select, derive, and recall. This approach allows for a more precise and efficient learning of a task's procedure by breaking it down into smaller, modular sub-tasks that can be re-used, re-arranged, and adapted for various objectives. We implement MobileGPT using online LLMs services (GPT-3.5 and GPT-4) and evaluate its performance on a dataset of 185 tasks across 18 mobile apps. The results indicate that MobileGPT can automate and learn new tasks with 82.7% accuracy, and is able to adapt them to different contexts with near perfect (98.75%) accuracy while reducing both latency and cost by 62.5% and 68.8%, respectively, compared to the GPT-4 powered baseline.
1 Introduction
MobileGPT addresses the reliability, consistency, and cost limitations of LLM-based mobile task automation with a human-like app memory. It decomposes tasks into reusable subtasks and reports strong accuracy, adaptation, and efficiency results.
Motivation
- LLM-based automators reduce manual development and training demands but remain limited by unpredictable execution and high time and monetary costs.The paper reports that tasks taking humans just over 30 seconds could take LLMs more than two minutes and cost over a dollar per execution.Source: §1 · p. 2
Contribution
- MobileGPT augments an LLM task automator with app memory designed for accurate, consistent, adaptable, and efficient repeated execution.Its stated design goals are accurate and consistent execution, contextual adaptation, and reduced cost and time for repeated tasks.Source: §1 · p. 2
Approach
- The system emulates human Explore, Select, Derive, and Recall behavior by decomposing tasks into modular subtasks that can be reused and adapted.A learned messaging procedure can be adapted to different recipients or applied to related tasks involving message reading.Source: §1 · p. 2
System Design
- MobileGPT combines prompting and user correction for reliable execution, hierarchical memory for storage, and pattern matching with in-context learning for flexible retrieval.Tasks are represented as function-call subtasks, each further decomposed into primitive actions.Source: §1 · p. 2
Results
- 82.7% task completion on new tasks outperformed AutoDroid by 8% and AppAgent by 15.3%, while learned-task adaptation reached 98.75%.Recalling learned tasks reduced completion time by 62.5% and LLM query costs by 68.8%.Source: §1 · p. 2
2 System Overview
MobileGPT learns mobile tasks through an Explore-Select-Derive workflow and later recalls learned procedures directly or reuses known subtasks. Its design addresses execution reliability, hierarchical sharing, parameterization, and contextual adaptation.
2.1 System Workflow
- MobileGPT explores app screens, selects subtasks for user instructions, derives primitive actions, and stores completed instructions hierarchically for recall.Recall bypasses the learning phases for matched tasks and reuses known subtasks when learning new tasks.Source: §2.1 · p. 3
2.2 System Design
- MobileGPT uses dual-strategy correction because LLM execution is not guaranteed to be accurate, allowing both self-feedback and user repair.The design targets mistakes such as entering text instead of selecting a required filter option.Source: §2.2 · p. 4
- A three-level hierarchy of tasks, subtasks, and actions shares common execution knowledge across related tasks instead of training repeated subtasks separately.For example, locating a contact can be shared between sending a message and opening a chat.Source: §2.2 · p. 4
- Function-call subtasks incrementally identify explicit and implicit parameters, enabling one learned task representation to cover varied instructions.The representation models instructions such as sending a message to a named recipient with specific content.Source: §2.2 · p. 4
- MobileGPT adapts recalled executions to changed parameters and screen content, using few-shot LLM guidance when direct adaptation fails.The system avoids blindly replaying actions across different recipients or interface states.Source: §2.2 · p. 4
3 Accurate and Reliable Task Execution
MobileGPT uses structured screen representations and an Explore-Select-Derive cycle to execute tasks, then combines automatic feedback with human repair when LLM behavior fails.
3.1 Prompting Mobile Screen to LLM
- MobileGPT converts accessibility layouts into indexed HTML-like screen representations, reducing average token use by 84.6%.Interactive elements become indexed tags such as button, input, and scroll for LLM reference.Source: §3.1 · p. 4
3.2 Explore, Select, and Derive
- During Explore, the LLM enumerates screen subtasks with names, descriptions, parameters, and UI indexes, while offline exploration caches available functionality.Select chooses and parameterizes a subtask; Derive repeatedly selects and executes low-level actions until completion or screen transition.Source: §3.2 · pp. 4–5
- Known subtasks can bypass Derive, whereas unknown subtasks alternate Select and Derive until the instruction is completed.Derive supplies the selected subtask, current screen representation, and allowed actions to the LLM.Source: §3.2 · p. 5
3.3 Dual Strategy Failure Handling
- MobileGPT detects invalid UI interactions and execution loops, then appends rule-based feedback to subsequent prompts for LLM self-correction.Detected errors include nonexistent or non-clickable UI indexes, unchanged screens, and repeated screen visits.Source: §3.3 · p. 5
- When self-correction fails, users can repair Explore, Select, or Derive errors and return control to MobileGPT with the correction encoded as subtask-level feedback.Users can modify available subtasks, selected parameters, or demonstrated action sequences.Source: §3.3 · pp. 5–6
4 Hierarchical App Memory
MobileGPT stores app knowledge in a hierarchical, reusable memory and classifies screens by available functionality rather than appearance. This supports cross-task subtask sharing and robust page matching.
4 Hierarchical App Memory
- The app memory archives Explore, Select, and Derive results so knowledge accumulates across the app rather than only within individual tasks.Its transition graph stores available subtasks, task subtask sequences, and primitive-action procedures.Source: §4 · p. 6, §4.1 · p. 6
4.1 Memory Structure
- Transition-graph nodes represent functional app pages, while edges represent subtasks implemented as primitive-action sequences shared across tasks.Screens with different appearances may share a node when they provide the same functionality.Source: §4.1 · pp. 6–7
- Tasks are stored as node-subtask pairs, allowing learned procedures to bypass Explore, Select, and Derive and execute from different current app states.Subtask knowledge is shared across screens and tasks rather than confined to one task instance.Source: §4.1 · pp. 6–7
4.2 Sub-task based Screen classification
- Subtask-based screen classification matches required UI elements and verifies functionality instead of repeatedly exploring every new screen.A cosine-similarity check prevents redundant nodes when newly generated subtasks match an existing node.Source: §4.2 · p. 7
- On 269 screens, subtask-based classification produced 3 false positives and no false negatives, compared with 38 and 57 false positives and 14 and 28 false negatives for text- and vision-based methods.The comparison reports lower error counts for MobileGPT than both existing approaches.Source: §4.2 · p. 7
5 Flexible Task Recall
MobileGPT recalls learned sub-tasks by adapting stored actions to task parameters and changing screen content. When attribute-based adaptation fails, it uses corrected examples to improve LLM-derived actions.
5.1 Attribute-based Action Adaptation.
- MobileGPT generalizes actions against screen attributes and task parameters, then reverses those steps to locate the correct UI in new contexts.For example, a stored contact-selection action can replace Bob with Alice and update the UI index accordingly.Source: §5.1 · p. 8
- MobileGPT adapts recalled tasks by slot-filling parameters from the instruction and current screen, asking the user when required values are unknown.It replays the learned sub-task sequence while adjusting each action to the current context.Source: §5 · p. 7, §5.1 · p. 8
- MobileGPT derives missing information incrementally, enabling granular adaptation to incomplete or implicit instructions such as missing message content or unstated contacts.It asks for missing content in one case and identifies the contact from the recent-call list in another.Source: §5.1 · p. 8
5.2 In-context Action Adaptation
- Attribute-based adaptation can fail when UIs lack distinctive attributes, share attributes, or change unexpectedly after app updates.These cases trigger renewed LLM-based action derivation.Source: §5.2 · p. 8
- When rule-based adaptation fails, MobileGPT supplies a previously correct derivation—including user-corrected actions—as a few-shot example to improve consistency and avoid repeated mistakes.The example contains a prior instruction, abbreviated screen representation, and correct output.Source: §5.2 · p. 8
6 Implementation
MobileGPT combines an Android app for screen capture and action injection with a Python server that manages memory and task phases. Different LLM models are assigned to reasoning and parameter-filling roles.
6 Implementation
- The implementation uses an Android app to capture screen representations and inject actions, while a Python server manages memory and Explore, Select, Derive, and Recall.The components communicate over Wi-Fi.Source: §6 · pp. 8–9
- GPT-4-turbo handles screen understanding and multi-step reasoning, while GPT-3.5-turbo handles instruction translation and sub-task parameter slot-filling.The division assigns the more capable model to Explore, Select, and Derive operations.Source: §6 · p. 9
- MobileGPT recommends apps by retrieving the three most relevant installed-app descriptions from a vector database, then launches the selected app with its app memory.Descriptions are collected from Google Play and indexed with text embeddings.Source: §6 · p. 9
- For scrolling without a definite target UI, MobileGPT skips intermediate scrolls when the next target is already visible and otherwise scrolls until that target appears.This reduces unnecessary replayed scrolling actions.Source: §6 · p. 9
7 Evaluation
The evaluation compares MobileGPT with AppAgent and AutoDroid across 185 tasks and 18 apps, using cold-start learning and warm-start recall. MobileGPT shows strong accuracy and substantially lower warm-start cost and latency.
7.1 MobileGPT Dataset
- The dataset contains 80 tasks across eight mobile apps, with two parameterized instructions per task designed to measure learning and adaptation.The resulting dataset contains 160 user instructions.Source: §7.1 · p. 9
7.2.1 Experimental Setup.
- MobileGPT is evaluated against AppAgent and AutoDroid on 185 tasks spanning 18 apps, with cold-start constructing memory and warm-start reusing it.The comparison excludes overlapping or deprecated apps and tasks, and baselines run each task once.Source: §7.2.1 · p. 9, §7 · p. 9
7.2.2 Accuracy.
- MobileGPT’s cold-start has the highest accuracy across all three datasets, while warm-start accuracy is nearly identical despite one additional recall failure.The paper attributes performance to preserving hierarchical UI information and decomposing tasks into sub-tasks.Source: Table 1 · p. 9, §7.2.2 · p. 10
7.2.3 Cost & Latency.
- AppAgent has the lowest accuracy and highest cost and latency, whereas AutoDroid lowers cost through concise representations that may omit information needed for accuracy.The comparison covers 77 tasks successfully completed by all three systems, with comparable step counts.Source: §7.2.3 · p. 10
- 87% and 90% reductions in latency versus AutoDroid and AppAgent, respectively, are achieved by MobileGPT during warm-start execution.Cold-start cost and latency fall between the two baselines, while warm-start reuses learned sub-tasks and actions.Source: §7.2.3 · p. 10, Figure 5 · p. 10
7.3 Ablation Study
MobileGPT’s ablation study finds that its decomposed, memory-augmented design improves task success and step accuracy while substantially reducing warm-start latency and cost. These gains arise from reusable subtasks, correction-informed memory, and adaptation across task parameters.
7.3.1 Experimental Setup.
- The ablation baseline shares MobileGPT’s prompts but removes its Explore-Select-Derive structure and memory to isolate architectural effects.Cold-start and warm-start evaluations test initial execution and adaptation to a second instruction with different parameters.Source: §7.3 · p. 10, §7.3.1 · p. 10
7.3.2 Task Success Rate
- 98.75% warm-start task accuracy is achieved, with one exception in Uber Eats, after corrected executions are stored for recall.Attribute-based and in-context adaptation support consistent execution when task parameters and resulting screens change.Source: §7.3.2 · p. 11, Figure 6 · p. 11
- MobileGPT outperforms the baseline in cold-start accuracy by decomposing tasks into subtasks and sharing learned subtasks across tasks.Corrections to a subtask prevent repeated mistakes in later tasks involving overlapping subtasks, unlike the baseline’s repeated scrolling errors.Source: §7.3.2 · p. 11
7.3.3 Step Accuracy.
- 100% in-context adaptation accuracy is achieved, with all 53 actions requiring adaptation successfully transferred to new instructions and screens.Warm-start recall has only one failed slot-filling query, while 53 of 327 primitive actions require in-context rather than attribute-based adaptation.Source: §7.3.3 · pp. 11–12
7.3.4 Cold-start latency & cost.
- 3.9% higher latency and 0.6% higher cost occur on average during cold starts despite the additional Select phase.Reuse of learned subtasks can nevertheless make MobileGPT outperform the baseline in Telegram, while Gmail’s 6.7% memory hit rate limits cold-start benefits.Source: §7.3.4 · p. 12
7.3.5 Warm start latency & cost.
- 62.5% lower latency and 68.8% lower cost are achieved during warm-start execution than with the baseline.The gains persist even when some actions require LLM-based in-context adaptation.Source: §7.3.5 · p. 12
7.4 Resource Consumption
MobileGPT imposes modest mobile-device resource use while user studies report improved usability for repeated and related tasks. Participants also viewed its repair mechanism as useful, though repairing all three phases remained challenging.
7.4 Resource Consumption
- 330.38 mW average power and 213 MB average memory consumption indicate no significant mobile-device overhead.Most computation is offloaded to the server; WLAN accounts for 26.18 mW and memory use is 0.04% of a typical 6 GB smartphone.Source: §7.4 · p. 12
7.5 User Study
- MobileGPT’s usability improved significantly across subsequent task scenarios, while the baseline was convenient but too slow and PbD required recreating scripts after changes.Reported tests were U=24.0, p=4.8e−8 and U=67.5, p=6.5e−6 for the later scenarios.Source: §7.5 · p. 13
- All participants expressed willingness to use MobileGPT in real-world applications, compared with 35% for the baseline and 30% for PbD tools.All but one participant favored MobileGPT most in the post-survey.Source: §7.5 · p. 13
- 64.6 average SUS score for task repair indicates acceptable user-friendliness despite the complexity of repairing Explore, Select, and Derive errors.The score had a standard deviation of 16.9.Source: §7.5 · p. 13
- Repair necessity and effectiveness were rated 4.7 and 4.8 out of 5, while acceptable accuracy fell from 96% without repair to 84% with repair.Participants were more tolerant of automation errors when a repair mechanism was available.Source: §7.5 · p. 13
8 Related Work
MobileGPT extends LLM-based mobile automation with human-like app memory and reusable sub-tasks to address reliability, cost, and generalization challenges.
Using LLM in UI Task automation
- MobileGPT integrates human-like app memory with LLMs to reuse learned sub-tasks, reducing LLM queries and optimizing latency and cost.The approach targets practical deployment concerns associated with unreliable and expensive LLM-based automation.Source: §8 · p. 13
Macro Mining
- Macro-mining methods generate task-oriented scripts by discovering and combining sub-tasks from app traces without human intervention.Li et al. [16] use LLMs to discover sub-tasks within app traces and combine them into macros.Source: §8 · p. 13
- Existing macro approaches struggle to generalize actions when screen similarity is ambiguous or user interfaces lack critical attributes.They commonly rely on embedding-based screen matching or attribute-based UI matching.Source: §8 · p. 14
- MobileGPT addresses macro generalization through sub-task-based screen classification and in-context action adaptation for more robust task automation.Its higher-level action space consists of reusable sub-tasks that can support different user goals and potentially extend beyond mobile tasks.Source: §8 · p. 14
Caching LLM responses
- MobileGPT is presented as the first LLM-based task automator to explore caching LLM responses rather than focusing caching research solely on chatbot applications.Related work includes caching token states, responses, or smaller models trained to replicate cached responses [9] [73] [52] [60].Source: §8 · p. 14
9 Discussion
The discussion covers privacy safeguards, local-memory extensions, cross-app planning, unsupported app types, and the latency trade-off of adding visual inputs.
Security & Privacy
- MobileGPT can request user confirmation for potentially risky actions and use a PII Scanner to replace personal data with non-private placeholders.Examples include agreeing to terms of service or confirming a payment.Source: §9 · p. 14
Sharing App Memory
- MobileGPT’s local app memory could be shared or crowd-sourced to reduce the need for each user to learn tasks individually.Memory may be categorized by device form factor, while hierarchical UI indices avoid dependence on pixel coordinates.Source: §9 · p. 14
Cross App Task execution
- A global dataset of known tasks could extend MobileGPT to cross-app instructions by planning which task to execute in which app and order.The discussion illustrates this with a banking task followed by a food-delivery task.Source: §9 · p. 14
Unsupported Apps
Using Vision Language Model
- Providing both screenshots and screen representations improves accuracy with GPT-4-turbo, but vision-language models increase latency compared with text-only LLMs.Memory caching can reduce the number of queries, making the higher per-query latency more beneficial to the design.Source: §9 · p. 14
10 Conclusion
MobileGPT enhances mobile task automation by emulating human cognitive processes, with the authors anticipating broader integration of intelligent automation into everyday technology use.
10 Conclusion
- MobileGPT enhances the efficiency and reliability of mobile task automation by emulating human cognitive processes.The conclusion presents MobileGPT as an LLM-powered mobile task automator.Source: §10 · p. 14