GPTVoiceTasker: Advancing Multi-step Mobile Task Efficiency Through Dynamic Interface Exploration and Learning

• Existing assistants often require developers to predefine narrow app-specific actions and struggle with unclear or unanticipated commands.These constraints leave users unable to control all app functions through voice interactions.Source: §1 · p. 2
• GptVoiceTasker combines LLM command interpretation with dynamic interface exploration for unprecedented tasks and stored usage knowledge for subsequent similar tasks.It predicts the next on-screen action and records navigation knowledge for later reuse.Source: §1 · pp. 2–3
• The system uses prompt engineering to map natural-language commands to actions such as scrolling, tapping, and text input without extensive model training.GptVoiceTasker was released as open source for continued research and improvement.Source: §1 · p. 3
• 34.85% faster task completion was observed in a user evaluation, with positive usability feedback.The evaluation involved 18 participants comparing GptVoiceTasker with two state-of-the-art baselines.Source: §1 · p. 3
• The command parser exceeded 90% accuracy, while multi-step task completion reached 85.7% success.Automated execution reached 82.7% for direct-match tasks and 72.0% for tasks with different parameters.Source: §1 · p. 3

2.1 Voice Control & Automation on Mobile Devices

• Earlier voice-control systems faced usability limits because rigid parsing and narrow use cases required precise, machine-like commands.Record-and-replay tools also required users to manually create execution paths for each use case.Source: §2.1 · p. 3
• GptVoiceTasker predicts suitable actions from current UI screens without pre-programming and automatically constructs its task database during use.This contrasts with systems requiring manual recording of execution paths.Source: §2.1 · p. 3

2.2 Large Language Models for Enhanced Human-AI Collaboration

• LLMs enable natural-language commands to be mapped to executable tasks across visualization, operating-system, and robotic settings.They can handle varied expressions sharing similar intentions beyond traditional heuristic command patterns.Source: §2.2 · p. 4
• Existing mobile LLM agents support single-screen or multi-step tasks but do not fully address continuity between tasks within the same application.GptVoiceTasker collects domain knowledge and uses prompt techniques to support both familiar and novel tasks.Source: §2.2 · p. 4

• GptVoiceTasker prioritizes saved in-app navigation knowledge to streamline familiar tasks and improve execution efficiency and reliability.The database is expanded whenever the system encounters a new task.Source: §3 · p. 4
• For unprecedented tasks, the assistant predicts and executes a step-by-step on-screen navigation sequence, recording the resulting knowledge for later reuse.This creates a loop between exploration of new tasks and automation of subsequent similar requests.Source: §3 · p. 4

3.1 Unprecedented Task Exploration

• The system progressively predicts and executes each action for an unprecedented task using contextual mobile UI, task, application, and system information.These data are assembled into prompts for LLM action selection.Source: §3.1 · p. 4, §3.1.1 · p. 4

3.1.1 Data Collection Module.

• UI knowledge represents each screen as a hierarchy of containers and visible elements with types, labels, unique IDs, locations, and allowed actions.Resource names and alternative captions help label graphical elements, while action constraints prevent unsupported interactions.Source: §3.1.1 · pp. 4–5
• Runtime UI heuristics remove out-of-bounds, empty, invisible, and uninterpretable views to reduce mismatches between collected information and visual screens.These mismatches can lead to incorrect LLM interactions.Source: §3.1.1 · p. 5
• Task, application, and system knowledge records prior actions and visited pages, identifies the current app and activities, and supplies screen-resolution context.Together, these fields help align later actions with the task history and validate navigation.Source: §3.1.1 · pp. 5–6

3.1.2 Private Information Anonymisation.

• Private UI text is detected with lightweight named-entity recognition and replaced by standardized tags before processing by the LLM.Examples include replacing sensitive values with tags such as <address> or <phone number>.Source: §3.1.2 · p. 6

3.1.3 Prompt Creation.

• Least-to-most prompting first selects an action type and then identifies its target UI element, while few-shot chain-of-thought exemplars guide the response.The number of exemplars is dynamically adjusted to remain within the token limit.Source: §3.1.3 · p. 6

3.1.4 Action Executor.

• The action executor adapts decisions to the current UI, provides audio feedback, detects screen transitions, retries failed actions, and delays collection until content is loaded.For text entry, it verifies that the entered text appears in the target field.Source: §3.1.4 · p. 6

3.2.1 Transition Graph.

• GptVoiceTasker automatically records app navigation as a directed graph whose nodes represent UI pages and whose edges encode transition actions and targets.The graph expands as users encounter new pages and transitions, avoiding manual shortcut-task construction.Source: §3.2.1 · p. 7, §3.2 · p. 7

3.2.2 Screen Description & Command Pattern Matching.

• LLM-generated screen descriptions and semantic matching identify both the current page and the destination page despite dynamic UI content.Descriptions summarize page functionality, app and activity names, and interactive elements; saved commands help rank destination screens for user requests.Source: §3.2.2 · pp. 7–8

3.2.3 Path Finding & Execution.

• The system uses the Shortest Path algorithm to derive and execute the action sequence connecting the current screen to the destination screen.Saved paths can also be parameterized by replacing task-specific entities, such as changing “sushi” to “spaghetti” in a text-entry action.Source: §3.2.3 · p. 8
• After each action, semantic validation compares the observed page with the expected node and can revert actions, seek alternatives, or explore a new path when they differ.This validation addresses changing coordinates, pop-ups, and other dynamic GUI conditions.Source: §3.2.3 · p. 8

3.2.4 Human Feedback Loop.

• User feedback validates completed tasks, records natural-language problems for future prompts, and supports manual command amendments for personalization.The feedback loop is intended to improve subsequent execution and enable users to customize or shorten automation triggers.Source: §3.2.4 · pp. 8–9, Figure 4 · p. 8

3.3 Implementation

• The Android application uses Accessibility Service events and AccessibilityNodeInfo objects to detect UI changes and build a hierarchical representation of on-screen elements.Application-level information is collected through Android’s PackageManager.Source: §3.3 · p. 9
• GptVoiceTasker integrates GPT-4 while storing screen descriptions and transition-graph data locally for efficient personalized access.The system and its prompt templates were made publicly available for further research.Source: §3.3 · p. 9

4 TECHNICAL EVALUATION

• The evaluation comprises three experiments covering command parsing and on-screen interaction, unseen multi-step task exploration, and usage-based multi-step execution.These experiments assess complementary capabilities of the proposed system.Source: §4 · p. 9

4.1.1 Experiment Setup & Metric.

• The dataset was constructed because prior test sets contained synthetically biased language patterns and examples that became obsolete after application updates.The authors therefore used participant-generated commands to align the test set more closely with real-world interaction.Source: §4.1.1 · p. 9
• 278 natural-language commands were collected from 31 participants who issued spoken commands while completing screenshot-based Android tasks.The commands were annotated with intended actions and target UI elements to better reflect real-world interactions.Source: §4.1.1 · p. 9
• The evaluation reports Exact Match Accuracy, Target F1, and Action F1, measuring exact sequence agreement and micro-F1 for targets and actions.Target denotes the UI component being interacted with, while action denotes the executable operation.Source: §4.1.1 · p. 9

4.1.2 Evaluation Result.

• 84.7% EM, 91.7% Action F1, and 84.7% Target F1: GptVoiceTasker outperforms all baselines across the three reported metrics.Table 1 compares the systems using Exact Match Accuracy, Action F1, and Target F1.Source: §4.1.2 · p. 10, Table 1 · p. 10
• GptVoiceTasker’s multi-level knowledge prompts, few-shot learning, and Chain of Thought improve action and target prediction across linguistic variants.The results also indicate that LLMs learn associations such as PRESS with buttons, ENTER_TEXT with text fields, and OPEN with app names.Source: §4.1.2 · p. 10

4.2.1 Experimental Setup & Metrics.

• The evaluation sampled 140 Android-in-the-wild Google Apps tasks containing 1–15 steps, with a mean of 6.705 steps and standard deviation of 2.764.GptVoiceTasker explored each task without database guidance, and success was determined by matching the demonstration’s final screen within the step allowance.Source: §4.2.1 · p. 10

4.2.2 Results.

• 85.7% success (120/140) versus 56.4% (79/140) for Wang et al.’s baseline on unseen multi-step tasks.Tasks were successful when the system reached the dataset’s ground-truth screen within three additional steps.Source: §4.2.2 · p. 10, §4.2.1 · p. 10
• The system used app and package names, runtime app availability, historical messages, and error handling to select actions without cyclic navigation or repeated taps.It could also choose alternative routes, such as directly searching Settings options instead of following demonstrated scrolling paths.Source: §4.2.2 · p. 10
• Failures included time-related commands because the LLM lacked current date and time knowledge, preventing selection of dates such as Friday of the following week.The authors suggest adding temporal information to prompts.Source: §4.2.2 · p. 11
• The system also exceeded the step budget for Google Maps settings, where reaching the relevant control required pressing a profile picture unrelated to the command.Suggested remedies include contextual prompt knowledge and random exploration before repeating a command.Source: §4.2.2 · p. 11

4.3.1 Experimental Setup & Metric.

• Each test case counted as successful when one voice command opened the desired feature, using manually populated transition graphs and stored screen descriptions as task knowledge.Success rate was the primary metric.Source: §4.3.1 · p. 11

4.3.2 Results.

• 82.7% of direct-match tasks and 72.0% of parameterized tasks were completed successfully using saved task execution.Table 2 reports results across five application categories, with tasks averaging 5.19 steps.Source: §4.3.2 · p. 11, Table 2 · p. 11, §4.3.1 · p. 11
• Messaging and directions applications showed especially strong performance, which the authors attribute to relatively static interface structures supporting semantic matching and parameter substitution.Alarm-setting tasks showed diminished accuracy.Source: §4.3.2 · p. 11
• Complex parameterized tasks failed when additional UI steps were required, such as selecting minutes separately while setting an alarm for 7:30 instead of 7:00.Pop-up ads and unusual runtime UI elements not encountered during saving also caused difficulties.Source: §4.3.2 · p. 11

5.1 Tasks

• The study used six tasks containing 6–10 sequential steps, with 18 participants completing six counterbalanced tool-and-task trials.Participants were familiar with smartphones but had no prior experience with the experimental tools; task times were capped at 60 seconds per step.Source: §5.1 · p. 12, §5.2 · p. 12, §5.3 · p. 12

5.4 Result

• 92.5 seconds average completion time outperformed Voice Access at 162.2 seconds and Voicify at 141.9 seconds.GptVoiceTasker also automated an average of 2.2 steps across six tasks, although network latency and unfamiliarity with saved-task triggers remained issues.Source: §5.4.1 · p. 12, Figure 5 · p. 12, Table 4 · p. 13

5.4.2 Cognitive Load & Usability Ratings.

• GptVoiceTasker reduced participants’ reported mental demand, temporal demand, and effort relative to the baseline systems while achieving better performance.Cognitive load was assessed with NASA-TLX and analyzed using a Friedman test.Source: §5.4.2 · pp. 12–13, Figure 6 · p. 13
• GptVoiceTasker achieved an average SUS score of 79.861, surpassing Voicify at 47.917 and Voice Access at 36.528.Participants also rated it less complex and inconsistent, better integrated, and quicker to learn, with reported differences at p < 0.001.Source: §5.4.2 · p. 13, Figure 7 · p. 14

5.4.3 Qualitative Feedback.

• Participants valued natural command interpretation and robustness to input mistakes, describing the system as comfortable and stress-free for everyday tasks.They contrasted this with traditional approaches requiring fixed input formats.Source: §5.4.3 · pp. 13–14
• Participants praised saved-task automation for accelerating tasks but requested task suggestions, saved-task lists, better onboarding, live transcription, loading indicators, editable tasks, and clearer audio feedback.These suggestions target discoverability, transparency, customization, and execution-delay communication.Source: §5.4.3 · p. 14

Towards the adoption of the voice-centric interface.

• GptVoiceTasker supports voice-centric interaction by mapping user intentions to visual elements and decomposing tasks into flexible action sequences.Historical usage enables personalization, while anonymization is presented as a privacy measure for screen-based processing.Source: §6 · pp. 14–15
• Voice commands can substitute touch interactions for motor and visual impairments, including shortcuts for locating familiar-screen controls.The discussion also identifies potential benefits for users who struggle with technology and elderly users.Source: §6 · p. 15
• Voice recognition remains challenging in diverse environments, limiting broader adoption across settings such as smart homes and healthcare.The discussion frames this as an area requiring continued research.Source: §6 · p. 15

LLMs for task automation on user visual interfaces.

• LLM-based UI automation benefits from spatial and hierarchical interface representations, especially for unlabeled icons and images.The discussion calls for better adaptability to unconventional interfaces and complex tasks, while visual models may improve accuracy at a response-time cost.Source: §6 · p. 15

Towards responsible AI in software systems.

• Processing on-screen personal data through LLMs creates privacy and security concerns, while consequential actions may require explicit user confirmation.The discussion identifies anonymization and confirmation mechanisms as safeguards for responsible deployment.Source: §6 · p. 15

Limitations.

• Usage-based execution is inapplicable to unused applications, and the system’s effectiveness beyond Android smartphones remains untested.Different devices may require different interaction logic, layouts, and output lengths, motivating broader investigation.Source: §6 · p. 15

7 CONCLUSION

• GptVoiceTasker uses prompt engineering and stored previous usages to interpret commands and automate subsequent repetitive smartphone tasks.The system is designed to improve interaction and performance through LLM reasoning and usage-based execution.Source: §7 · p. 15
• User evaluation validated high usability in real-world tasks, improved user performance, and reduced mental stress load.The paper presents these findings as aligned with its design objectives.Source: §7 · p. 16