PaperlayerView exact revision on arXiv

Source-linked AI summary

AutoTask: Executing Arbitrary Voice Commands by Exploring and Learning from Mobile GUI

Lihang Pan, Bowen Wang, Chun Yu, Yuxuan Chen, Xiangyu Zhang, Yuanchun Shi

arXiv:2312.16062v1cs.HC

TL;DR

Existing voice command interfaces cover only a limited set of GUI tasks, despite the effort required to build and configure them. AutoTask explores mobile GUIs, tries and backtracks between operation sequences, and learns from experience without developer or user configuration. An evaluation found it significantly better than a baseline without accumulated knowledge, with further improvement from knowledge accumulation.

  • Problem

    Existing VCIs support only limited predefined intents, while constructing them for GUI tasks requires significant effort from developers.

    Source: §1 · p. 2

  • Method

    AutoTask explores an unknown GUI, tries operation sequences and backtracks from errors, while accumulating and summarizing experience as knowledge.

    Source: §1 · p. 2

  • Results

    AutoTask performed significantly better than the baseline without accumulated knowledge, and knowledge accumulation further improved its performance.

    Source: §9 · p. 21

  • Takeaways & Limitations

    The evaluation supports the feasibility of a VCI that automates tasks across mobile applications without developer or end-user configuration.

    Source: §1 · p. 3, §9 · p. 21

    AutoTask uses an online GPT-4 service over HTTPS, and its computational process is slow.

    Source: §8 · p. 21

Abstract

from arXiv · show

Voice command interfaces (VCIs) have gained increasing importance, enabling hands-free and eyes-free interaction with digital devices. However, the inherent complexity in constructing effective voice interfaces has limited the VCIs' functionalities to only a small fraction of GUI applications and tasks. This paper presents AutoTask, a VCI capable of automating any task in any mobile application without configuration or modification from developers or end users. The primary challenge for AutoTask is the lack of knowledge, as it needs to accomplish unknown tasks (e.g., user commands) within an unknown environment (e.g., GUI). To address this challenge, AutoTask employs two strategies: (1) trial and error: AutoTask explores the GUI, attempts potential operation sequences, and recovers from errors through backtracking; (2) learning from the environment: AutoTask accumulates experiences during exploration and summarizes correct knowledge from these experiences. We implemented AutoTask on Android devices and conducted an evaluation study, which proved the feasibility of AutoTask.

1 INTRODUCTION

Existing voice command interfaces cover only a limited set of intents, while building and configuring them requires substantial effort. AutoTask addresses unknown tasks in unknown GUIs through exploration, backtracking, and learning from experience; an Android evaluation found the approach feasible.

1 INTRODUCTION

  • VCIs remain limited to predefined intents, and developing them requires effort to specify supported intents, configure execution, and address LLM errors.LLMs can help understand commands and reduce development burden, but developers still need to define intents and their execution.Source: §1 · p. 2
  • AutoTask operates without developer or end-user configuration and aims to execute any intent in any application through a single voice command.It addresses the lack of knowledge involved in completing unknown tasks in unknown GUI environments.Source: §1 · pp. 2–3
  • AutoTask explores the GUI, tries possible operation sequences, evaluates whether they succeed, and backtracks after errors.For example, it abandons a route that reaches a page unable to batch-import contacts and tries another button.Source: §1 · p. 2
  • AutoTask learns environmental, task, and execution knowledge from exploration, including unsuitable actions, required command parameters, and successful operation sequences.Learned sequences can be replayed for similar tasks and help execute other commands.Source: §1 · p. 2
  • An Android implementation and evaluation study found AutoTask usable.The reported experimental results proved its usability.Source: §1 · p. 3

2 RELATED WORK

Existing VCIs trade broad GUI coverage against developer and user effort: they may rely on single-step commands, predefined intents, handcrafted command understanding, or scripts. AutoTask instead computes and checks GUI actions at runtime and learns from exploration without user intervention.

2 RELATED WORK

  • VCIs can reduce interaction burden, but development effort to determine intents, understand commands, and execute tasks correctly limits their GUI coverage.Table 1 compares existing VCIs with AutoTask across these four aspects.Source: §2 · p. 3, Table 1 · p. 3

2.1 Supported intents of VCIs

  • Single-step interfaces cover current GUI contents, while task-oriented assistants automate multi-step tasks but support limited predefined intents [81] [83] [69].AutoTask handles multi-step GUI tasks without a predefined intent set or additional developer or end-user overhead.Source: §2.1 · pp. 3–4

2.2 Understanding User Commands

  • Handcrafted command-understanding rules require developer effort, while LLMs can understand commands from prompts and optional examples; AutoTask adjusts its understanding from the GUI without user involvement.Interactive systems may misinterpret commands, and existing remedies can require developer programming or additional user interaction.Source: §2.2 · p. 4

2.3 Executing the Command

  • Existing VCIs use developer-created scripts or user-provided demonstrations and histories, whereas AutoTask calculates operation sequences at runtime and checks and handles errors without user intervention.User-collected scripts can fail after pop-ups or application updates, sometimes requiring users to handle exceptions.Source: §2.3 · pp. 4–5

2.4 Self-Improvement

  • AutoTask summarizes knowledge from GUI exploration without user intervention, a self-improvement approach related to reinforcement learning.A prior GUI agent, AppBuddy, faces sparse rewards and excessive trial-and-error steps that make it unsuitable for direct use in interactive systems [71] [42].Source: §2.4 · p. 5

3 PROBLEM FORMULATION & SOLUTION

AutoTask addresses the problem of agents facing unknown tasks in unknown environments, where they lack environmental, task, and execution knowledge. It uses trial and error with backtracking and learns from experience to support current and future task execution.

3 PROBLEM FORMULATION & SOLUTION

  • AutoTask simulates GUI operation sequences to execute natural-language interaction intents across applications.The paper frames this as broad coverage of GUI tasks for voice assistants.Source: §3 · p. 5
  • Unknown-task, unknown-environment problems arise because agents lack knowledge of environmental changes, task semantics, and how to execute tasks.For AutoTask, this includes having no predefined intent set, intent-recognition scripts, or execution scripts.Source: §3 · pp. 5–6
  • The proposed explore-learn strategy combines forward exploration with backtracking after errors and learning from the environment.The agent accumulates action-and-observation experiences and summarizes knowledge from them.Source: §3 · p. 6
  • Learned knowledge can support direct execution of previously completed intents and expedite exploration for unknown intents.The system design applies this strategy to voice assistants.Source: §3 · p. 6

4 SYSTEM DESIGN

AutoTask replays known task sequences when possible; otherwise, it explores, checks outcomes, backtracks when needed, and turns experience into reusable knowledge. Its stored knowledge covers GUI behavior, task semantics, and execution strategies.

4 SYSTEM DESIGN

  • Previously executed intents are replayed with current parameter values; failed replay or an unseen command enters explore-learn mode.Exploration alternates forward actions and backtracking, with checks determining whether to stop, continue, or undo an action.Source: §4 · pp. 6–7

4.1 Learning from the GUI: Summarizing Experiences into Knowledge

  • AutoTask records module decisions and GUI observations as a graph whose nodes are pages and edges record module outcomes.It summarizes correct knowledge from experience and applies it to execute completed intents or expedite exploration.Source: §4.1.1 · p. 7, §4 · p. 7

4.1.2 Environmental knowledge.

  • Environmental knowledge represents GUI pages and transitions, either as operation-linked page triplets or as natural-language descriptions of absent GUI possibilities.A triplet records the before-and-after pages and an operation composed of an element, action, and parameters; one example describes what the Add button does not support.Source: §4.1.2 · pp. 7–8

4.1.3 Task knowledge.

  • Task knowledge records an intent, its parameter names, a corresponding command, and that command’s parameter values.For contact import, the example records the intent, file-name parameter, command, and contacts.vcf value.Source: §4.1.3 · p. 8

4.1.4 Execution knowledge.

  • Execution knowledge includes correct operation sequences and natural-language lessons for avoiding errors, such as entering contact details before clicking Save.The correct import sequence is to open Fix & Manage, choose Import from file, and select the file parameter.Source: §4.1.4 · p. 9

4.1.5 From experiences to knowledge.

  • GUI observations become Type-1 environmental knowledge immediately, while correct paths and task knowledge are summarized only after completion.A correct execution path is the shortest experienced path connecting the start and endpoint while covering all parameters.Source: §4.1.5 · pp. 9–10
  • AutoTask compares experience with the correct path to identify errors, then uses an LLM to summarize lessons that can prevent future errors.It waits until task completion to summarize these lessons, when it can better determine whether a step was correct.Source: §4.1.5 · p. 10

4.2 The Understanding Module

The understanding module augments GUI and command information with accumulated knowledge to guide later modules. It infers relevant reachable GUI elements and interprets commands as intents with parameter values.

4.2 The Understanding Module

  • The understanding module produces GUI and command interpretations for subsequent modules and uses stored knowledge to augment both.GUI semantics include page contents and transitions, but resulting content cannot always be foreseen from current observations.Source: §4.2 · p. 10, §4.2.1 · p. 10

4.2.1 Understanding the GUI.

  • For GUI understanding, AutoTask retrieves reachable elements from environmental knowledge and retains those whose semantic similarity to the command exceeds a threshold.This can associate the command “enable app pinning” with the reachable App pinning button under Security & privacy.Source: §4.2.1 · pp. 10–11

4.2.2 Understanding the command.

  • For command understanding, AutoTask identifies an intent and parameter values, using similar historical commands, their operation sequences, current GUI contents, and an LLM.It repeats command understanding each iteration because results may change as experience accumulates.Source: §4.2.2 · pp. 11–12

4.3 The Deciding Module

The deciding module ranks available GUI operations and selects the highest-scoring one. Scores combine estimated relevance with penalties for repeated or incorrect actions.

4.3 The Deciding Module

  • Click, text input, and scroll forward are used for deciding, while the other supported operation types serve backtracking.Source: Table 3 · p. 12
  • AutoTask assigns each available operation a score equal to its basic score divided by one plus its penalty, then chooses the highest-scoring operation.For text input, an LLM determines the content later in the process.Source: §4.3 · pp. 12–13
  • The basic score combines an LLM’s 1.0–7.0 relevance rating with a 0.0–1.0 semantic-similarity tie-breaker.The tie-breaker differentiates options when relevance ratings lack granularity and would otherwise tie.Source: §4.3 · p. 12
  • The penalty factor combines a fixed repetition penalty of 10 with a backtracking penalty for operations judged incorrect.A penalized operation can still be retried if other operations score even lower.Source: §4.3 · pp. 12–13

4.4 The Executing Module & the Backtracking Module

AutoTask injects and reverses GUI operations, then checks whether the task is complete and whether its latest operation was correct. User confirmation and penalty-aware correctness checks guide whether it terminates or continues exploring.

4.4 The Executing Module & the Backtracking Module

  • Executing and backtracking modules inject GUI events, retrieve the hierarchy as HTML, and mark newly appearing elements by comparing pages before and after operations.The hierarchy also omits elements meeting both low-interaction and low-layout-importance criteria.Source: §4.4 · p. 13

4.5 The Checking Module

  • After operations and backtracking, the checking module assesses completeness and correctness; an LLM checks completion, with a 20-step threshold also marking tasks complete.Completion checking also addresses overshoot by detecting unnecessary operations after task completion.Source: §4.5 · p. 13, §4.5.1 · p. 13

4.5.1 completeness check.

  • At completion, users review the shortest path and can confirm, continue as incomplete, or force termination; knowledge summarization varies with their choice and the step threshold.The displayed path includes screenshots, operated-element boxes, and operation descriptions.Source: §4.5.1 · pp. 13–14

4.5.2 correctness check.

  • If a task is incomplete, an LLM judges the last forward step, tolerating high-backtracking-penalty actions; rejected steps accrue a 0–9 error-severity penalty.The penalty accumulates across repeated rejections and may exceed 9.Source: §4.5.2 · p. 14

5 IMPLEMENTATION

AutoTask’s implementation supplies context about the command, operation history, and GUI, then uses embedding similarity or text completion to answer computational questions. The text-completion output follows a parseable JSON format.

5.1 AutoTask’s context

  • AutoTask’s context combines the user command, executed operations, HTML-formatted GUI contents augmented with environmental knowledge, and any latest semantic-comprehension result.The implementation section describes LLM use for computational purposes.Source: §5.1 · p. 14, §5 · p. 14

5.2 Embedding & Similarity: Choosing One or More Answers from Several Candidates

  • Embedding & Similarity scores candidate answers by cosine similarity between vectors for the question and candidates, using OpenAI’s embedding API.Element descriptions can include surrounding elements because they may be semantically relevant.Source: §5.2 · p. 14

5.3 Text Completion: Answering Questions

  • Text completion sends a question to GPT-4 and uses its generated response as the answer; an output template specifies parseable JSON.Table 5 summarizes this method’s usage.Source: §5.3 · pp. 14–15, Table 5 · p. 15

6 EVALUATION STUDY

On PixelHelp and UGIF, AutoTask significantly outperformed the AITW baseline, and accumulated knowledge improved completion and efficiency on Type-2 and Type-3 tasks. The evaluation used Android tasks and metrics for success, step accuracy, and redundancy.

6 EVALUATION STUDY

  • The study tested execution without predefined knowledge and knowledge gains on Android 11; revised PixelHelp retained 67 commands and Settings-focused UGIF retained 100 tasks.PixelHelp and UGIF commands were revised into command form and supplemented with missing parameters.Source: §6 · p. 16, §6.1 · p. 16, §6.2 · p. 16

6.3 Metrics

  • Metrics covered task success, correct-step proximity, redundant steps, and completion without redundancy; completed tasks always receive step accuracy of 1.Step accuracy uses the longest correct subsequence that begins the minimum steps needed to complete the task.Source: §6.3 · p. 16

6.4 Baseline

  • AITW lacked explicit self-checking, backtracking, and knowledge accumulation; both systems processed shuffled commands, while knowledge transfer was tested using 20%–100% of Type-A tasks.Knowledge-transfer results averaged ten random selections at each percentage.Source: §6.4 · p. 17, §6.5 · p. 17

6.6.1 Success rate.

  • Completion detection reached 99.4% precision and 100% recall; AutoTask step accuracy was 93.5% versus the baseline’s 82.9%.The baseline’s completion-check recall was relatively low, with 36.1% of its errors attributed to overshoot.Source: §6.6.1 · p. 17

6.6.3 Performance improvement through knowledge accumulation.

  • With all knowledge, Type-3 success rate and step accuracy reached 85.7% and 97.6%; Type-2 and Type-3 redundancy fell from 46.7% to 16.1% and 97.1% to 4.76%.Only one Type-3 task remained incomplete; three tasks still required backtracking after all available knowledge was acquired.Source: Figure 6 · p. 18, §6.6.3 · p. 19, Figure 7 · p. 19

6.6.1 Success rate.

  • In experiments, AutoTask failed on three tasks by misgrounding GUI instructions, nine by pursuing irrelevant tasks prompted by command information, and once by treating “show system applications” as complete.Source: §6.6.1 · p. 18

6.6.2 Redundancy.

  • AutoTask’s step redundancy rates were 8.54% and 8.96%, versus 46.0% and 32.0% for the baseline; it backtracked in 12 tasks, averaging 2.83 backtracks there.The two baseline comparisons were significant at p < 0.001.Source: §6.6.2 · p. 18

6.6 Results

  • 91.2% vs 52.2% accuracy on PixelHelp and 93.0% vs 67.0% on UGIF; AutoTask significantly outperformed the baseline on both datasets (chi-square p < 0.001).The evaluation-results table covers the baseline and AutoTask on both datasets.Source: §6.6.1 · p. 17, Table 6 · p. 18

7 DISCUSSION

AutoTask’s accumulated knowledge can transfer across versions or similar applications, though inaccurate generalized knowledge may reduce efficiency. The discussion also describes possible extension to other interfaces and tasks.

7.1 Generalization of Knowledge: Across Versions and Applications

  • Knowledge can transfer across application versions or similar apps, but inaccurate generalization can reduce efficiency when functionality moves, even if backtracking still completes the task.The paper illustrates this with contact import moving from Settings in one Contacts version to another location in a later version.Source: §7.1 · p. 20

7.2 The Generalization of AutoTask: to Other Devices and Tasks

  • With APIs for retrieving GUI contents and simulating interactions, AutoTask may generalize to other platforms; its explore-learn strategy is also proposed for unknown tasks in unknown environments.Examples include web browsers, command-line interfaces, prompt generation, and embodied intelligence.Source: §7.2 · p. 21

8 LIMITATION & FUTURE WORK

AutoTask does not interact with users during execution, relies on a slow online LLM, and is scoped to tasks that can be handled as a single GUI task. Future work could add clarification, faster models, and task decomposition.

8 LIMITATION & FUTURE WORK

  • AutoTask does not interact with users during execution, though clarification could help resolve incomplete or ambiguous commands and guide GUI exploration.The paper gives asking whether App pinning is related to Security as an example of pruning exploration.Source: §8 · p. 21
  • AutoTask uses an online GPT-4 service over HTTPS, and its computational process is slow.The authors suggest smaller, faster models, potentially built through knowledge distillation, to reduce user waiting time.Source: §8 · p. 21
  • Complex intents that span multiple GUI tasks, such as retrieving a schedule and messaging it to Alice, remain outside a single-task scope; future work could add task decomposition.Source: §8 · p. 21

9 CONCLUSION

AutoTask explores GUI operation sequences and learns from those experiences to execute voice commands without configuration or modification. The evaluation reports better performance than a baseline without accumulated knowledge, with further improvement from knowledge accumulation.

9 CONCLUSION

  • AutoTask performs significantly better than the baseline when no knowledge is accumulated, and knowledge accumulation further improves its performance.Source: §9 · p. 21
  • AutoTask automates voice commands by simulating GUI interactions, exploring operation sequences and accumulating knowledge without developer or end-user configuration.Source: §9 · p. 21
  • AutoTask addresses the special case of accomplishing unknown tasks in unknown environments, a problem pattern the paper says applies to other similar scenarios.The authors hope this work inspires future efforts to apply general AI to everyday tasks with little effort from developers or end users.Source: §9 · pp. 21–22

Related papers

Loading 2312.16062v1…