VisionTasker: Mobile Task Automation Using Vision Based UI Understanding and LLM Task Planning

Motivation

• Smartphones support routine activities, but repetitive tasks and interactions can be challenging for people with sensory or motor limitations or for elderly users navigating complex menus.The introduction also identifies divided-attention situations such as driving as a context where smartphone use can be less accessible.Source: §1 · p. 1
• View-hierarchy methods can encounter inaccessible data, missing object descriptions, or misaligned structures, while PBD depends on previously demonstrated tasks and can be affected by app updates.Dataset-trained models also face challenges from noisy human trials and limited adaptability to new instructions and interfaces.Source: §1 · pp. 1–2
• The paper asks whether mobile tasks can be automated through sequential visual recognition and planning that resembles how people navigate unfamiliar interfaces.The motivating example breaks bill payment into successive UI choices.Source: §1 · p. 2

Approach

• VisionTasker converts screenshots into natural-language UI semantics and has an LLM choose the next action from the current interface, with PBD supporting complex planning.The authors present one interface at a time to keep decisions grounded in visible UI elements.Source: §1 · p. 2

Results

• Across four public datasets, VisionTasker provides effective UI representations, and on 147 Android tasks it shows advantages over humans on unfamiliar tasks.The paper also reports performance comparable to human abilities overall and significant improvement when combined with PBD.Source: §1 · p. 2

2 RELATED WORK

• Mobile task-automation research broadly uses recorded user actions, large datasets to link instructions with UI elements, or LLMs and multimodal LLMs to plan tasks.Source: §2 · p. 2

2.1 Programming by Demonstration

• PBD systems capture user actions as replayable scripts, but reliance on precise user-defined procedures limits flexibility when GUIs or workflows change.Examples include systems for conversational script creation, task-oriented bots, and parameterized macros.Source: §2.1 · p. 2

2.2 Understanding the Interfaces

• Interface-understanding work ranges from whole-screen representations to extracting attributes, structures, icons, or textual descriptions; dataset-trained models link UI elements and instructions but can be noisy and less adaptable to changed UIs.The cited work includes datasets with synthetic instruction pairs, text-image samples, screens with view hierarchies, and task sequences.Source: §2.2 · pp. 2–3

2.3 Automation with Large Language Models

• LLM automation systems use view hierarchies, app-specific knowledge, or image segmentation to identify UI elements, while multimodal LLMs combine UI understanding and planning but still need better element grounding.These approaches can rely on impractical UI sources or produce low-quality element identification.Source: §2.3 · p. 3

Workflow

• The agent reviews a new screenshot and prior actions after each interaction, repeating its planning and execution until the task finishes or an action limit is reached.Users can monitor visible progress and halt the process at any time.Source: §3 · p. 3

Motivation

• View hierarchies may be inaccessible or contain incomplete descriptions, misaligned structures, empty hierarchies, incorrect descriptions, or elements that do not exist.The examples are drawn from common apps, including Facebook, Uber, Spotify, Temu, Shopee, and Shein.Source: §3 · p. 3, Figure 2 · p. 4

Design rationale

• VisionTasker’s viability assumes that expressive UI design aligns with users’ mental models, allowing visual layout and element content to support human-like interpretation and LLM planning.Source: §3 · p. 3

System components

• The agent’s UI understanding, planning, and execution modules identify and group screen elements, produce low-level actions from task context, and translate those actions into operations such as taps at coordinates.The planner receives the task description, action history, and UI semantics.Source: §3 · pp. 3–4

4 UI UNDERSTANDING

• The UI understanding module examines screenshots to detect text, buttons, and widgets, then organizes them into meaningful clusters and outputs natural-language UI semantics.This screenshot-based approach does not depend on view hierarchies or accessibility services.Source: §4 · p. 4

4.1 Widget Detection

• Widget detection retrains YOLOv8 on UI datasets for 12 common classes and augments underrepresented widget types with 4,856 synthetic UI images.The synthetic additions target five less-represented types: ratio group, page indicator, spinner, seek bar, and rating bar.Source: §4.1 · p. 4

4.2 Text Recognition

• PaddleOCR detects text regions and recognizes their content, recording only text with confidence above 0.95.The system uses PP-LCNetV3 for detection and SVTR for recognition.Source: §4.2 · p. 4

4.3.1 Text-Widget Matching.

• Text-widget matching uses bounding-box proximity to pair widgets with nearby text, allowing multiple matches when alignment maintains continuity.For home-screen app icons, where only the widget is clickable, the method retains the widget alone.Source: §4.3.1 · p. 5

4.3.2 Unmatched Button Interpretation.

• For unmatched buttons lacking text, the method fine-tunes CLIP [40] to infer functions from visual design and match each button to a description of a common function.Fine-tuning uses IconSeer, a public dataset comprising over 180k buttons with text descriptions spanning more than 170 common categories.Source: §4.3.2 · p. 5

4.3.3 Semantic Block Division.

• Semantic block detection quantizes UI colors, detects edges, groups nearby similarly oriented line segments, and identifies rectangular areas from intersecting or nearly intersecting lines.The approach targets visually distinct UI sections despite variation in border orientation, length, and spacing.Source: §4.3.3 · pp. 5–6

4.3.4 Block Captioning and Active Tab Detection.

• Block captioning identifies left- or top-justified text without an associated image near a block’s top, helping supply context such as the month for date entries.The authors note that month labels separated from dates can otherwise lead LLM navigators to associate later dates incorrectly.Source: §4.3.4 · p. 6
• Active-tab detection first identifies the tab-bar block, then compares buttons’ hue distributions and marks the button with the greatest difference as selected.The method uses the HSV hue component to represent color variation as perceived by humans.Source: §4.3.4 · p. 6

5.1 Planning Step by Step

• A chain-of-screens loop uses the previous action’s output and the current UI to plan each next step, reducing confusion and restricting actions to available elements.The approach addresses LLM hallucinations of features absent from the current app.Source: §5.1 · p. 6
• Action history helps the LLM finish multiple actions on one interface and records failed paths to prevent repeated selection or useless loops.For example, history can preserve both language selection and the subsequent save action on the same page.Source: §5.1 · p. 6
• The prompt combines the user’s task, action history, and an ordered natural-language description of UI blocks and their elements, categories, functions, and IDs.High-level task-solution examples from PBD or help documentation are optional prompt content.Source: §5.1 · p. 6
• If the LLM names an element absent from the current UI, it is prompted to reassess; each next step begins a new dialogue to limit token growth.The history carries past actions forward rather than repeating each interface’s semantic description.Source: §5.1 · pp. 6–7

5.2 Programming by Demonstration

• When the LLM cannot plan a correct path, a user can execute the task while the UI understanding module translates touchscreen actions into semantic steps for the LLM.Unlike playback-only demonstrations, the semantic action descriptions are intended to accommodate app-layout changes and screen-resolution changes.Source: §5.2 · p. 7

6 EXECUTING MODULE

• Screenshot capture scrolls by half a screen, taking images before each scroll and stitching overlapping portions into a long view of the page.Scrolling continues to the bottom or until a specified limit, including for pages that load content continuously.Source: §6 · p. 7
• Command parsing checks whether the requested UI element exists and asks the LLM to reconsider when it is absent; valid elements supply bounding-box coordinates.Commands use action-object pairs such as tapping a button or entering text in a search bar.Source: §6 · p. 7
• Action execution taps the center of an element’s bounding box, adjusting off-screen coordinates with a swipe before converting operations into ADB commands.Executed commands are recorded in the action history for subsequent planning.Source: §6 · p. 7

7.1.1 Dataset.

• The UI-understanding evaluation used 136 screens from 31 apps and assessed element detection, icon-only button interpretation, and LLM question answering.Testing used recent Chinese apps while training used older English apps; the comparison included GPT-4V and a VH-based method.Source: §7.1.1 · p. 7, §7.1 · p. 7, §7.1.3 · p. 8

7.1.4 Results and Analysis.

• VisionTasker’s UI element detection significantly surpassed VH-based approaches, including when trained on English apps and tested on Chinese apps.The method could still produce false-positive detections.Source: §7.1.4 · p. 8
• For icon-only buttons, VisionTasker and GPT-4V exceeded 80% accuracy, while VH-based methods accurately described 63%.The paper attributes VH’s shortfall partly to missing or custom icon descriptions in app code.Source: §7.1.4 · pp. 8–9
• VisionTasker’s UI descriptions outperformed the other approaches overall for LLM question answering, though GPT-4V led on element-status queries.VisionTasker identifies status using cues such as toggles and checkboxes, but color comparisons struggle with complex tabs or navigation bars.Source: §7.1.4 · p. 9

7.2.3 Results and Analysis.

• 67% accuracy across four datasets—over 15% above baseline methods—was achieved in one-step prediction from the current interface.Observed errors included both UI understanding and task planning errors.Source: Figure 6 · p. 9, §7.2.3 · p. 10
• VisionTasker reduced missed detections, overdetections, and misreads compared with VH, while 21 fewer task-planning errors occurred than with HTML-based VH descriptions.The authors attribute planning support to natural-language descriptions that use semantic blocks to enrich element semantics.Source: §7.2.3 · p. 10

7.3.3 Results and Analysis.

• 47% of unfamiliar tasks were completed by VisionTasker, versus 26% by human evaluators; adding PBD prompts raised overall completion to 94%, above humans’ 87%.Without PBD, VisionTasker completed 76% of all 147 tasks, compared with 87% for humans.Source: §7.3.3 · p. 10, Figure 8 · p. 11

7.4 Impact of Programming By Demonstration

• PBD prompts based on demonstrations completed about 70% of previously failed tasks, with robustness to screen-size, resolution, and language variations.Among tasks still failing with PBD, the primary challenge shifted from task planning to UI understanding.Source: §7.4 · pp. 11–12, Figure 9 · p. 11

8.1 Latency and Token Cost

• In real-device experiments on Intel i9 10850k CPU and RTX 2080 Ti GPU, UI understanding averaged 5.9 seconds and LLM communication 4.5 seconds, yielding over 12 seconds per action.The passage says UI-understanding delays could be reduced by employing lighter models or knowledge distillation in place of Yolov8 and CLIP.Source: §8.1 · p. 12
• The UI description uses approximately a quarter of the tokens required by the other two methods, and action-history prompts keep added token use from longer task chains negligible.A 13-step task was completed; lighter models or knowledge distillation are suggested to reduce UI-understanding latency.Source: §8.1 · p. 12

8.2 Reading from Images

• Vision-based reading can detect task-relevant elements omitted by view hierarchies, while false-positive icons often do not affect subsequent tasks.Search boxes are especially likely to be missed when top-of-screen advertisements make them semitransparent.Source: §8.2 · p. 12

8.3 Adapting to Desktop Tasks

• Desktop adaptation may require desktop-specific training because buttons are smaller and more icon-dependent; highly specialized apps may remain unsuitable.The process and framework need not change, but models must handle the different interface characteristics and professional workflows.Source: §8.3 · p. 12

8.4 Utilizing Other LLMs

• Other LLMs, including open-source models, can be used for planning, though open-source models may need fine-tuning; extensive PBD demonstrations can compensate for some deficiencies.Source: §8.4 · p. 12

8.5 Limitations and Future Work

• Some screens, including login screens and payment QR-code scans, cannot be captured, and subtle visual distinctions can escape UI understanding.Fixed-interval screenshots are not optimal for games or slow-loading apps, and tasks with unspecified critical parameters may use defaults instead of prompting the user.Source: §8.5 · pp. 12–13

• VisionTasker combines screenshot-based natural-language UI understanding with step-by-step LLM planning, avoiding view hierarchies.Source: §9 · p. 13

A EXAMPLE DEMONSTRATION

• For the WeChat task of setting a status to “Happy,” the system describes each current UI in natural language and uses that description with action history to plan the next step.The plan adjusts to the current UI context until the task is completed.Source: A EXAMPLE DEMONSTRATION · p. 14

B ALGORITHM FOR BLOCK DIVISION

• The semantic-block algorithm detects edges, finds aligned neighboring pixels, fits line segments, then filters and groups them into rectangles.Its stated result is a set of semantic-block coordinates.Source: B ALGORITHM FOR BLOCK DIVISION · p. 16

• Figure 11 shows typical UI layouts for the 136 screenshots used in the first experiment.Source: C TYPICAL UI LAYOUTS · p. 16, Figure 11 · p. 16

D QUESTIONS EXAMPLES

• Table 3 presents sample questions for five UI question-answering categories used in the first experiment in Section 7.1.Source: D QUESTIONS EXAMPLES · p. 16, Table 3 · p. 16

E LIST OF 147 REAL TASKS

• Table 4 lists 147 real tasks from Section 7.3, including each source app, task category, description, and number of human evaluators able to complete it.Source: E LIST OF 147 REAL TASKS · p. 16, Table 4 · p. 17