PaperlayerView exact revision on arXiv

Source-linked AI summary

Enabling Conversational Interaction with Mobile UI using Large Language Models

Bryan Wang, Gang Li, Yang Li

arXiv:2209.08655v2cs.HCcs.AI

TL;DR

Mobile-UI conversational interaction typically demands task-specific datasets and models, and the feasibility of adapting language-trained LLMs to GUIs was unclear. The paper converts mobile UIs to HTML and applies few-shot prompting across four tasks, achieving competitive performance with two examples per task and supporting rapid prototyping. Its evaluation also identifies limitations from unused visual modalities and limited steerability and reliability.

  • Problem

    Existing assistants lack computational GUI understanding, while prior mobile-UI interaction methods address limited aspects and require considerable dedicated data and training.

  • Method

    The paper represents Android view hierarchies as HTML and prompts a single LLM with task exemplars to perform four mobile-UI conversational tasks.

  • Results

    The approach achieves competitive performance across four tasks using only two data examples per task, with LLM summaries judged more accurate than Screen2Words on 64.1% of screens by majority vote.

  • Takeaways & Limitations

    The results support rapidly realizing novel language-based mobile interactions without traditional pipelines of expensive data collection and model training.

  • Takeaways & Limitations

    The study uses only view-hierarchy information, lacks direct controls for steering LLM behavior, and may produce hallucinated or irrelevant text.

Abstract

from arXiv · show

Conversational agents show the promise to allow users to interact with mobile devices using language. However, to perform diverse UI tasks with natural language, developers typically need to create separate datasets and models for each specific task, which is expensive and effort-consuming. Recently, pre-trained large language models (LLMs) have been shown capable of generalizing to various downstream tasks when prompted with a handful of examples from the target task. This paper investigates the feasibility of enabling versatile conversational interactions with mobile UIs using a single LLM. We designed prompting techniques to adapt an LLM to mobile UIs. We experimented with four important modeling tasks that address various scenarios in conversational interaction. Our method achieved competitive performance on these challenging tasks without requiring dedicated datasets and training, offering a lightweight and generalizable approach to enable language-based mobile interaction.

1 INTRODUCTION

The paper addresses the difficulty of supporting diverse conversational interactions on mobile UIs, where existing approaches require dedicated datasets and models. It adapts a single LLM through prompting and evaluates four tasks, achieving competitive performance with only two examples per task.

  • Natural-language interaction can improve access to computing systems, especially when other input or output modalities are unavailable or cumbersome.The motivation includes users with motor, visual, or situational impairments.
  • Existing conversational assistants remain limited on mobile UIs because they lack computational understanding of graphical user interfaces.For example, they cannot answer questions about specific information displayed on a screen.
  • Prior mobile-UI interaction components address limited aspects of conversation and require considerable dataset curation and dedicated model training.The paper argues that the broad range of possible mobile-UI conversations motivates a lightweight, generalizable approach.
  • The paper represents Android view hierarchies as HTML and uses prompting techniques to adapt an LLM to mobile UIs.The method is evaluated on Screen Question-Generation, Screen Summarization, Screen Question-Answering, and Mapping Instruction to UI Action.
  • Using only two data examples per task, the approach achieves competitive performance across four modeling tasks.The work also establishes benchmark performances for Screen Question-Generation and Screen Question-Answering.
  • Human evaluation finds the generated screen summaries more accurate than Screen2Words, a benchmark trained with tens of thousands of examples.The study concludes that the suite of evaluations supports the feasibility and effectiveness of the approach.
  • The approach could let interaction designers and developers prototype and test novel language interactions before investing in dedicated datasets and models.The paper presents this as a potential transformation of conversational interaction design workflow.

2 RELATED WORK

Related work connects GUI understanding, natural-language interaction, and LLM-based interactive applications. The paper positions its contribution as applying prompting to versatile conversational interaction with mobile UIs.

  • Widget Captioning and Screen Recognition predict meaningful alt-text labels for GUI components, while Screen2Words generates summaries of entire screens.
  • Other systems use conversational or multimodal agents to help users accomplish mobile tasks through demonstrations or developer-created bot templates.Examples include SUGILITE and KITE.
  • Prompting offers an alternative to finetuning by using a few examples to adapt pretrained models to downstream tasks without updating model parameters.
  • LLMs have also supported HCI applications such as story ideation and language-based web-design modification.The related applications use language models to support control, sensemaking, or interface changes.
  • Figure 1 categorizes mobile conversation scenarios using initiative and purpose, providing a framework for selecting interaction tasks.

3 CONVERSATION FOR MOBILE UI TASKS

The paper frames mobile-UI conversation as exchanges between users and agents organized by who initiates the interaction and whether information is solicited or provided. Four task types represent these scenarios.

  • The framework uses two dimensions—Initiative and Purpose—to categorize conversations as user- or agent-initiated and information-soliciting or information-providing.
  • Agents may initiate conversations by soliciting or providing information needed for users to complete mobile tasks.
  • Mobile UIs commonly request goal-relevant information through input fields, motivating natural-language questions in Screen Question-Generation.Examples include asking for a destination city, hotel, or check-in date.
  • Screen Summarization asks an agent to describe the purpose of the current screen in a short textual form.A representative example describes a list of hotel rooms available at a specific hotel.
  • Users can initiate conversations by requesting information or proactively providing information for the agent to process.
  • Screen Question-Answering responds to questions about information displayed on a mobile screen.It is useful for locating specific information in text-heavy screens and for users who rely on screen readers.
  • Mapping Instruction to UI Action requires interpreting a user’s instruction in the current UI context and responding with language or mobile actions.

4 PROMPTING LARGE-LANGUAGE MODELS FOR MOBILE UI TASKS

The prompting design adapts text-only LLMs to multimodal mobile UIs by converting view hierarchies into compact HTML and supplying task exemplars. It also supports reasoning and element reference through structured prompts and indexed elements.

  • 4 PROMPTING LARGE-LANGUAGE MODELS FOR MOBILE UI TASKS: Mobile-UI prompting must address how to represent multimodal interfaces as text and how to elicit reasoning when needed.
  • 4 PROMPTING LARGE-LANGUAGE MODELS FOR MOBILE UI TASKS: The paper presents prompting techniques as a strong baseline for adapting LLMs to mobile UIs without finetuning or retraining.
  • 4.1 Screen Representation: The screen representation converts Android view hierarchies into HTML to preserve UI-element properties and structural relationships in a language-model-compatible form.
  • 4.1.2 View Hierarchy Properties.: The representation limits retained view-hierarchy properties because including all properties can exceed model input limits.It focuses on properties related to textual descriptions of elements.
  • 4.1.2 View Hierarchy Properties.: The selected element properties include class, visible text, resource identifiers, and accessibility content descriptions.Content descriptions function as alt-text for UI elements.
  • 4.1.2 View Hierarchy Properties.: Heuristics map common Android classes to functionally similar HTML tags, including TextView to <p>, buttons to <button>, images to <img>, and text inputs to <input>.
  • 4.1.4 Text, Resource_Id, and Content Description.: Resource identifiers contribute descriptive purpose information, while depth-first-search indexes let the model reference specific UI elements compactly.
  • 4.1.4 Text, Resource_Id, and Content Description.: A prompt contains a preamble, zero or more task exemplars, and a test screen represented as HTML; the LLM generates the task output autoregressively.Exemplars may include screen HTML, chain-of-thought reasoning, and task-specific outputs.

4.2 Chain-of-Thought Prompting

Chain-of-Thought prompting adds intermediate reasoning steps to few-shot mobile-UI prompts, while prompt structure controls how screens and exemplars are represented within input limits.

  • 4.2 Chain-of-Thought Prompting: LLMs’ reasoning ability can be improved by chaining intermediate results before producing final answers.The approach appends intermediate thoughts to the prompt so the model follows that pattern during inference.
  • 4.2 Chain-of-Thought Prompting: Chain-of-Thought prompting is presented as a method for making reasoning-task outputs more interpretable.The model articulates its thought process before giving the answer.
  • 4.3 Prompt Structure: Each prompt begins with a preamble explaining its purpose, followed by exemplars containing inputs, optional reasoning, and outputs.The structure follows a prompt format proposed in prior work and is adapted for each task.
  • 4.3 Prompt Structure: Exemplar inputs represent mobile screens in HTML syntax rather than raw visual screens.The HTML representation allows the language model to process UI content as text.
  • 4.3 Prompt Structure: Only visible leaf nodes are included to remain within LLM input-length limits.Non-leaf nodes are usually containers without textual information, so they are omitted from the prompt.
  • 4.3 Prompt Structure: For applicable tasks, prompts place a chain of thoughts between each screen input and its desired output.Outputs include task outcomes such as screen summaries or answers to user questions.
  • 4.3 Prompt Structure: During prediction, a new input screen is appended after the preamble and N exemplars to form an N-shot prompt.The same structure supports one-shot and few-shot prediction by varying the number of exemplars.

5 FEASIBILITY EXPERIMENTS

The feasibility experiments evaluate whether few-shot prompting can support four mobile-UI conversational modeling tasks with PaLM.

  • Experimental scope: The experiments cover Screen Question-Generation, Screen Summarization, Screen Question-Answering, and Mapping Instruction to UI Action.These four tasks represent different conversational interactions with mobile UIs.
  • Experimental scope: Few-shot prompts are constructed from a handful of exemplar data for each task.This follows common few-shot prompting practice.
  • Evaluation: Task-specific metrics are used to evaluate the effectiveness of the prompting approach.The evaluation criteria are detailed separately for each experiment.

5.1 Screen Question-Generation

The screen question-generation experiment tests whether an LLM can produce coherent, relevant questions for mobile-UI input elements using screen context and few-shot prompting.

  • Task definition: Screen Question-Generation asks the model to synthesize grammatically correct questions relevant to UI elements requiring user input.The task occurs when an agent requests user input to proceed on the interface.
  • Prompt construction: The prompt elicits three intermediate results: input-field count, screen summary, and input enumeration.The generated questions are enclosed by <SOQ> and <EOQ> tokens for convenient parsing.
  • Experimental setup: The experiment evaluates 400 RICO screens, each containing at least one EditText input field.Human ratings assess grammar correctness and relevance to the highlighted input field.
  • Experimental setup: The rule-based baseline fills the template “What is {res_tokens}?” using words from each field’s resource_id.resource_id is used instead of field text because text inputs are usually blank by default.
  • Results: 4.98 average grammar correctness was achieved by the LLM, versus 3.6 for the rule-based approach.The difference was statistically significant according to a Mann–Whitney U test with p < 0.0001.
  • Results: 95.9% F1 question coverage was achieved by the LLM, with 95.4% precision and 96.3% recall.The rule-based method had naturally 100% coverage because it generated a question for every input field.
  • Results: The LLM generated 8.7% more relevant questions than the baseline.The analysis also found that the model used both the target input field and broader screen context when generating questions.

5.2 Screen Summarization

The study evaluates few-shot LLM screen summarization against Screen2Words, finding that human annotators generally preferred LLM summaries despite weaker token-overlap metrics.

  • Screen2Words provides human-labeled summaries for more than 24k screens, while evaluation uses a 4310-screen test set from 1254 apps.
  • Two randomly sampled screens and one summary were used as exemplars for constructing prompts.
  • Zero-shot prompting produced no meaningful summaries, whereas one exemplar significantly improved performance across all automatic metrics.
  • Two-shot LLM summaries averaged 7.15 words versus 6.64 for Screen2Words and used 3062 unique words versus 645.
  • Human evaluation rated LLM summaries more accurate in 63.5% of annotations, compared with 28.6% for Screen2Words.
  • LLM summaries were preferred on 64.1% of screens by majority vote and unanimously on 52.8%, but Screen2Words was unanimously preferred on 20.5%.
  • The approach may perform poorly on screens without text because the language model lacks visual information.
  • Examples show that the LLM uses specific screen text, combines multiple elements, and can infer London Tube context from station names.

5.3 Screen Question-Answering (QA)

The screen QA experiment tests whether an LLM can answer factual questions from mobile UI content using few-shot prompts and generated answers.

  • The task requires answering open-ended factual questions using information presented on a mobile UI screen.
  • Prompts contain screen HTML and question-answer exemplars under zero-, one-, or two-shot settings, without chain-of-thought prompting.
  • The evaluation uses 300 human-labeled question-answer pairs from 121 RICO screens.
  • Because generated answers may use equivalent aliases, evaluation reports Exact Match, Contains GT, Sub-String of GT, and Micro-F1 against DistilBERT.
  • One-shot prompting achieved 65.8% Exact Match and 83.6% relevant answers, while Micro-F1 reached 62.9% versus 37.2% for the baseline.
  • In zero-shot evaluation, the LLM achieved 30.7% Exact Match, 6.5% Contains GT, and 5.6% Sub-String of GT, slightly below DistilBERT on the reported metrics.
  • Examples indicate that generated answers can include additional context, whereas the baseline sometimes retrieves HTML code from the screen.

5.4 Mapping Instruction to UI Action

The mapping task evaluates whether few-shot prompting can translate natural-language UI instructions into target element IDs under in-app and cross-app conditions.

  • 5.4 Mapping Instruction to UI Action: The task predicts the UI element ID needed to perform a natural-language instruction, supporting language-based control of mobile apps.
  • 5.4.2 Prompt Construction.: Prompts pair screen HTML and instructions with ground-truth IDs across zero-, one-, and two-shot settings, without chain-of-thought prompting.
  • 5.4.2 Prompt Construction.: PixelHelp contains 187 multi-step instructions for everyday tasks on Google Pixel phones, with prompt modules sampled by app package.
  • 5.4.3 Experiment Setup.: Experiments compare in-app prompts containing a module from the test screen’s app package with cross-app prompts that do not.
  • 5.4.3 Experiment Setup.: Performance is measured using partial and complete matches of target element sequences.
  • 5.4.4 Results.: Zero-shot prompting produced nearly zero partial or complete accuracy, while cross-app one-shot prompting achieved 74.69 partial and 31.67 complete accuracy.
  • 5.4.4 Results.: The best setting, two-shot LLM with in-app exemplars, achieved 80.36 partial and 45.0 complete accuracy.
  • 5.4.4 Results.: The approach underperformed Seq2Act but remained competitive using only two examples, without updating model parameters.

6 DISCUSSIONS AND FUTURE WORK

The discussion highlights rapid prototyping, accessibility, and broader UI applicability, while identifying limits in interaction scope, input representation, model reliability, and prompt length. Future work targets richer conversational settings, multimodal inputs, improved steerability, and efficient inference.

  • Implications: Prompting can make prototyping novel mobile-UI language interactions nearly as simple as designing a data exemplar.Designers can create functioning mock-ups and explore task possibilities before investing in datasets and models.
  • Implications: The approach may support accessibility and multimodal interaction alongside touch input and screen readers.The paper also identifies end-user prompting as a possible use case when agents fail to execute commands.
  • Findings: The feasibility experiments showed promising results across tasks, but mapping instructions to UI actions underperformed the benchmark.The approach was also reported to produce more accurate screen summaries than the benchmark in human evaluation.
  • Limitations and future work: The study is limited to single-turn, single-screen conversation tasks, leaving multi-screen and multi-turn interactions for future work.The paper illustrates that more complex tasks could span screens and require sequential interaction.
  • Limitations and future work: The first prompt exemplar usually provides the largest performance gain, while additional examples offer marginal improvement under input-length constraints.Future work could dynamically select prompt screens by length, although mismatched prompt and test lengths remain uncertain.
  • Limitations and future work: Using only view-hierarchy-derived HTML limits performance when important information appears solely in pixels, images, audio, or other modalities.This boundary is especially relevant to visually rich applications such as photo editors.
  • Limitations and future work: The method does not explicitly control question combination, prior-knowledge use, or hallucinated output, leaving steerability and reliability unresolved.The paper describes hallucinations as potentially incorrect or irrelevant generated information.
  • Generalization: The approach may extend to web, iOS, and macOS UIs with view-hierarchy data, but sophisticated interfaces remain difficult to represent within text and length constraints.Video editors are given as an example of a challenging UI type.

7 CONCLUSION

The paper finds that prompting a single LLM can support varied conversational interactions on mobile UIs with competitive performance and without expensive task-specific pipelines.

  • Conclusion: Prompting a single LLM enabled varied mobile-UI conversational interactions with competitive performance compared with expensive data-collection and model-training pipelines.The approach enables rapid realization of novel language-based interactions.

A EXAMPLE PROMPTS FOR FEASIBILITY EXPERIMENTS TASKS

The appendix illustrates task-specific prompt formats that represent mobile screens as structured text and map screen content or instructions to conversational outputs or UI actions.

  • A.1 Screen Question-Generation: The question-generation demonstrations use one or two exemplars, with intermediate reasoning and explicit question delimiters in the two-shot example.The examples cover password confirmation, hints, email addresses, refund amounts, and other fields.
  • A.1 Screen Question-Generation: Screen Question-Generation prompts ask the agent to identify input elements and generate corresponding user questions.The examples include reasoning about input counts, screen purpose, and requested fields.
  • A.1 Screen Question-Generation: Prompt outputs use structured delimiters and element identifiers to connect generated questions or predicted actions to specific screen elements.The examples include SOQ/EOQ delimiters for questions and an SOI/EOI-delimited prediction for an action target.
  • A.1 Screen Question-Generation: The demonstrations represent screens with HTML-like elements carrying identifiers, text, classes, and accessibility labels.These fields expose both visible content and element metadata to the prompt.
  • A.1 Screen Question-Generation: The question-generation example combines related fields, such as SSN components, into a single question when appropriate.The prompt output asks for the last four SSN digits using the relevant element identifier.
  • A.2 Screen Summarization: Screen Summarization prompts ask for a screen’s purpose and produce a concise textual summary from visible UI elements.The example uses contact-related labels and controls to generate a summary of contact settings options.
  • A.3 Screen Question-Answering: Screen Question-Answering prompts provide a mobile screen and a question, requiring an answer based on the screen information.The example includes a message field and a question about available email addresses.
  • A.4 Mapping Instruction to UI Action: Mapping Instruction to UI Action prompts provide a screen and instruction, then predict the identifier of the UI element that performs the instruction.The example maps opening the device’s Clock app to a predicted element identifier.
Loading 2209.08655v2…