ILuvUI: Instruction-tuned LangUage-Vision modeling of UIs from Machine Conversations

1 INTRODUCTION

• UI layouts, visual styles, text, and stateful elements make understanding and conveying interface information through language challenging.The paper also notes that UI elements can encode more than their interactive affordances alone.Source: §1 · p. 2
• VLM performance on UI tasks falls short of performance on natural images because training data lacks UI examples.Text-only UI descriptions also omit the interface’s visual information.Source: §1 · p. 2
• The generated phrases cover single-step Q&A, detailed descriptions, available actions, action outcomes, goal-based element selection, and goal-based plans.The recipe is intended to work with existing UI datasets, including datasets without textual descriptions.Source: §1 · p. 2
• Fine-tuning LLaVA [40] produces ILuvUI, which describes UI elements and screens, provides contextual help, and plans multi-step interactions.The paper benchmarks element detection, compares response quality with a baseline VLM, and presents planning and reasoning examples.Source: §1 · pp. 2–3
• The paper adapts LLaVA [40] to generate paired text-image training data from UI screenshots without human-provided captions or annotations.The method combines a UI element detector with an LLM and produces six kinds of text pairs.Source: §1 · pp. 2–3

2.1 UI Understanding

• UI-understanding applications include interface adaptation, GUI testing, data-driven GUI search [10], prototyping, code generation, and security.Other approaches address hierarchical groupings, UI code generation, platform migration, and interface improvement.Source: §2.1 · p. 3
• Pixel-based UI research includes layout description and skeleton generation, widget detection, tappability prediction, and methods combining image processing with learned models.The paper uses Zhang et al.’s [69] method to derive textual UI representations for its data-generation process.Source: §2.1 · p. 3

2.2 Large Language Models and Vision Language Models

• The paper bases its work on LLaVA [40], a model combining CLIP and Vicuna, while also mentioning generated high-quality instruction-following examples to enhance LLM alignment.Source: §2.2 · p. 3
• Spotlight [36] takes a UI image and a specified region of interest, then outputs text about that region for tasks such as widget captioning.The paper contrasts this input requirement with ILuvUI, which does not require a region of interest and accepts a text prompt with the UI image.Source: §2.2 · pp. 3–4

3 GENERATING MULTIMODAL DATA FOR USER INTERFACES

• Because general-purpose VLMs have limited UI-task performance and lack latent knowledge of UI design principles and tasks, the paper adapts LLaVA [40] to UI data.The underlying UI datasets provide screenshots and bounding-box annotations but lack human-annotated captions.Source: §3 · p. 4
• The generated dataset contains 353K language-image instruction-following samples, including 224K conversations and five other sample categories.The other categories are concise description, detailed description, logical reasoning, potential actions, and UI transition.Source: §3 · p. 5

3.1 Detecting UI Elements and Generating Screen Captions

• A UI detector supplies element lists, bounding boxes, and types; the authors use these detections in prompts to generate screen captions.They use an object-detection model rather than existing UI bounding-box annotations to support use with unannotated UI examples.Source: §3 · p. 4, §3.1 · p. 5

3.2 Generating LLaVA-Style Data

• LLaVA-style data includes detailed UI descriptions and single-step Q&A conditioned on a screenshot, with questions about element attributes, placement, functionality, and purpose.Only questions with definite answers are retained, and one question is randomly selected for each detailed-description example.Source: §3.2 · p. 5

3.3 Generating UI-Specific Data

• The paper replaces generic complex reasoning with four UI-specific tasks: listing possible actions, predicting action outcomes, selecting an element for a goal, and planning toward a goal.The tasks are designed to familiarize the VLM with UI-component capabilities and common design-pattern affordances.Source: §3.3 · pp. 5–6

3.3.1 Listing Available UI Actions.

• For listing available actions, the LLM receives a screen caption and detected bounding boxes and proposes actions such as tapping, swiping, or entering text.This task uses a zero-shot LLM response.Source: §3.3.1 · p. 6

3.3.2 Predicting the Outcome of UI Actions.

• For action-outcome examples, the authors match a tap coordinate to the smallest containing detected element and condition the LLM on that element and the resulting screen caption.The paired interaction traces provide the tapped element and destination screen, grounding the generated answer in the transition.Source: §3.3.2 · p. 6

3.3.3 Selecting a UI Element Given a Goal.; 3.3.4 Goal-Based Planning.

• Goal-based element selection asks which element reaches a target view, while goal-based planning asks for directions that may change the current view or span multiple views.The planning examples are often multi-step; the paper uses two few-shot examples for this task.Source: §3.3.3 · p. 6, §3.3.4 · p. 6

4 VISION LANGUAGE MODEL

• ILuvUI uses a CLIP visual encoder, a feature-alignment projection, and a Vicuna-13B language decoder, following LLaVA’s design.UI images are padded before resizing to the encoder’s input size.Source: §4 · pp. 7–8
• Resizing UI screenshots to 224×224 and a square shape loses information and distorts intricate UI details, impairing interpretation of UI elements and text.Source: §4 · pp. 7–8
• The 336×336 input improved performance compared with the smaller image size.The visual encoder is CLIP-L-336px.Source: §4 · p. 8

Listing Available UI Actions

• Available actions include tapping buttons, icons, text, and tabs, including Listen Now, Browse, Library, and Search.Source: Listing Available UI Actions · p. 9

Goal-Based Planning

• The language decoder generates a response from concatenated projected visual features and the instruction’s word embeddings.A trainable projection aligns visual features with the language model’s embedding dimension.Source: Goal-Based Planning · p. 9

4.1 Data Preparation

• Training examples arrange each UI image and its question–answer pairs into a sequence, with the first turn randomly ordering the image and question.Subsequent turns use each question as the user instruction and its answer as the assistant response.Source: §4.1 · p. 9

4.2 Fine-tuning Process

• Fine-tuning uses LLaVA’s pretrained visual encoder and projection with a Vicuna-13b-v1.3 decoder, freezing the encoder while updating the projection and LLM weights.Source: §4.2 · pp. 9–10

5 EVALUATION

• The evaluation sampled 100 previously unseen UI images and compared ILuvUI with LLaVA using a matched 336-pixel-square vision encoder and language decoder.The authors identify further in-depth analyses as future work.Source: §5 · p. 11

5.1 UI Understanding Tasks

• 67.94% accuracy: ILuvUI outperformed LLaVA on UI-element detection.Source: §5.1 · p. 11
• 25.67% versus 9.03% accuracy: ILuvUI exceeded LLaVA on identifying UI-element types from 12 categories.The evaluation sampled five elements per UI and randomized the category list for each query.Source: §5.1 · p. 11

5.2 Human Evaluation of UI Descriptions

• 72% preferred ILuvUI’s screen descriptions, compared with 20% who preferred LLaVA’s.All authors served as raters, and the two descriptions were presented in randomized order.Source: §5.2 · pp. 11–12

6 DISCUSSION AND FUTURE WORK

• ILuvUI demonstrates UI-centric instruction-tuned behavior using synthetic textual data produced from a large UI dataset.Source: §6 · p. 12
• Potential uses include accessible screen descriptions and contextual help; the authors report informal success on some tasks and show navigation in Table 4.Source: §6 · p. 12
• Evaluation begins with basic detection and element-type identification, while broader tasks include widget captioning, screen summarization, command grounding, and tappability prediction [36].Source: §6 · p. 12
• The authors call for an agreed benchmark set that is easy to evaluate automatically, alongside more comprehensive evaluation of model performance.Source: §6 · p. 12
• Future work also includes refining reasoning, expanding UI-design knowledge, developing agents for interaction and navigation, and testing robustness across broader UI variations.Source: §6 · p. 12
• Generated-data quality and diversity need improvement and deeper analysis; proposed directions include reducing hallucination, adding annotations, and varying UI domains.Source: §6 · p. 12
• Model performance could improve with more advanced VLM architectures, especially variable-resolution, high-resolution encoders that better accommodate traditional desktop interfaces.Source: §6 · p. 12
• UI navigation and software testing need machine-interpretable outputs such as JSON, requiring additional training and fine-tuning for reliability.Source: §6 · p. 12

• The proposed data-generation and training approach builds on LLaVA to adapt VLMs to UI tasks involving interactive complexity.The paper notes that UIs include interactive components with structural and functional complexity.Source: §7 · p. 15
• The approach covers responses from individual UI element properties to descriptions, reasoning, potential user actions, and dynamic UI transitions.Source: §7 · p. 15
• The evaluation highlights ILuvUI’s effectiveness compared with existing models that are not trained on UI data.Source: §7 · p. 15