Lexi: Self-Supervised Learning of the UI Language

1 Introduction

• Existing UI data collection either requires difficult-to-scale human labeling or depends on metadata that may be missing, incomplete, or inaccessible.Such metadata includes accessibility labels and structural representations such as DOM trees and Android View Hierarchies.Source: §1 · p. 1
• Instruction manuals, tutorials, and how-to guides offer crawlable text and visual content across apps and platforms for learning UI representations.The resources include screenshots and images of referenced icons.Source: §1 · p. 1
• UI elements can vary in appearance while retaining a function, and similar-looking elements can have different functions depending on context.UI screens are also text-rich, and general-domain language models may not represent technical instructions and UI terms adequately.Source: §1 · pp. 1–2
• Lexi adapts vision-language modeling to UI screens with specialized region detection, recognized-text features, and noun-verb POS tagging for functional captions.The model is evaluated on UI action entailment, instruction-based image retrieval, grounding referring expressions, and entity recognition.Source: §1 · p. 2
• The paper contributes a cross-platform dataset of 134k image-functional-caption pairs and a UI-language model that does not depend on UI metadata.It also evaluates Lexi with baseline comparisons, out-of-domain performance, and ablation analysis.Source: §1 · p. 2

2 Related work

• Prior vision-language models learn shared visual-language representations from paired datasets such as Conceptual Captions, SBU Captions, and MS COCO.Lexi brings this modeling approach to UI images paired with functional captions and adds an encoder designed for UI text richness.Source: §2 · p. 3
• Supervised UI systems use human demonstrations or labeled commands to ground language in UI elements and actions, with some methods requiring controlled environments or app-specific tasks.Concept-level action mapping seeks better generalization but requires labeled commands and an execution-environment graph.Source: §2 · p. 3
• Unsupervised systems train on Android screens but depend on UI trees or accessibility labels, whereas Lexi uses screenshots and web data across mobile and desktop platforms.Lexi does not require app-hosting infrastructure to collect that web data.Source: §2 · p. 3
• To the authors’ knowledge, prior work had not used instruction manuals to learn UI representation models; earlier uses studied recipes, physical commonsense, or game-agent strategy.The passage cites Yagcioglu et al. (2018), Bisk et al. (2020), and Branavan et al. (2011) for those other uses.Source: §2 · p. 3

3 UICaption dataset

• Images and accompanying text come from Microsoft and Zoom manuals, Android support websites, and 58k general technology websites.The collection process filters near-duplicates, blurred images, and content-unrelated images.Source: §3 · p. 3
• Each image receives one to five functional captions from surrounding webpage text, with alt-text appended when available.These captions describe UI functionality rather than image appearance, using surrounding context.Source: §3 · p. 3
• Instructions can link neighboring UI elements by function, so textual context can encode their spatial relationship.For example, a caption about selecting either a video-call or audio-call icon corresponds to the icons appearing next to each other.Source: §3 · p. 3
• UICaption contains 113,971 unique UI images and 133,817 image-caption pairs, with varied apps, platforms, element counts, text density, pixel density, and image quality.Images range from single icons to complete desktops.Source: §3 · p. 3

4 Method

• Lexi uses a two-stream vision-language architecture with image and text encoders and co-attentional layers for cross-modal representation learning.Detected text is infused early into corresponding visual features, and the model outputs representations for both streams.Source: §4 · p. 4
• The UI detector combines computer-vision-based box detection, ResNeXt-101 classification, and OCR, then merges predictions using box alignment and distance.It is designed for cross-platform, cross-app screens without requiring annotated UI screens.Source: §4 · pp. 4–5
• Each visual region feature is augmented with recognized-text features and spatial-location encoding before entering the visual stream.The encoder keeps up to 64 high-scoring boxes and combines visual, text, and location features.Source: §4 · p. 5
• The text encoder processes instructions and alt-text as tokens, joined using a special [ALT] token.Its encoder is initialized from RoBERTa and uses byte-level BPE tokenization.Source: §4 · p. 5
• Pre-training combines masked language modeling, masked region modeling with KL divergence, image-text alignment, and a new weakly supervised noun-and-verb POS task.The POS task uses spaCy predictions as labels and classifies tokens with a BIO scheme.Source: §4 · pp. 5–6

5 Evaluation on downstream tasks

• The evaluation covers action entailment, instruction-based image retrieval, referring-expression grounding, and UI entity recognition, with an ablation study of pre-training components.The entity-recognition task uses a named-entity scheme to identify action verbs, objects, and input parameters in instructions.Source: §5 · p. 6, §5.6 · p. 8

5.1 Experimental setup

• Pre-training uses UICaption splits with no overlap in UI images or functional captions, and evaluation includes zero-shot, fine-tuned, and OOD settings.The training split contains 103k images and 116k image-text pairs; Zoom and Teams are held out for OOD analysis.Source: §5.1 · p. 6, §5.2 · p. 6

5.2 Baseline models

• Baselines include ViLBERT variants using Lexi’s detector, with or without detected image text, and a multi-task ViLBERT model; metadata-dependent ActionBert and UIBert are not directly comparable.RoBERTa-UI is added as a text-only baseline for entity recognition.Source: §5.2 · p. 6

5.3 UI action entailment

• Lexi (FT) outperforms the strongest baseline by 24% on UI action entailment; zero-shot accuracy is 57.4%, and OOD accuracy is 39.6–40.1%, 16–19% above baselines.The OOD tests use Teams and Zoom, which were excluded from pre-training and fine-tuning.Source: §5.3 · p. 7

5.4 Instruction-based UI image retrieval

• Fine-tuned instruction-based retrieval achieves 1.9× higher R@1 than ViLBERT-12-1-UI in-domain, while OOD R@1 is 1.4–1.8 percentage points above the strongest baseline.Zero-shot performance is below fine-tuned baselines; increasing visual layers helps in-domain, while Lexi-6 performs better than Lexi-10 in some OOD cases.Source: §5.4 · p. 7

5.5 Grounding referring expressions

• For referring-expression grounding, Lexi outperforms the best baseline by 6.4%, although the task remains challenging and some predictions select the wrong matching element.The task selects a referenced UI element from detected screen elements; each expression is matched against an average of 20 candidates.Source: §5.5 · p. 8, Table 2 · p. 8

5.6 UI entity recognition

• On UI entity recognition, Lexi beats RoBERTa by 3.9% and ViLBERT-12-1-UI by 1.3% in F1; multimodal input improves performance over text-only input.The task extracts action verbs, objects, and input parameters from instructions; MS+Android contains 43.4k image-instruction pairs annotated across these entity types.Source: §5.6 · pp. 8–9, Table 2 · p. 8

5.7 Ablation analysis

• In ablations, removing ITA drops UI action entailment accuracy by 17.8%, while removing POS or MRM-KL drops entity-recognition F1 by 2.6%; detected text benefits action entailment.The analysis compares Lexi-6 variants with one pre-training task or detected image text removed.Source: §5.7 · p. 9, Table 3 · p. 9

6 Limitations

• Web data favors popular applications, while documentation and images may be sparse or lower quality for less popular apps and some categories.Instruction manuals are rare or missing for shopping, transportation, weather, and news apps.Source: §6 · p. 9
• Ensuring zero overlap between web-crawled pre-training data and downstream evaluation is difficult: aggregator sites may mix app manuals, and app names may be absent from page source.A URL-based filter does not eliminate this overlap risk.Source: §6 · p. 9
• Evaluation covers English instructions and UIs with English text, and instructions are limited to 512 tokens.Pre-training requires significant GPU resources, and the model is not optimized for edge devices such as mobile phones, potentially raising privacy concerns.Source: §6 · p. 9

7 Conclusion

• The paper proposes learning generic UI-language representations from instruction manuals and how-to guides.It builds a dataset based on functional captions, designs a vision encoder that leverages detected UI text, and creates a POS pre-training task.Source: §7 · p. 9
• Across four downstream tasks, Lexi largely outperforms the baselines, including a multi-task ViLBERT model trained on much larger datasets.The conclusion names UI screens’ text richness and context sensitivity as challenges addressed by the work.Source: §7 · p. 9
• The work applies prior vision-language models to UI data, where text richness and context sensitivity pose challenges.Source: §7 · p. 9

Ethical considerations

• Lexi could give visually impaired users access to more applications through screen readers when accessibility labels are missing or incomplete.The passage stresses unbiased application and platform coverage for this use.Source: Ethical considerations · p. 12
• Task automation is a potential use, but wrong-button actions raise questions about undo options and human review.The passage frames human-reviewable task completion as a frontier for systems intended to increase productivity.Source: Ethical considerations · p. 12
• The UI entity-recognition dataset combines four annotators’ Microsoft-app labels with adapted AndroidHowTo commands.It contains 43.4k image-instruction pairs and labels for verbs, objects, and inputs.Source: Ethical considerations · p. 12
• The authors released UICaption URLs and scripts for generating training samples.Source: Ethical considerations · p. 12

A UI data collection

• Custom crawlers extract manuals for 23 Microsoft products and Zoom, while a general crawler collects UI images, alt-text, and surrounding text from technical and how-to sites.The general crawler searches using queries derived from UI-element names and app names or from single-step PixelHelp instructions, then visits image-search results.Source: A UI data collection · p. 12
• The mixed crawl’s seed queries were specific to Microsoft apps and PixelPhone but reached other apps and platforms, including Apple and web interfaces.The passages attribute this reach to queries general enough to trigger results across apps and platforms.Source: A UI data collection · pp. 12–13
• Caption generation uses surrounding webpage text, splitting numbered instructions into steps and producing one to five captions per image.Reported average lengths are 32 words for captions, 4.5 for alt-text, and 29 for instruction text.Source: A UI data collection · p. 13
• UICaption contains 134k image-caption pairs across Mixed, Android, MS, Teams, and Zoom portions; Teams and Zoom are reserved for testing in out-of-domain analysis.The MS, Teams, and Zoom portion is described as having the highest image-text alignment and instruction-detail quality.Source: A UI data collection · p. 13

B Lexi UI element detector

• The Lexi detector combines contour-based boxes with OCR text boxes, then classifies detected regions into nine UI-element classes.Its output includes bounding boxes, class labels, probability vectors, and recognized text when available.Source: B Lexi UI element detector · p. 13

• Lexi uses 512-token text inputs, truncates alt-text to 64 tokens, and represents image regions with 2048-dimensional features.Source: C Additional training details · p. 13
• Pretraining loss coefficients are α=0.01, β=0.8, γ=0.15, and δ=0.2.Source: C Additional training details · p. 13
• The method discards recognized-text boxes when the recognized text is shorter than 2 characters and the confidence is lower than 0.7, so the stated rule requires both conditions.Source: C Additional training details · p. 13
• Table 4 reports dataset sizes, effective batch sizes, learning rates, and maximum epochs; Zoom and Teams data are used only for out-of-domain testing.Source: Table 4 · p. 14

• Lexi selects the correct image-instruction pair in examples with overlapping text or controls, including call-pad and “Join” distractors.Source: D Analysis of UI action entailment… · p. 14
• In one example, Lexi selects a participant-menu screen that lacks the “more” button specified by the instruction, despite a correct alternative.Source: D Analysis of UI action entailment… · p. 14
• The hardest example pairs one UI screen with instructions describing the same functionality through a download icon or an attendance button.Source: D Analysis of UI action entailment… · p. 14

E Labeling data for UI entity recognition

• Three annotators labeled instruction spans as UI action verbs, UI action objects, or input parameters, with a fourth consulted when they disagreed.Annotators received entity definitions and examples, including ambiguous cases.Source: E Labeling data for UI entity… · pp. 14–15

F Adapting AndroidHowTo to UI entity recognition

• The AndroidHowTo adaptation turns phrase-tuple sequence generation into multimodal entity extraction, tagging verbs and objects in instructions.The example tags “tap” as an action verb and “A” and “B” as action objects.Source: F Adapting AndroidHowTo to UI entity… · pp. 15–16