Source-linked AI summary
Screen2Vec: Semantic Embedding of GUI Screens and GUI Components
Toby Jia-Jun Li, Lindsay Popowski, Tom M. Mitchell, Brad A. Myers
TL;DR
Existing GUI semantic representations capture limited aspects, although screens encode task flow, interactions, and design patterns useful to data-driven methods. Screen2Vec learns representations from textual, visual, layout, and app-context features using interaction traces, and is effective on downstream retrieval, composability, and mobile-task tasks.
Problem
Existing GUI representations are limited in the aspects they capture, while GUI screens encode task flow, interactions, and design patterns useful to data-driven methods.
Source: §1 · p. 1Method
Screen2Vec uses a self-supervised, two-layer pipeline informed by GUI structure and interaction traces to represent textual, visual, layout, and app-context features.
Source: §1 · p. 2Results
Screen2Vec is effective in downstream tasks including nearest-neighbor retrieval, composability-based retrieval, and representing mobile tasks.
Source: §7 · p. 13Takeaways & Limitations
The representations could support interactive task-learning systems, suggestive interfaces, assistive tools, and GUI design aids.
Source: §7 · p. 13Screen2Vec has only been trained and tested on Android GUIs, and the authors expect desktop GUI embedding to be more difficult.
Source: §5 · p. 11
Abstract
from arXiv · showhide
Representing the semantics of GUI screens and components is crucial to data-driven computational methods for modeling user-GUI interactions and mining GUI designs. Existing GUI semantic representations are limited to encoding either the textual content, the visual design and layout patterns, or the app contexts. Many representation techniques also require significant manual data annotation efforts. This paper presents Screen2Vec, a new self-supervised technique for generating representations in embedding vectors of GUI screens and components that encode all of the above GUI features without requiring manual annotation using the context of user interaction traces. Screen2Vec is inspired by the word embedding method Word2Vec, but uses a new two-layer pipeline informed by the structure of GUIs and interaction traces and incorporates screen- and app-specific metadata. Through several sample downstream tasks, we demonstrate Screen2Vec's key useful properties: representing between-screen similarity through nearest neighbors, composability, and capability to represent user tasks.
1 INTRODUCTION
GUI screens encode task flow, supported interactions, and app design patterns, but existing representations capture only parts of their semantics or require task-specific annotation. Screen2Vec combines textual content, visual design and layout patterns, and app metadata in self-supervised embeddings, with downstream uses including retrieval, composability, and task representation.
1 INTRODUCTION
- GUI screens encode task flow, supported interactions, and app design patterns, making them useful data sources for computational methods and applications.The paper identifies a lack of comprehensive GUI representations as a limitation across computational HCI research.Source: §1 · pp. 1–2
- Text-focused representations miss layout, design patterns, and task context, while visual approaches can miss screen content.Visually similar screens with different content may yield similar results.Source: §1 · pp. 1–2
- Approaches combining text and visual patterns use supervised, task-specific models that require substantial manual labeling and do not transfer across downstream tasks.The cited models are trained on large manually annotated corpora linking language utterances and GUI actions.Source: §1 · p. 2
- Screen2Vec generates self-supervised GUI screen and component embeddings from text, visual design and layout patterns, and app metadata.Its two-layer pipeline draws on GUI and interaction-trace structure and incorporates screen- and app-specific metadata.Source: §1 · p. 2
- Screen2Vec embeddings support nearest-neighbor retrieval, composability-based retrieval, and representing mobile tasks without manual data labeling.The model can be trained using GUI screens and user interaction traces, such as those in the Rico dataset.Source: §1 · p. 2
- The authors release Screen2Vec code and a Rico-trained model that encodes Android app screens off the shelf.The code can train models for other platforms when suitable interaction-trace datasets are available.Source: §1 · p. 2
2 OUR APPROACH
Screen2Vec learns component and screen embeddings from component and interaction-trace context, combining textual, class, layout, and app-description features. Its evaluation compares prediction performance across Screen2Vec variants and single-feature baselines.
2 OUR APPROACH
- The two-level pipeline learns component embeddings from component context and screen embeddings from screen sequences, combining component content with layout and app-description features.Source: §2 · pp. 2–3
2.1 Dataset
- Rico supplies traces and GUI hierarchies for 66,261 screens from 9,384 Android apps, including component text, classes, bounding boxes, and interaction transitions.Source: §2.1 · p. 3
2.2 Models
- Screen layouts are encoded by an autoencoder into 64-dimensional vectors; components map into 6-dimensional embeddings across 26 categories, and each component’s CBOW context comprises its 16 nearest components.Context distance can be measured as pixel-based Euclidean distance between component bounding boxes or as hierarchical-tree distance; direct siblings are distance 2.Source: §2.2 · p. 4
2.3 Training Configurations
- Component training sums text and class-type cross-entropy losses; screen prediction uses negative sampling that includes screens from the same trace to promote screen-specific learning.The data split uses 90% for training and 10% for validation.Source: §2.3 · pp. 4, 6
2.4 Baselines
- The comparison baselines isolate textual content, layout, or screenshot appearance rather than combining these screen features.Source: §2.4 · p. 6
2.5 Prediction Task Results
- Screen2Vec variants with spatial information outperform TextOnly, LayoutOnly, and VisualOnly baselines in top-1 and top-k screen prediction accuracy.Source: §2.5 · p. 6
- Spatial information especially improves Top-1 and Top-0.01% accuracy by distinguishing same-app screens with semantically similar text; HIERARCHICAL is worse than EUCLIDEAN in screen prediction.Source: §2.5 · pp. 6–7
- Baseline models outperform the no-spatial variant in normalized RMSE, although spatial Screen2Vec variants outperform baselines on both accuracy and similarity.Source: §2.5 · p. 7
3 SAMPLE DOWNSTREAM TASKS
The sample downstream tasks examine Screen2Vec embeddings through nearest-neighbor retrieval, vector composability, and mobile-task representation. Their results illustrate distinct uses and properties of the representations.
3.1 Nearest Neighbors
- The nearest-neighbor study sampled 50 familiar screens and had 79 Mechanical Turk workers rate 5,608 screen pairs on a 1–5 scale.Workers saw app descriptions but not which model produced each result.Source: §3.1 · p. 7
- For Lyft’s request-ride screen, Screen2Vec retrieved transportation screens with similar map-and-card layouts, while TextOnly favored payment-related text and LayoutOnly favored similar layouts.The Screen2Vec examples included trip-configuration screens from Uber Driver, Waze, and Free Now.Source: §3.1 · pp. 7, 9
- Screen2Vec’s nearest neighbors received higher mean similarity ratings than TextOnly and LayoutOnly, with both differences significant at p<0.0001.The ratings covered app, screen-type, and content similarity.Source: §3.1 · p. 7, Table 4 · p. 8
3.2 Embedding Composability
- A composite Marriott hotel-booking embedding adjusted by Cheapoair’s search-result-minus-hotel-booking embeddings retrieved Marriott’s search-result screen.Filtering out Marriott also returned list-result screens from other travel-related apps.Source: §3.2 · p. 10
3.3 Screen Embedding Sequences for Representing Mobile Tasks
- The task evaluation used 10 common smartphone tasks with two app variations each, representing each task by averaged screen embeddings rather than action or component context.The authors note that averaging does not preserve screen order.Source: §3.3 · pp. 10–11
- Screen2Vec identified cross-app variants of the same mobile task with 18/20 (90%) accuracy, versus 14/20 (70%) for TextOnly.The evaluation represented each task as the average of its screen embeddings and searched among 20 task variations.Source: §3.3 · p. 11
4 POTENTIAL APPLICATIONS
The paper describes potential uses of Screen2Vec embeddings in design aids, generative GUI models, and interactive task-learning systems.
4 POTENTIAL APPLICATIONS
- Designers could use nearest neighbors to find examples with similar content or app domains, and composable queries to retrieve designs conditioned on another page’s context.The proposed checkout query combines App A’s order page with the difference between App B’s checkout and order pages.Source: §4 · p. 11
- Screen2Vec embeddings could add GUI semantics and interaction context to generative models, including heuristics for component compatibility and screen fit within planned flows.Task embeddings from an existing app could also inform designs for the same task.Source: §4 · p. 11
- Task embeddings may help generalize a demonstrated procedure from one app to another in the same domain, a direction the authors plan to explore.The paper gives ordering coffee across Starbucks and Dunkin’ Donuts as an example.Source: §4 · p. 11
5 LIMITATIONS AND FUTURE WORK
Screen2Vec was trained and tested only on Android GUIs, and the authors identify gaps in Rico’s interaction traces and screens. The current model also omits semantics of graphic icons without text.
5 LIMITATIONS AND FUTURE WORK
- Screen2Vec has only been trained and tested on Android app GUIs; the authors expect desktop GUIs to be harder because of their complexity and heterogeneous layouts.They state the approach should apply to GUI apps with hierarchical structures, such as iOS view hierarchies and web DOM structures.Source: §5 · p. 11
- Rico contains only within-app interaction traces, and its missing categories of screens may affect pretrained performance on underrepresented app screens.The authors plan to evaluate cross-app traces with an expanded dataset.Source: §5 · pp. 11–12
- Screen2Vec does not encode the semantics of graphic icons that have no textual information.The authors suggest exploring computer-vision or crowd-based annotations and explicit GUI design-system rules and examples in future work.Source: §5 · p. 12
6 RELATED WORK
Related work spans distributed language representations, computational GUI modeling, and interactive task learning. Screen2Vec combines GUI content, layout, and app context in distributed representations, while offering a way to compare screens and tasks across apps.
6.1 Distributed Representations of Natural Language
- Distributed word embeddings represent words as continuous vectors that can capture syntactic and semantic relationships, with language-modeling approaches using local or corpus-wide contexts.ELMo and BERT allow phrase representations to vary with context.Source: §6.1 · p. 12
- Screen2Vec applies context-prediction embeddings to GUI entities and combines text and class-type vectors with app metadata and a hierarchy of screen components.The authors describe it as the first approach, to their knowledge, to encode GUI semantics, context, and design patterns using distributed representations; Word2Vec uses word contexts without such metadata.Source: §6.1 · p. 12
6.2 Modeling GUI Interactions
- Prior GUI interaction mining and design-semantic methods classify flows or components into discrete types, while Screen2Vec represents screens in a continuous vector space.Other visual approaches retrieve screens by layout or predict icon alt text; Screen2Vec additionally encodes textual content, component classes, and app-specific metadata.Source: §6.2 · p. 13
- Prior systems that map language commands to GUI actions use supervised learning and manual annotations, whereas Screen2Vec needs no annotations of user intents, tasks, or GUI screens.Source: §6.2 · p. 13
6.3 Interactive Task Learning
- Sugilite’s GUI graph supports grounding instructions but is difficult to aggregate or compare across screens and apps, limiting procedure generalization to the same app and screens.Source: §6.3 · p. 13
- Visual interactive-task systems support generalization by visual similarity but lack semantic understanding; Screen2Vec task sequences support cross-app task comparison and retrieval and may help transfer procedures.Source: §6.3 · p. 13
7 CONCLUSION
Screen2Vec generates self-supervised distributed representations of GUI screens and components from textual content, visual design and layout, and app metadata. The authors report effectiveness on retrieval and task-representation tasks, and identify potential uses in interactive systems and GUI design aids.
7 CONCLUSION
- Screen2Vec generates self-supervised distributed semantic representations of GUI screens and components using text, visual design and layout patterns, and app metadata.Source: §7 · p. 13
- The authors report Screen2Vec effective for nearest-neighbor retrieval, composability-based retrieval, and representing mobile tasks.Source: §7 · p. 13
- The authors suggest Screen2Vec could support task-learning systems, intelligent suggestive interfaces, assistive tools, and GUI design aids.Source: §7 · p. 13