Source-linked AI summary
Screen2Words: Automatic Mobile UI Summarization with Multimodal Learning
Bryan Wang, Gang Li, Xin Zhou, Zhourong Chen, Tovi Grossman, Yang Li
TL;DR
Mobile UI summarization must turn complex, multimodal screens into concise language that conveys important content and functionality. Screen2Words introduces a generative multimodal approach and a large human-annotated dataset, and its full model achieves the strongest reported automatic and human-evaluation results among tested methods. The authors note that the model can still produce generic or wrong summaries and that human ratings may not correlate well with automatic scores.
Problem
Existing UI representation and component-labeling methods do not adequately bridge multimodal screen understanding to concise natural-language descriptions of entire mobile screens.
Method
Screen2Words combines a Transformer for structural-textual UI information with a ResNet for screenshot pixels, trained and evaluated on human annotations.
Results
The full model outperformed heuristic baselines and deep model variants on automatic metrics and subjective human ratings.
Takeaways & Limitations
The dataset, models, and empirical results establish a benchmark for bridging user interfaces and natural language.
Takeaways & Limitations
The model is not always accurate, sometimes producing generic or wrong summaries, and human ratings may not correlate well with automatic scores.
Abstract
from arXiv · showhide
Mobile User Interface Summarization generates succinct language descriptions of mobile screens for conveying important contents and functionalities of the screen, which can be useful for many language-based application scenarios. We present Screen2Words, a novel screen summarization approach that automatically encapsulates essential information of a UI screen into a coherent language phrase. Summarizing mobile screens requires a holistic understanding of the multi-modal data of mobile UIs, including text, image, structures as well as UI semantics, motivating our multi-modal learning approach. We collected and analyzed a large-scale screen summarization dataset annotated by human workers. Our dataset contains more than 112k language summarization across $\sim$22k unique UI screens. We then experimented with a set of deep models with different configurations. Our evaluation of these models with both automatic accuracy metrics and human rating shows that our approach can generate high-quality summaries for mobile screens. We demonstrate potential use cases of Screen2Words and open-source our dataset and model to lay the foundations for further bridging language and user interfaces.
1 INTRODUCTION
Screen2Words frames mobile UI summarization as generating concise language descriptions for entire screens, requiring holistic multimodal understanding. It introduces a generative approach, a large human-annotated dataset, and evaluations showing benefits from multimodal deep models.
- Motivation: Screen summaries can help conversational agents, end users, and screen-reader users understand a mobile UI’s purpose and state.The paper motivates summaries as a flexible language representation for language-based interaction and accessibility.
- Research Gap: Mobile screen summarization requires jointly understanding textual, visual, structural, and semantic UI information rather than isolated components.Prior work encoded multimodal UI data or predicted labels for individual widgets, but did not summarize the entire screen.
- Approach: Screen2Words introduces a generative approach that encapsulates complex mobile UI information into succinct language descriptions.The task extends screen understanding and language generation to descriptive overviews of complete mobile screens.
- Evaluation: The full multimodal model outperformed heuristic baselines and model variants on automatic metrics and subjective human ratings.Its strongest configuration combines Transformer and ResNet encoders with text, image, and UI-structure inputs.
- Dataset: 112,085 human annotations cover 22,417 unique Android screens in the first dataset dedicated to UI screen summarization.The dataset was collected through a designed labeling process and supports data-driven model development.
2 RELATED WORK
Prior work covers content summarization, multimodal UI representation, and mobile UI datasets. Screen2Words extends this foundation from component-level understanding and retrieval toward language summaries of complete interfaces.
- Content Summarization: Automatic content summarization has been widely studied for text documents, images, and videos using learned content representations and generated short text.These areas provide methodological context for mobile UI summarization.
- Mobile Screen Understanding: Deep learning has been used to encode multimodal mobile UI representations for downstream tasks such as retrieval and interaction-related prediction.Examples include UI retrieval, sketch-based design retrieval, and tappability prediction.
- Mobile Screen Understanding: Existing mobile UI language work predicts alt-text labels for individual icons, widgets, or GUI components rather than complete-screen summaries.Screen2Words addresses the broader challenge of describing complex screen contents concisely.
- Screen2Words: Screen2Words extends mobile screen understanding by generating concise language descriptions for entire mobile graphical interfaces.This requires holistic screen understanding together with language-generation capability.
- Mobile UI Datasets: Large-scale mobile GUI repositories provide visual, textual, structural, and interactive data for data-driven model development.Rico contains 66k unique UI screens from 9.7k Android apps spanning 27 categories, while other datasets support interaction and retrieval research.
3 DATASET CREATION
The dataset is built from Android screens with screenshots and view hierarchies, annotated by multiple professional labelers under concise, functionality-focused guidelines. It also records focus areas and analyzes summary consistency and language statistics.
- Dataset Scope: 112,085 human-annotated English summaries were collected for 22,417 unique UI screens.The screens came from the Rico-SCA corpus, creating a foundation for data-driven screen summarization.
- Mobile UI Corpus: The mobile UI corpus pairs each screen’s screenshot with a view-hierarchy JSON representing UI elements and their properties.The corpus was filtered from Rico to remove screens with missing or inaccurate view hierarchies.
- Data Annotation: Five different labelers produced annotations for each screen, with 85 professional English-fluent labelers recruited for the task.Labelers saw the screenshot, instructions, and app description and could skip screens they did not understand.
- Annotation Guidelines: Annotations were guided toward 5–10-word, functionality-focused summaries using screen text and a “NOUN + CLAUSE” structure.Guidelines also required summarizing the whole screen rather than only images or icons and avoiding subjective descriptions.
- Focus Areas: Summary Focus Areas identify UI regions labelers considered most informative, supporting analysis of perceived summarization importance.The areas cover 66.1% of screens on average, with 74.1% average pair-wise IoU agreement across labelers.
4 MODEL DESIGN
Screen2Words uses an encoder-decoder model to generate screen summaries from multimodal UI information. Its design encodes structural-textual and visual information, fuses them for each UI element, and decodes a variable-length summary.
- 4 MODEL DESIGN: Screen2Words uses an encoder-decoder architecture in which encoded screen information supports natural-language summary generation.The model is based on Transformer components for encoding and decoding.
- 4 MODEL DESIGN: The model combines UI structure, screen text and app description, and screenshot imagery as three input modalities.Structural-textual information and raw pixels are encoded through separate pathways.
- 4.1.1 Encoding Structural and Textual Information: A Transformer encoder represents structural and textual information by using self-attention across screen elements.The view hierarchy is flattened into a sequence, with spatial and tree positions supplying positional information.
- 4.1.2 Encoding the UI Screenshot: A ResNet encodes cropped grayscale images of individual UI elements, avoiding the need to detect elements from the entire screenshot.The resulting visual encoding is concatenated with the structural-textual encoding for each UI element.
- 4.2 Decoding Screen Summaries: The Transformer decoder generates variable-length summaries from contextual multimodal element encodings.Training uses teacher forcing with cross-entropy, while prediction is autoregressive and uses beam search for candidate generation.
5 EXPERIMENTS
The experiments compare multimodal generative models with template-based retrieval baselines for screen summarization. Results show that combining pixel, layout, screen-text, and app-description information yields the strongest summaries.
- Experimental goals: The experiments test whether deep models outperform heuristic methods and whether combining multiple UI modalities improves summarization.Models are evaluated against commonly used automatic metrics on an app-wise held-out test set.
- Model variants: PixelOnly, LayoutOnly, Pixel+Layout, Pixel+Layout+ScreenText, and the full model isolate the contributions of visual, structural, textual, and app-description inputs.PixelOnly uses UI-element images, while LayoutOnly encodes structural properties from the view hierarchy.
- Baselines: Template retrieval baselines represent screens with TF-IDF vectors, pixel values, or CNN-autoencoder features and select similar training screens.TF-IDF treats each screen as a document and similarity is computed with cosine scores over screen representations.
- Automatic results: All generative models outperform template-based retrieval baselines by a large margin across BLEU, CIDEr, ROUGE-L, and METEOR.The metrics are computed from top-1 beam-search predictions on the test set.
- Automatic results: Adding layout, screen text, and app description progressively improves results, with the full model achieving the best accuracy among model variants.The findings indicate that mobile UI modalities complement one another for summarization.
- Qualitative analysis: Test examples show coherent summaries overall, while complex screens can still receive generic or irrelevant descriptions.Layout helps with some complex screens, and text adds context such as app categories.
6 HUMAN EVALUATION
A Mechanical Turk study evaluates generated summaries on sampled test screens using human ratings of summary quality. Human judgments generally favor the full multimodal model and reveal differences from automatic metrics.
- Study setup: The study compares three deep model variants with two template baselines on 1,000 randomly sampled test screens.Three raters assessed each screen, with 1,041 unique workers participating independently of dataset labeling.
- Study setup: Raters assess whether summaries accurately describe screen type, UI elements, and app type, using a five-level quality scale.The highest rating requires an error-free summary that also provides useful details rather than a generic description.
- Study results: Human ratings correlate well with automatic metrics, ranking the full model first, followed by Pixel+Layout and Pixel-Only.The Template (Pixel) baseline receives the lowest average rating.
- Study results: Template (TF-IDF+Pixel) receives a higher human rating than the two deep variants without textual information despite lower automatic-metric performance.The authors speculate that screen-text keywords influence perceived relevance, whereas automatic metrics weight words equally.
- Study results: 38.6% of full-model ratings receive the highest score, compared with 32.2% for Template (TF-IDF+Pixel) and 29.5% for Pixel+Layout.The full model’s mean-rating advantage over every other setting is statistically significant at p<0.0001.
7 POTENTIAL APPLICATIONS
Screen2Words is presented as a bridge between mobile UI content and natural language through several mock-up applications. These include language-based retrieval, screen-reader enhancement, screen indexing, and conversational interaction.
- Potential applications: The paper proposes three mock-up applications: language-based UI retrieval, enhancing screen readers, and screen indexing for conversational mobile applications.These applications illustrate potential uses of automatic screen summarization beyond evaluation.
- Language-Based User Interface Retrieval: Screen2Words supports language-based UI retrieval by letting designers search for mobile interface examples with natural-language queries.The proposed retrieval can capture UI semantics beyond words directly appearing in screen text.
- Enhancing Screen Readers: Screen2Words can enhance screen readers by predicting an overview summary when UI metadata is missing.This complements existing screen-reader behavior that renders text and image content based on metadata describing the UI.
- Enhancing Screen Readers: During app switching, screen summaries can help visually impaired users identify which cached screen they are returning to.Without visual access, existing screen readers may announce only the app name and not the specific cached screen.
- Screen Indexing for Conversational Mobile Applications: Screen summaries can provide language metadata for indexing screens and launching desired pages through spoken descriptions combined with app metadata.Examples include requesting a settings page in Gmail or an ordering page in the Starbucks app.
8 DISCUSSIONS AND FUTURE WORK
The discussion identifies accuracy, evaluation, missing metadata, whole-screen scope, and future UI-generation applications as key considerations for Screen2Words.
- Limitations: Screen2Words sometimes produces generic or wrong summaries, reflecting an acknowledged accuracy limitation.The authors compare this issue to hallucination in image captioning.
- Limitations: Human ratings and automatic scores may not always correlate, motivating better evaluation metrics.The authors suggest investigating new architectures and evaluation metrics.
- Limitations: Missing app metadata can reduce performance, although pixel-only configurations remain usable and pixel input is crucial when screen text is sparse or absent.The multimodal approach can still perform when one modality is missing.
- Limitations: Screen2Words describes entire screens but cannot be steered toward a specified section or answer arbitrary UI questions.The authors propose Screen VQA as a natural extension for questions about actions or particular screen elements.
- Future Work: The open-source dataset could support language-based UI generation by conditioning graphical layout models on screen summaries.This extends the demonstrated use of summaries for UI retrieval with language queries.
9 CONCLUSION
Screen2Words summarizes multimodal mobile UI screens with concise language using deep learning and a large human-annotated dataset. Deep models outperform heuristic baselines, while the full model using image, text, and structure performs best in both automatic and human evaluations.
- Conclusion: Screen2Words summarizes multimodal mobile UI information as concise language using deep learning methods.The approach targets automatic UI screen summarization.
- Conclusion: 22,417 Android UI screens formed the first large-scale human-annotated dataset for investigating automatic UI screen summarization.The dataset supported training and evaluation of multiple deep models.
- Conclusion: Deep learning models outperform heuristic baselines by a significant margin across automatic metrics.The evaluation examined the feasibility of automatic UI screen summarization.
- Conclusion: The full model using image, text, and UI structural information achieves the best results among model variants and baselines in automatic and human evaluations.Mechanical Turk ratings also favored the full model significantly.
- Conclusion: The dataset, benchmark models, and experimental results lay groundwork for future automatic UI screen summarization research.The paper frames this work as contributing to bridging natural language and mobile user interfaces.