PaperlayerView exact revision on arXiv

Source-linked AI summary

UIBert: Learning Generic Multimodal Representations for UI Understanding

Chongyang Bai, Xiaoxue Zang, Ying Xu, Srinivas Sunkara, Abhinav Rastogi, Jindong Chen, Blaise Aguera y Arcas

arXiv:2107.13731v2cs.CVcs.AI

TL;DR

UI understanding needs multimodal representations that remain effective when labeled data is scarce. UIBert uses five self-alignment-based pretraining tasks on unlabeled UI data to learn generic UI and component representations, achieving the best performance across nine downstream tasks and outperforming baselines by up to 9.26% accuracy.

  • Problem

    UI understanding must combine image, text, and structural metadata despite noisy cross-modal information and limited high-quality labeled data.

    Source: §1 · p. 1

  • Method

    UIBert is a transformer-based multimodal model that uses five pretraining tasks to exploit alignment among heterogeneous features in unlabeled UIs.

    Source: §1 · p. 2, §4.3 · p. 4

  • Results

    UIBert achieves the best performance across nine downstream UI tasks and outperforms strong baselines by up to 9.26% accuracy.

    Source: §1 · p. 2, §5.2 · p. 6, §6 · p. 7

  • Takeaways & Limitations

    The results support learning generic UI representations from multimodal self-alignment for broad downstream UI understanding.

    Source: §1 · p. 2, §6 · p. 7

Abstract

from arXiv · show

To improve the accessibility of smart devices and to simplify their usage, building models which understand user interfaces (UIs) and assist users to complete their tasks is critical. However, unique challenges are proposed by UI-specific characteristics, such as how to effectively leverage multimodal UI features that involve image, text, and structural metadata and how to achieve good performance when high-quality labeled data is unavailable. To address such challenges we introduce UIBert, a transformer-based joint image-text model trained through novel pre-training tasks on large-scale unlabeled UI data to learn generic feature representations for a UI and its components. Our key intuition is that the heterogeneous features in a UI are self-aligned, i.e., the image and text features of UI components, are predictive of each other. We propose five pretraining tasks utilizing this self-alignment among different features of a UI component and across various components in the same UI. We evaluate our method on nine real-world downstream UI tasks where UIBert outperforms strong multimodal baselines by up to 9.26% accuracy.

1 Introduction

UI understanding supports accessibility and device operation but requires combining noisy image, text, and structural metadata despite limited labeled data. UIBert addresses these challenges with self-alignment-based pretraining and achieves broad downstream gains.

  • UI understanding supports accessibility, voice control, screen-reader descriptions, and UI design assistance across several practical tasks.Source: §1 · p. 1
  • UI data combines images, text, and structural metadata, while cross-modal noise and expensive task-specific labeling make general UI understanding difficult.Source: §1 · p. 1
  • UIBert learns contextual UI representations from unlabeled data by exploiting predictive alignment among heterogeneous features within a UI.Source: §1 · p. 2
  • UIBert outperforms strong baselines on all nine downstream tasks across five categories, including zero-shot evaluations.Source: §1 · p. 2
  • The work releases two Rico-extended datasets for similar UI component retrieval and referring-expression component retrieval.Source: §1 · p. 2

2 Related Work

Prior UI understanding methods were generally task-specific and label-intensive, whereas UIBert targets reusable UI knowledge from unlabeled data. Its closest comparison, ActionBert, uses temporal UI connectivity, while UIBert exploits multimodal self-alignment within one UI.

  • Earlier UI models addressed individual tasks such as action grounding, description generation, and retrieval, typically requiring substantial labeled data.Source: §2 · p. 2
  • UIBert instead learns general UI knowledge from large-scale unlabeled data for application across multiple tasks.Source: §2 · p. 2
  • Compared with ActionBert, UIBert designs pretraining around self-alignment among multimodal features in a single UI rather than temporal connections across UI sequences.Source: §2 · p. 2
  • UIBert uses trainable lightweight image and text encoders, enabling domain-specific knowledge to be represented in encoder parameters.Source: §2 · p. 2

3 Background

The background defines Android view hierarchies as structured UI inputs and explains BERT as the conceptual basis for UIBert’s pretraining. View hierarchies provide useful but incomplete and noisy information about UI elements.

  • An Android view hierarchy is a tree whose nodes describe UI-element attributes such as bounds, functions, and class information.Source: §3 · p. 2
  • View-hierarchy fields include visible text, content descriptions, resource IDs, class names, and bounding-box locations, though several fields may be empty.Source: §3 · p. 3
  • Because view hierarchies are noisy, nonstandardized, and potentially insufficient to determine the screen layout alone, UIBert combines them with other modalities.Source: §3 · p. 3
  • BERT uses masked language modeling and next-sentence prediction, which UIBert adapts into three masked-input and two sentence-prediction-style UI pretraining tasks.Source: §3 · p. 3

4 Pre-training

UIBert pretrains a six-layer multimodal Transformer on unlabeled UI screenshots and view hierarchies, using component-level image, OCR, and structural features. Five self-alignment tasks learn contextual UI and component embeddings, while qualitative analyses show clustered representations and useful generated descriptions.

4.2 UIBert Architecture

  • UIBert combines IMG, OCR, and VH components from each UI and fuses their modalities with a six-layer Transformer using 512 hidden units and 16 self-attention heads.Inputs are organized as CLS, IMGs, SEP, OCRs, SEP, VHs, SEP; type, positional, and content embeddings are summed before encoding.Source: §4.2 · p. 3

4.3 Pre-training Tasks

  • Five pretraining tasks use cross-component alignment and masked-component reconstruction on 537k screenshot–view-hierarchy pairs collected with the Firebase Robo crawler.The tasks comprise real UI prediction, real component prediction, masked image prediction, masked OCR generation, and masked VH generation.Source: §4.3 · pp. 3, 5
  • Real UI and real component prediction distinguish fake UIs and misaligned components created by replacing 15% of one component type with components from another UI.The component-level task uses surrounding context to identify switched components, while the UI-level task predicts whether the whole UI is real.Source: §4.3 · p. 4, Figure 3 · p. 4
  • Masked image prediction uses contrastive learning, while masked OCR and VH generation reconstruct text or structural information from contextual UI embeddings.VH masking targets Content description and Class name; OCR masking predicts selected WordPiece tokens with a GRU decoder.Source: §4.3 · pp. 4–5
  • Masked VH generation addresses missing accessibility metadata because content descriptions can support screen-reading tools, while more than 66% of buttons lack such descriptions.The paper reports this as a potential application of the generated descriptions.Source: §4.3 · pp. 4–5

4.4 Qualitative Evaluation

  • Zero-shot embeddings cluster UIs by app type and components by icon type, and generated content descriptions match ground truth in 70% of RicoSCA examples.The qualitative evaluation uses t-SNE visualizations and generates descriptions without fine-tuning; some incorrect descriptions are nevertheless reasonable.Source: §4.4 · p. 5, Figure 4 · p. 5

5 Downstream Tasks

UIBert is evaluated as a generic contextual UI representation across nine practical tasks in five categories, including zero-shot settings. It outperforms the reported baselines across the evaluated tasks, with gains especially large on zero-shot retrieval.

5 Downstream Tasks

  • The downstream evaluation fine-tunes all parameters end-to-end with minimal task-specific parameters across nine tasks, including two zero-shot tasks.EfficientNet+Albert provides separate image and text encodings without cross-modal attention, while ActionBert is pretrained with user interaction traces.Source: §5 · p. 5, §5.1 · p. 5

5.2 Similar UI Component Retrieval

  • UIBert outperforms both baselines on all four similar UI component retrieval variations by 0.85%–9.26%, with especially large gains on zero-shot tasks.The task selects the functionally closest candidate component from an anchor UI and search UI; experiments use Rico and web UI datasets.Source: §5.2 · pp. 5–6

5.3 Referring Expression Component Retrieval

  • UIBert achieves 90.81% accuracy on referring expression component retrieval, outperforming ActionBert by 2.43%.The model retrieves the UI component referred to by an expression from an average of 20 candidates.Source: §5.3 · p. 6, Table 2 · p. 6

5.4 Image-VH Sync Prediction

  • UIBert achieves 79.07% accuracy and 77.06% macro-F1 on image-VH synchronization prediction, outperforming the baseline.The task predicts whether a UI screenshot and its view hierarchy are synchronized, using synchronized and unsynchronized RicoSCA examples.Source: §5.4 · p. 6

5.5 App Type Classification

  • UIBert exceeds EfficientNet+Albert by 4.50% accuracy and 4.72% Macro-F1 on app type classification.The task classifies Rico UIs into 27 app types, and concatenated embeddings of all UI components perform better than the CLS embedding alone.Source: §5.5 · pp. 6–7

5.6 Icon Classification

  • UIBert consistently outperforms the baselines in both accuracy and F1 score for icon classification at 32- and 77-class granularities.Icon representations concatenate the embeddings of corresponding image and view-hierarchy components.Source: Table 3 · p. 6, §5.6 · p. 7

6 Conclusion

UIBert learns multimodal UI representations through novel pretraining tasks and achieves the best performance across nine UI-related downstream tasks. Visualization of embeddings and generated content descriptions further demonstrates the pretrained model’s efficacy.

6 Conclusion

  • UIBert learns multimodal UI representations through novel pretraining tasks and achieves the best performance across nine UI-related downstream tasks.The paper positions this approach as facilitating development toward generic UI understanding.Source: §6 · p. 7
Loading 2107.13731v2…