ActionBert: Leveraging User Actions for Semantic Understanding of User Interfaces

1 Introduction

• UI understanding lacks a generic feature representation for multiple tasks, while task-specific models can be inefficient to deploy and may overfit limited task data.UI data also includes domain-specific structures such as VH and DOM, whose semantic information is not in natural-language form.Source: §1 · p. 1
• Similar-looking UI components can have different functions, so understanding functionality requires context beyond appearance.The introduction illustrates this with house-like icons that have different functionalities on their source screens.Source: §1 · p. 1, Figure 1 · p. 2
• ActionBert is a pre-trained transformer-style model that uses sequential user actions to represent UI components’ functionality.Its approach combines transformer representations with domain-specific information such as view hierarchies and user actions.Source: §1 · p. 1
• The paper introduces self-supervised pre-training tasks based on user actions and UI-specific features to capture component context and semantics.It evaluates the model on four types of UI downstream tasks.Source: §1 · p. 2

2 Background of UI View Hierarchy

• A view hierarchy is a tree-based UI representation whose leaf nodes provide attributes related to component appearance and functionality.The attributes used here include content description, resource ID, component class, and component text.Source: §2 · p. 2, Figure 2 · p. 2

3.1 Revisiting BERT

• BERT is a transformer-based bidirectional language model that processes word-token embeddings through a multi-layer bidirectional transformer.ActionBert is inspired by BERT and extends its concepts to UI embeddings.Source: §3.1 · p. 2
• BERT-style models transfer learned features to multiple NLP tasks.The paper notes BERT’s success as motivation for adapting pre-training concepts to UI understanding.Source: §3.1 · p. 2
• BERT pre-training uses masked language modeling to predict masked words and next sentence prediction to classify whether one sentence immediately follows another.The NSP classifier operates on the BERT embedding of two sentences separated by [SEP].Source: §3.1 · p. 2

3.2 ActionBert: Semantic UI understanding with user actions

• ActionBert treats UI components as tokens, a UI as a sentence, and a user interaction trace as a paragraph or document linked by clickable components.Component meaning can be identified from components in the same UI and the UI that follows; the tick button's role is inferred as online check-in from both screens' content.Source: §3.2 · pp. 2–3, Figure 4 · p. 3

Text embedding

• Each text token concatenates a view-hierarchy leaf's content description, resource ID, component class, and text, then uses a sentence-level text encoder to produce its embedding.Source: Text embedding · p. 3

Vision embedding

• Vision tokens use crops of UI components from view-hierarchy bounding boxes, or detected component boxes when the hierarchy is unavailable, and a vision encoder produces their embeddings.Source: Vision embedding · p. 3

Positional embedding

• Nine normalized geometric features, including bounding-box coordinates, center, height, width, and area, represent each component's position in the UI.Source: Positional embedding · p. 3

Segment embedding

• Segment embeddings distinguish UI-A text, UI-B text, UI-A vision, and UI-B vision; the model combines input features and uses attention across components and modalities.The [CLS] output represents the UI pair, while each component position yields a contextual component embedding.Source: Segment embedding · p. 3

3.3 Pre-training ActionBert

• ActionBert uses link component prediction, consecutive UI prediction, and masked view-hierarchy text prediction to pre-train on UI sequences with user actions.The first two tasks learn connectivity and relationships between UIs; masked text prediction learns relationships between component text and visual and textual context.Source: §3.3 · pp. 3–4, Figure 5 · p. 4
• Pre-training uses 60,328 automatically crawled user-action sequences, yielding 2.69 million UIs and 5.4 million UI pairs with actions and view hierarchies.The sequences range from two UIs to hundreds; negative pairs are sampled evenly from non-consecutive same-sequence pairs and pairs from different sequences.Source: §3.3 · p. 4

Pre-training task #1: Link component prediction

• Link component prediction selects which text or vision component can be clicked to transition between two UIs, with the training click location identifying the correct link component.The model must learn both UIs' semantics to identify the component that connects them.Source: §3.3 · p. 4

Pre-training task #2: Consecutive UI prediction

• Consecutive UI prediction determines whether UI-B follows UI-A through a single interaction, using consecutive pairs as positives and 50%-50% negative sampling.Negative pairs are drawn either from non-consecutive UIs in the same sequence or from different interaction sequences.Source: §3.3 · p. 4

Pre-training task #3: Masked VH text prediction

• Masked view-hierarchy text prediction masks 15% of text components and regresses their high-dimensional text embeddings rather than predicting individual word tokens.The text tokens combine multiple view-hierarchy fields and are treated as sentence-like units.Source: §3.3 · p. 5

3.4 Fine-tuning ActionBert

• Fine-tuning adds a task-specific loss and jointly tunes all parameters, including text and vision encoders, without substantial task-specific architecture changes or large task-specific datasets.Source: §3.4 · p. 5

Single-UI tasks

• For single-UI tasks, the UI-B text and vision segments are left empty; for natural-language retrieval, the description goes in UI-A text and the screenshot serves as its vision token.When view hierarchy is unavailable, OCR and object detection extract text and vision components.Source: Single-UI tasks · p. 5, §3.4 · p. 5

Two-UI tasks

• Two-UI tasks assign each UI's text and vision components to the corresponding UI-A and UI-B segments.Source: Two-UI tasks · p. 5

Extension to multi-UI tasks

• ActionBert can be extended to three-or-more-UI inputs by separating UI and modality segments with [SEP] and training additional segment embeddings.The paper notes that such tasks are not common in practice.Source: Extension to multi-UI tasks · p. 5

4.1 Pre-training

• Pre-training used 60,328 UI sequences and 5.4M UI pairs, with app-disjoint 80:10:10 data splits and 6-layer and 12-layer model variants.Source: §4.1 · p. 5

4.2 Fine-tuning

• The fine-tuning evaluation compares ActionBert with ResNet+BERT and randomly initialized ActionBert across four representative downstream tasks.The random-initialization comparison assesses the impact of the transformer architecture and pre-training tasks.Source: §4.2 · p. 5

Similar UI component retrieval

• 3.06% accuracy improvement over ResNet+BERT on app retrieval and 14.25% over MobileNet + Universal Sentence Encoder on web retrieval were reported.Source: Table 2 · p. 6, Similar UI component retrieval · p. 6

Referring expression component retrieval

• ActionBertLARGE performs best on referring-expression retrieval, with a 3.97% improvement over the ResNet+BERT benchmark.Source: Table 3 · p. 6, Referring expression component retrieval · p. 6

Icon classification

• For icon-32 classification, ActionBertLARGE reaches 97.50% macroaccuracy and 0.8766 micro-F1; on 77 classes, it improves accuracy by 0.37% over baseline.ActionBertBASE performs slightly better on micro-F1, which the authors attribute to Rico's skewed class distribution.Source: Table 4 · p. 7, Icon classification · p. 7

App type classification

• 15.5% accuracy and 16.6% macro F1 gains over ResNet+BERT were reported for app type classification.Source: Table 5 · p. 7, App type classification · p. 7

Link component prediction

• ActionBert models significantly outperform ResNet+BERT on link component prediction; pre-trained ActionBert exceeds its non-pretrained version by 3.4%.Source: Table 6 · p. 7, Link component prediction · p. 7

• Earlier UI embeddings targeted specific tasks: similar UI retrieval (Deka et al. 2017; Huang, Canny, and Nichols 2019; Liu et al. 2018) and referring-expression retrieval (Wichers, Hakkani-Tür, and Chen 2018).Source: §5 · p. 7
• The authors position their work as the first to show generic pre-trained UI representations can improve varied UI tasks and to investigate user actions for UI understanding.Source: §5 · p. 7
• Vision-language pre-training models such as VL-BERT (Su et al. 2019), ViL-BERT, and UNITER (Chen et al. 2020) focus on natural language and images, not UI-specific features or actions.Source: §5 · p. 7

• ActionBert uses user actions to build generic feature representations for UI understanding.Source: §6 · p. 7
• The model is pre-trained on sequential user actions and UI domain-specific features, and is presented as applicable to multiple UI understanding tasks.Source: §6 · p. 7
• Pre-training improves performance across four representative UI task types, and ActionBert outperforms multimodal baselines across downstream tasks by up to 15.5%.Source: §6 · p. 7