Source-linked AI summary
Predicting and Explaining Mobile UI Tappability with Vision Modeling and Saliency Analysis
Eldon Schoop, Xin Zhou, Gang Li, Zhourong Chen, Björn Hartmann, Yang Li
TL;DR
Mobile UI elements can be difficult to judge as tappable, while automated tools may offer less design guidance than user studies or require fully specified apps. This paper predicts perceived tappability from screenshot pixels and explains predictions with saliency and contrasting examples. It presents the system, analyzes its behavior, and reports exploratory feedback from professional designers.
Problem
Varied mobile UI styles can make tappability difficult to judge, and automated usability tools may lack user-study explanations or require fully specified interfaces.
Source: §1 · p. 2Method
The authors train a vision-based model on crowdworker judgments and explain predictions with influential screenshot regions and similar UIs with opposing tappability influences.
Source: §1 · p. 2, §5.1 · p. 5Results
The paper presents a tappability-prediction system and analyzes its behavior with examples and exploratory feedback from professional UI/UX designers.
Source: §1 · p. 2, §9 · p. 13Takeaways & Limitations
The system is intended to help designers assess how users may perceive UI elements, including designs not yet implemented in an application.
Source: §1 · p. 2The model uses a static UI snapshot and may become less reliable as design styles change; its Android-only data may also limit generalization to other platforms.
Source: §7.2.2 · p. 11, §8.4 · p. 13
Abstract
from arXiv · showhide
We use a deep learning based approach to predict whether a selected element in a mobile UI screenshot will be perceived by users as tappable, based on pixels only instead of view hierarchies required by previous work. To help designers better understand model predictions and to provide more actionable design feedback than predictions alone, we additionally use ML interpretability techniques to help explain the output of our model. We use XRAI to highlight areas in the input screenshot that most strongly influence the tappability prediction for the selected region, and use k-Nearest Neighbors to present the most similar mobile UIs from the dataset with opposing influences on tappability perception.
1 INTRODUCTION
Mobile UI elements can be difficult to recognize as tappable, while existing automated usability tools may lack the explanations and design guidance of user studies or require fully specified interfaces. The paper introduces a pixel-based tappability model and explanations intended to support rapid evaluation and design iteration.
1 INTRODUCTION
- The work addresses difficulty judging tappability in varied mobile UI styles, including false affordances and designs whose features are hard to discover.User studies can yield actionable feedback but are costly and time-consuming, while some automated tools require functional apps or detailed specifications.Source: §1 · p. 2
- The paper builds a pixel-only model to predict perceived tappability from a screenshot and selected region, using five crowdworkers’ labels per element to estimate perception.The dataset draws on thousands of mobile UI screenshots and supports designs that may not yet be implemented in an application.Source: §1 · p. 2
- The system offers local explanations of influential screenshot regions and global explanations through visually similar UIs with opposing influences on tappability perception.The authors use these explanations to provide designers with insights for iterating their designs.Source: §1 · p. 2
- The paper evaluates model behavior on random dataset examples and reports an exploratory evaluation with professional UI/UX designers.Its contributions include an in-depth analysis and initial insights into how AI-based tools may assist designers.Source: §1 · p. 2
2 RELATED WORK
Prior work supports UI exploration and automated usability evaluation, but often leaves designers to interpret results themselves. This work combines pixel-based tappability modeling with interpretable saliency and contrasting example retrieval.
2 RELATED WORK
- The paper situates its approach among automated tools for exploring and evaluating UIs and interpreting deep-network predictions.Source: §2 · p. 2
2.1 Data-Driven UI Design and Exploration
- UI design research uses large datasets such as RICO and ERICA, while related tools retrieve or organize examples from screenshots, sketches, wireframes, and annotated layouts.The paper adds tappability annotations from multiple crowdworkers to a cleaned subset of RICO.Source: §2.1 · p. 2
2.2 Computationally Mediated UI Evaluation
- Unlike many automated usability tools, this work uses pixels to evaluate mockups and interpretability methods to show influential regions and contrasting UI examples.This extends Tap-Shoe [37], which models tappability perception, with pixel-only input and more actionable design information.Source: §2.2 · p. 3
2.3 Interpreting and Explaining Deep Neural Network Predictions
- The authors modify XRAI [21] to attribute input features that influence tappability predictions.The related work describes saliency methods that attribute pixels or merge salient pixels into regions, among other interpretability approaches.Source: §2.3 · p. 3
- The system retrieves similar examples in the model’s latent space and separates them by tappability prediction to surface contrasting cases.The approach draws on example-based explanations [31], contrasting examples, and variation theory of learning.Source: §2.3 · p. 3
3 CROWDSOURCING PERCEIVED TAPPABILITY FROM SCREENSHOTS
The authors collect crowd judgments of tappability for selected elements in Android screenshots, using multiple raters per element. The resulting labels include both strong agreement and substantial ambiguity.
3 CROWDSOURCING PERCEIVED TAPPABILITY FROM SCREENSHOTS
- Each element was labeled by five raters, with workers completing up to 90 elements and a median of 30.Raters saw a screenshot with one element highlighted and indicated whether it was tappable.Source: §3 · p. 3
- 18,667 unique UI elements were collected from 3,218 RICO screens, with up to five unique clickable and non-clickable elements selected per screen.Workers could discard examples whose bounding boxes did not align with UI elements; selection also followed stated view-hierarchy filtering rules.Source: §3 · p. 3
- 44.4% of elements received unanimous agreement, while 24.1% were ambiguous, with at most three raters agreeing.The collection included 24 element types, 77% of which were leaves in their view hierarchies.Source: §3 · p. 3
4 MODELING PERCEIVED TAPPABILITY FROM IMAGES
The model predicts perceived tappability from a full screenshot and a mask identifying the selected region, rather than relying on view-hierarchy features. It outperforms a replicated prior model with pixel-only inputs, while comparison with that model using all features is mixed across metrics.
4 MODELING PERCEIVED TAPPABILITY FROM IMAGES
- Using screenshot pixels alone allows evaluation of visually realistic mockups, but omits explicit text, element type, and intended-clickability inputs.The authors note that the prior model may have advantages when designers use non-visual signifiers such as text to indicate tappability.Source: §4 · p. 4
- The model takes a full RGB screenshot plus a binary mask marking the target region, preserving screen context while indicating which element to assess.The authors explain that cropping the target alone discards context that may be important for learning.Source: §4 · p. 4
- At a 50% decision threshold, the model achieves 91.54% precision, 80.23% recall, and 0.9030 AUC on the test set.The model is a modified ResNet-18 trained with cross-entropy loss and stochastic gradient descent.Source: §4 · p. 4
- The pixel-only model clearly outperforms the replicated Swearngin et al. [37] model using pixels alone; against its full-feature version, it achieves better AUC and similar precision but slightly lower recall.The authors suggest the recall difference may reflect the tappable-element distribution and note that threshold fine-tuning can address it.Source: §4 · p. 4
5 EXPLAINING TAPPABILITY PREDICTIONS
The section explains tappability predictions through local saliency maps and global examples from visually similar UIs. The explanations show influential screen regions and contrasting examples associated with opposing tappability predictions.
5 EXPLAINING TAPPABILITY PREDICTIONS
- The section presents local explanations of influential input regions and global explanations showing how similar designs can relate to tappability perception.The explanations are intended to help designers judge relevant visual cues and consider design changes.Source: §5 · p. 4
- On the new dataset, the pixel-only model outperforms [37] in its pixels-only configuration; against [37] with all features, it has slightly higher AUC, similar precision, and slightly lower recall.Table 2 reports these comparisons.Source: Table 2 · p. 5
5.1 Attributing Tappable UI Elements with Saliency Techniques
- XRAI produces an element-specific heatmap of input regions influential to a selected element’s prediction, but its attributions are correlative rather than causal.The method aggregates pixel attributions over segments; using available UI-element bounding boxes lets it summarize widget regions and reduce automated-segmentation noise.Source: §5.1 · pp. 5–6
5.2 Explaining Predictions with Similar Contrasting Examples
- Global explanations retrieve visually similar UIs and split them by predicted tappability to show examples with opposing influences on perception.The method searches for five neighbors from each of the predicted-tappable and predicted-nontappable lists, filtered to probabilities above 65% or below 35%.Source: §5.2 · p. 6
6 ANALYSIS OF SELECTED EXAMPLES
The section examines four randomly selected Android UI elements, pairing tappability predictions with XRAI and nearest-neighbor explanations. Across the examples, the explanations highlight cues in the selected elements and their surrounding UI context.
6.1 ImageView: Food App Header Logo
- The food-app logo was predicted non-tappable (10.01%); XRAI emphasized the logo, while neighbors contrasted larger central graphics with tappable action icons.The authors infer that changing other screen elements would likely not affect perceived tappability because the logo’s appearance or position may signal non-tappability on its own.Source: §6.1 · pp. 6–7
6.2 Button: Health App Card Button
- The health-app Dismiss button was predicted tappable (99.07%); XRAI focused on the button, while non-tappable neighbors were varied and sometimes disagreed with raters.The authors suggest the model associates Material Design buttons with tappability and needs little surrounding context for a confident prediction.Source: §6.2 · p. 7
6.3 TextView: Finance App List Item
- The finance-app bid-price TextView was predicted non-tappable (36.38%); XRAI highlighted tab text and nearby views, while neighbors suggest context distinguishes visually similar text.Tappable neighbors sometimes included nearby icons or graphics and ListViews; non-tappable text was often brighter and descriptive.Source: §6.3 · pp. 7–8
6.4 EditText: Entertainment App Login Field
- The entertainment-app Password field was predicted tappable (99.47%); XRAI focused on the EditText, while neighbors contrasted similar text fields with descriptions and hyperlinks.XRAI can aggregate underlying pixel attributions for large UI objects, although many such objects inherit a single tappability attribute.Source: §6.4 · pp. 8–9
7 EXPLORATORY EVALUATION WITH PROFESSIONAL DESIGNERS
The exploratory evaluation gathered professional designers’ feedback on the model and its explanations. Participants saw potential for rapid design checks and contrasting examples, while also identifying limits in context and interpretability.
7.1 Participants & Study Design
- The study recruited 14 designers at a large technology company and presented five randomized review examples after onboarding, with each session lasting about 45 minutes.Six examples were selected to vary prediction, confidence, and worker agreement; raters’ tappability labels were not shown to participants.Source: §7.1 · pp. 9–10, Figure 15 · p. 11
7.2.1 Tappability predictions can save significant time and effort compared to user studies.
- 11 participants perceived the system as accurate, and 7 saw it as a time-saving alternative to user studies, including for checks at multiple design stages.Because the system takes pixels as input, the authors note it could work on fully implemented UIs or visually realistic mockups.Source: §7.2.1 · pp. 10–11
7.2.2 Analysis of a single screen offers limited notion of context in a UI flow.
- Four participants were critical or skeptical of predictions for UI elements sensitive to other screens in a UI flow; authors note static screenshots lack flow context and suggest temporal input or text.Source: §7.2.2 · p. 11
7.2.3 Contrasting similar examples provided design feedback for iteration.
- At least five participants valued contrasting examples as inspiration, though preferences differed between diverse comparisons and closer semantic matches.Participants suggested filtering examples or showing their distances; the authors see potential for the examples to complement tappability scores with design inspiration.Source: §7.2.3 · p. 11
7.2.4 Participants desire more explicit explanations beyond the heatmap.
- Two participants found the heatmap useful, while four found it confusing or wanted guidance; others requested clearer reasons for attributions or direct design suggestions.The authors suggest that a mismatch between model and designer mental models may help explain the difficulty of interpreting heatmaps.Source: §7.2.4 · p. 11
8 THEMES IN MODEL BEHAVIOR: DISCUSSION AND LIMITATIONS
The discussion examines visual signals used by the model, interpretation of XRAI and nearest neighbors, and limitations in the data and input scope.
8.1 Persistent Signals and Signifiers
- Text can signal tappability for nearby elements without itself being tappable, while icons beside text often signal tappable regions and image content can confound predictions.Checkbox predictions may be uncertain because crowdworkers can label the checkbox, its description, or their parent element as the target; varied image content may overwhelm contextual cues.Source: §8.1 · p. 12
8.2 Challenges in Interpreting XRAI Attributions
- XRAI highlights regions influencing a selected element’s prediction, not regions that are necessarily tappable themselves.Removing highly attributed text near a button could change its prediction; the authors note that heatmaps may require practice and could benefit from heuristic explanations.Source: §8.2 · p. 12
- Region-based XRAI can omit fine pixel details, while pixel-level methods may be harder for people to interpret and susceptible to errors; raw XRAI values are specific to each example.The authors contrast XRAI with methods such as DeepSHAP, which sum to prediction probabilities and may be compared across examples.Source: §8.2 · p. 12
8.3 Browsing Nearest Neighbor Examples
- Nearest neighbors reflect bounding-box location, size, and aspect ratio as well as visual similarity, suggesting the model uses contextual layout cues.Filtering and sorting by component type or application properties could help designers narrow those cues.Source: §8.3 · p. 12
- Splitting neighbors into binary tappability groups discards probability and distance information that can distinguish clearly tappable examples from uncertain ones.The authors propose more continuous, interactive visualizations to expose these differences.Source: §8.3 · p. 12
- Designers may value diverse examples for inspiration, and future retrieval could sample more distant UIs, filter by element type, or draw from other datasets.The authors also mention learned concepts as a possible way to retrieve examples.Source: §8.3 · p. 13
8.4 Additional Limitations
- Because the model uses a static snapshot of Android UIs, changing design trends may reduce reliability over time, and generalization to other platforms may require fine-tuning.The authors identify ambiguous perceived tappability as another constraint on possible model accuracy; predictions near the decision boundary may indicate user confusion.Source: §8.4 · p. 13
9 CONCLUSION
The paper presents an automated system for predicting perceived tappability and explaining predictions. It combines a pixel-based model, a crowdworker-labeled dataset, and analysis of model behavior and designer feedback.
9 CONCLUSION
- The system predicts mobile UI elements’ perceived tappability from pixels and provides contextual and instance-level explanations, supported by a new dataset with multiple crowdworker labels per element.The authors report extensive example analysis and feedback from experienced professional UI/UX designers.Source: §9 · p. 13
A USER STUDY FIGURES
The appendix section presents the UIs and explanation outputs shown to participants in the user evaluation.
A USER STUDY FIGURES
- The section identifies the UIs and explanation-algorithm outputs shown to participants during the user evaluation.Source: A USER STUDY FIGURES · p. 15