Unblind Your Apps: Predicting Natural-Language Labels for Mobile GUI Components by Deep Learning

Jieshan Chen; Chunyang Chen; Zhenchang Xing; Xiwei Xu; Liming Zhu; Guoqiang Li; Jinshui Wang

Source-linked AI summary

Unblind Your Apps: Predicting Natural-Language Labels for Mobile GUI Components by Deep Learning

Jieshan Chen, Chunyang Chen, Zhenchang Xing, Xiwei Xu, Liming Zhu, Guoqiang Li, Jinshui Wang

arXiv:2003.00380v2Mar 1, 2020cs.HCcs.CVcs.SE

TL;DR

Mobile screen readers can help people with vision impairment use apps, but they require meaningful labels for image-based components, which are missing from more than 77% of analyzed apps. The paper introduces LabelDroid, a CNN and transformer encoder-decoder model that predicts these labels from UI images. It reports 60.7% exact match and 0.654 ROUGE-L, outperforming state-of-the-art baselines, while also producing higher-quality descriptions than junior Android developers.

Problem
Screen readers depend on content descriptions, yet more than 77% of apps lack natural-language labels for at least one image-based button.
Method
LabelDroid uses a deep-learning model with a CNN and transformer encoder-decoder to predict labels for image-based buttons from UI images.
Results
60.7% exact match and 0.654 ROUGE-L are reported, outperforming state-of-the-art baselines; predictions also receive higher-quality ratings than those from junior Android developers.
Takeaways & Limitations
The model is presented as a way to assist developers in filling content descriptions and improve access to mobile apps for people with vision impairment.
Takeaways & Limitations
The authors identify visually unusual or ambiguous buttons as sources of prediction errors and plan to incorporate app metadata and assess existing labels’ concision and informativeness.

Abstract

from arXiv · show

According to the World Health Organization(WHO), it is estimated that approximately 1.3 billion people live with some forms of vision impairment globally, of whom 36 million are blind. Due to their disability, engaging these minority into the society is a challenging problem. The recent rise of smart mobile phones provides a new solution by enabling blind users' convenient access to the information and service for understanding the world. Users with vision impairment can adopt the screen reader embedded in the mobile operating systems to read the content of each screen within the app, and use gestures to interact with the phone. However, the prerequisite of using screen readers is that developers have to add natural-language labels to the image-based components when they are developing the app. Unfortunately, more than 77% apps have issues of missing labels, according to our analysis of 10,408 Android apps. Most of these issues are caused by developers' lack of awareness and knowledge in considering the minority. And even if developers want to add the labels to UI components, they may not come up with concise and clear description as most of them are of no visual issues. To overcome these challenges, we develop a deep-learning based model, called LabelDroid, to automatically predict the labels of image-based buttons by learning from large-scale commercial apps in Google Play. The experimental results show that our model can make accurate predictions and the generated labels are of higher quality than that from real Android developers.

1 INTRODUCTION

Mobile screen readers can improve access for people with vision impairment, but they depend on developers providing meaningful labels for image-based UI components. LabelDroid addresses widespread missing and unclear labels by predicting content descriptions from UI images.

Accessibility motivation: Screen readers require content descriptions for unlabeled UI components to provide meaningful feedback to blind users.Without these labels, TalkBack cannot adequately support interaction with the app.
Accessibility motivation: More than 77% of 10,408 analyzed apps miss labels for at least one image-based button.The authors attribute many issues to developers’ limited awareness or knowledge of accessibility guidelines.
Accessibility motivation: Developers may write “add” instead of the more informative “add playlist” for an image-based button.The example illustrates that labels can be present yet insufficiently descriptive.
Approach: LabelDroid predicts content descriptions for image-based buttons from their UI images using a deep-learning model.The model targets these buttons because they are important interaction proxies and cannot be read directly without labels.
Results: 60.7% exact match and 0.654 ROUGE-L are reported for LabelDroid, outperforming both state-of-the-art baselines.The authors also report higher-quality predictions than those produced by junior Android developers.
Contributions: The paper contributes an empirical accessibility study, a large-scale dataset of UI content descriptions, and an automatic labeling model.The dataset is released to support future research.

2 ANDROID ACCESSIBILITY BACKGROUND

Android accessibility relies on content descriptions that screen readers can vocalize for UI components. The paper focuses on image-based buttons, including clickable ImageViews and ImageButtons, and documents substantial label-missing rates.

Content descriptions: Android UI components can use android:contentDescription to store natural-language descriptions for accessibility.The description is read by screen readers rather than displayed visually in the app.
Screen readers: TalkBack lets users traverse Android screens with gestures and hear text or content descriptions for focused components.It is a pre-installed Android accessibility service designed to support blind and vision-impaired users.
Image-based buttons: Image-based buttons need labels because they support interaction while their images cannot be directly read by screen readers.The paper distinguishes clickable images from static images and treats null labels on clickable images as accessibility barriers.
Image-based buttons: Clickable Images are ImageViews with clickable set to true, whereas ImageButtons are implemented by the ImageButton class.ImageButton is described as a subclass of ImageView that visually presents an image instead of text.

3 MOTIVATIONAL MINING STUDY

The study examines accessibility-label coverage in Android apps using crawled GUI data and finds missing image-button labels are widespread across app categories and popularity levels.

3.1 Data Collection: 19,127 apps were randomly crawled from Google Play, and an app explorer collected screenshots and runtime GUI metadata where exploration succeeded.The crawler covered 25 categories and apps with installation numbers from 1K to 100M; successful GUI collection was reported for 15,087 apps.
3.2 Current Status of Image-Based Button Labeling in Android Apps: 61.98% of apps had image buttons without labels, while 74.07% had clickable images without labels.Across 278,234 screens, 60.79% contained at least one element without an explicit label.
3.2 Current Status of Image-Based Button Labeling in Android Apps: 72% of Personalization apps, 71.6% of Game apps, and 71.8% of Photography apps had more than 80% unlabeled image-based buttons.About 40% of Finance, Business, Transportation, and Productivity apps had less than 20% missing labels.
3.2 Current Status of Image-Based Button Labeling in Android Apps: The label-missing rate had only a 0.046 Spearman correlation with installation number, indicating a very weak relationship with app popularity.Popular apps therefore still contained many unlabeled buttons, potentially affecting larger audiences.
3.2 Current Status of Image-Based Button Labeling in Android Apps: Across 10,408 analyzed apps, more than 77% had at least one image-based button missing a label.The summary identifies especially severe missing-label rates in personalization, game, and photography categories.

4 APPROACH

The approach treats content-description prediction as image captioning: CNN features are encoded with a Transformer and decoded into label word sequences.

4 APPROACH: The model extracts button-image features with a CNN, encodes them into tensors, and generates content descriptions as word sequences.This reframes rendering a UI widget into a content description as an image-captioning task.
4.1 Visual Feature Extraction: A CNN uses convolution and pooling layers to extract salient visual features while reducing computational complexity.Pooling down-samples the input and retains dominant features that are invariant to position and rotation.
4.2 Visual Semantic Encoding: The Transformer encoder applies multi-head self-attention and position-wise feed-forward layers to the visual feature sequence.Self-attention allows each feature to be influenced by other sequence features, while residual connections and layer normalization are also applied.
4.2 Visual Semantic Encoding: The model preserves the specified spatial sequence of image-based buttons, ordered from left to right and top to bottom.This sequence is used when computing relationships among visual features.
4.3 Content Description Generation: The Transformer decoder adds cross-attention to relate previous generated outputs to encoder representations before predicting the next word.Future output positions are masked, enabling parallel computation during training and autoregressive generation during inference.
4.3 Content Description Generation: The model is trained with Kullback-Leibler divergence to measure differences between generated and reference probability distributions.The paper notes that this objective is equivalent to cross-entropy loss up to the constant entropy term H(p).

5 IMPLEMENTATION

The implementation constructs a cleaned image-button dataset from crawled Android interfaces and trains an end-to-end CNN–Transformer model.

5 IMPLEMENTATION: Screenshots are paired with runtime XML so GUI coordinates can be parsed and image-based buttons cropped from each screen.Repeated GUI screenshots are removed by comparing their XML files, reducing duplicate components in the dataset.
5 IMPLEMENTATION: The cleaning process removes duplicate components and manually identified non-informative labels, including element-class names and app names.Rules were written to filter these label types from the training data.
5 IMPLEMENTATION: Non-English labels are translated into English, and start, end, unknown, and padding tokens support sequence modeling and mini-batch training.Labels are limited to a maximum of 15 words.
5 IMPLEMENTATION: The cleaned dataset contains 19,233 image-button and content-description pairs from 7,594 apps.For each app category, 80% of apps are used for training, 10% for validation, and 10% for testing.
5 IMPLEMENTATION: The visual module uses ImageNet-pretrained ResNet-101, while the Transformer uses three encoder layers with 512-dimensional model embeddings and eight attention heads.The CNN and encoder-decoder are trained end-to-end with KL divergence loss and Adam.

6 EVALUATION

The evaluation examines LabelDroid’s accuracy, generality, and usefulness through automated testing and a user study.

6 EVALUATION: LabelDroid is evaluated in three aspects: automated accuracy testing, generality, and usefulness with a user study.The supplied evaluation passage states the evaluation dimensions but does not report their numerical outcomes.

6.1 Evaluation Metric

The evaluation uses five image-captioning metrics to compare generated content descriptions with ground truth, combining exact-match and graded similarity measures.

The study evaluates labels with exact match, BLEU, METEOR, ROUGE, and CIDEr metrics.These metrics are adopted from related image-captioning work.
Exact match counts a prediction as correct only when it exactly matches the ground truth.It cannot distinguish between predictions with one difference and predictions with several differences.
BLEU measures similarity using n-gram precision and a brevity penalty, reported as BLEU@1 through BLEU@4.The study uses n values from 1 to 4 because image-button descriptions are generally short.
METEOR accounts for synonyms and recall, while ROUGE-L uses longest-common-subsequence similarity.These metrics address limitations associated with BLEU and capture recall-oriented similarity.
All metrics range from 0 to 1, with higher scores indicating greater similarity to the ground truth.A perfect match receives a score of 1, and the metrics are computed with coco-caption.

6.2 Baselines

LabelDroid is compared with two encoder-decoder image-captioning baselines that use the same datasets and CNN encoder.

The first baseline combines a CNN encoder with an LSTM decoder to generate content descriptions.It uses a long short-term memory unit as the decoder.
The second baseline combines a CNN encoder with a CNN decoder that predicts vocabulary-word probabilities.Its final projection layer applies a linear transformation and softmax.
Both baselines use the same CNN encoder and the same training, validation, and testing datasets as LabelDroid.The methods are denoted CNN+LSTM and CNN+CNN for comparison.

6.3 Accuracy Evaluation

LabelDroid outperforms the two baselines on the held-out test set and shows qualitative strengths in fine-grained recognition, variable-length generation, and noisy images.

6.3.1 Overall Performance: 60.7% of 1,879 held-out image-based buttons receive labels exactly matching the ground truth.The test set consists of randomly selected apps excluded from model training.
6.3.1 Overall Performance: LabelDroid outperforms both baselines across all reported metrics, with gains of about 2% to 11.3%.The improvements are statistically significant across comparisons after BH correction, with p-value<0.01.
6.3.1 Overall Performance: The model maintains steady performance across app categories, including categories with few training apps.This supports the reported generalization across different app categories.
6.3.2 Qualitative Performance with Baselines: LabelDroid generates labels of different lengths, whereas CNN+LSTM favors medium-length labels and CNN+CNN tends toward short labels.The comparison is based on representative qualitative examples.
6.3.2 Qualitative Performance with Baselines: Self-attention and cross-attention help the model relate image patches to output tokens for long labels such as “open in google maps.”The paper reports correct long-sequence predictions that CNN+LSTM does not handle well.
6.3.2 Qualitative Performance with Baselines: The model recognizes labels despite noisy backgrounds, while both baselines can be distracted and output “<unk>.”The paper illustrates this behavior with a “watch” button.
6.3.3 Common Causes for Generation Errors: Among 98 sampled failures, 54 involve model errors, 41 are alternative labels, and 3 reflect incorrect ground truth.The authors argue that exact-match accuracy therefore underestimates performance.

6.4 Generalization and Usefulness Evaluation

On 12 completely unseen popular apps, LabelDroid’s generated descriptions receive higher acceptability scores than those from three junior developers, while the analysis identifies common weaknesses in manual labels.

6.4.1 Procedures: The evaluation uses 156 missing-label buttons from 12 apps with at least 1M installations and no overlap with training or testing data.The apps span 10 categories and contribute 13 buttons on average each.
6.4.1 Procedures: A professional accessibility-experienced developer rates model and human descriptions using a five-point acceptability scale.The evaluator is blinded to whether each label came from LabelDroid or a developer.
6.4.2 Results: LabelDroid achieves a 3.97 average acceptability score, compared with 3.06 to 3.62 for the three developers.The model significantly outperforms the developers by 30.0%, 21.6%, and 9.7%.
6.4.2 Results: 51.3% of LabelDroid labels receive the highest acceptability score, versus 18.59%, 33.33%, and 44.23% for the developers.The model performs better in most evaluated apps.
6.4.2 Results: Low-quality developer labels are commonly verbose, uninformative, ambiguous, or wrong.Among 135 score-1 labels, 18 are verbose, 21 uninformative, 6 ambiguous, and 90 wrong.
6.4.2 Results: Developer feedback indicates limited awareness of accessibility labeling and interest in automatic content-description generation.Participants reported that labels may vary by context and expressed uncertainty about their suitability.

7 RELATED WORK

Prior work addresses mobile-app quality and accessibility through platform guidelines, research tools, and testing practices. However, accessibility testing remains largely reactive and requires developer awareness, motivating more proactive assurance.

Research focus: Mobile-app research has largely focused on functional and non-functional properties, while comparatively few studies address accessibility for users with vision impairment.The surveyed properties include compatibility, performance, energy efficiency, GUI design, localization, privacy, and security.
Platform support: Platform providers publish accessibility guidelines, assistive technologies, and testing tools to help developers create accessible apps.Android and iOS support screen readers, while their guidance includes accessibility principles and assistive-technology documentation.
Accessibility research: Prior accessibility research examines small-scale apps across domains such as health, smart cities, and government engagement, repeatedly identifying missing image-component descriptions as a significant problem.Existing work has also rated the severity and frequency of accessibility errors.
Testing and repair approaches: Existing approaches include dynamic test generation, crowdsourced GUI annotation, runtime repair, manual screen-reader testing, static analysis, and automated testing frameworks.Examples include automated test generation, crowd annotation, interaction proxies, Accessibility Scanner, Android Lint, Espresso, and Robolectric.
Remaining gap: These tools require developer awareness and generally react to accessibility issues after they exist, leaving a need for proactive accessibility assurance.The authors identify both knowledge requirements and the risk that users may be harmed before detected issues are fixed.

8 CONCLUSION AND FUTURE WORK

The paper frames missing labels as a widespread accessibility problem and proposes a deep-learning model for predicting labels of image-based buttons. Future work targets better predictions through app metadata and evaluates whether existing labels are concise and informative.

Conclusion: More than 77% of apps have at least one image-based button without a natural-language label readable by users with vision impairment.This conclusion is based on the authors’ analysis of Android apps and motivates automated label prediction.
Conclusion: The proposed model combines a CNN with a transformer encoder-decoder to predict labels for image-based buttons.The model is intended to address developers’ difficulty in producing suitable labels.
Future work: Future work will incorporate app metadata to improve model quality and assess existing labels for conciseness and informativeness.These are the two future directions explicitly identified by the authors.