PaperlayerView exact revision on arXiv

Source-linked AI summary

Owl Eyes: Spotting UI Display Issues via Visual Understanding

Zhe Liu, Chunyang Chen, Junjie Wang, Yuekai Huang, Jun Hu, Qing Wang

arXiv:2009.01417v2cs.SE

TL;DR

UI display issues arise across devices and are difficult for conventional testing to detect proactively. OwlEye models screenshots with deep learning, detects and localizes issues, and achieves strong detection and localization results while uncovering confirmed or fixed bugs in real-world apps.

  • Problem

    Device-dependent UI display issues such as text overlap, blurred screens, and missing images harm usability, while conventional automated testing generally targets crashes rather than these visual defects.

  • Method

    OwlEye uses a CNN to classify screenshots, Grad-CAM to localize buggy regions, heuristics-based augmentation to expand training data, and DroidBot for automated exploration.

  • Results

    OwlEye achieves 85% precision, 84% recall, and 90% localization accuracy, and finds 57 previously undetected issues, 26 confirmed or fixed by developers.

  • Takeaways & Limitations

    OwlEye provides screenshot-based UI issue detection and localization applicable to real-world mobile-app testing, with confirmed practical findings in popular Android apps.

  • Takeaways & Limitations

    Blurred-screen training examples were obtained online because the category was difficult to generate heuristically, and tiny buggy regions remain difficult to recognize.

Abstract

from arXiv · show

Graphical User Interface (GUI) provides a visual bridge between a software application and end users, through which they can interact with each other. With the development of technology and aesthetics, the visual effects of the GUI are more and more attracting. However, such GUI complexity posts a great challenge to the GUI implementation. According to our pilot study of crowdtesting bug reports, display issues such as text overlap, blurred screen, missing image always occur during GUI rendering on different devices due to the software or hardware compatibility. They negatively influence the app usability, resulting in poor user experience. To detect these issues, we propose a novel approach, OwlEye, based on deep learning for modelling visual information of the GUI screenshot. Therefore, OwlEye can detect GUIs with display issues and also locate the detailed region of the issue in the given GUI for guiding developers to fix the bug. We manually construct a large-scale labelled dataset with 4,470 GUI screenshots with UI display issues and develop a heuristics-based data augmentation method for boosting the performance of our OwlEye. The evaluation demonstrates that our OwlEye can achieve 85% precision and 84% recall in detecting UI display issues, and 90% accuracy in localizing these issues. We also evaluate OwlEye with popular Android apps on Google Play and F-droid, and successfully uncover 57 previously-undetected UI display issues with 26 of them being confirmed or fixed so far.

1 INTRODUCTION

UI display issues arise from increasingly complex interfaces and device differences, but existing testing often misses them; OwlEye uses visual understanding to detect and localize these issues automatically.

  • Problem: Display issues such as text overlap, missing images, and blurred screens occur across devices and can harm usability and user experience.The paper attributes many issues to differing system settings across Android versions and device models.
  • Motivation: Manual GUI testing requires substantial effort and can miss minor errors, while automated tools commonly target crashes rather than screenshot-level display issues.The paper therefore seeks a proactive mechanism that checks UI display before release.
  • Approach: OwlEye combines CNN-based screenshot classification with Grad-CAM localization to identify UI display issues and guide developers toward buggy regions.It also uses heuristics-based augmentation and can integrate with DroidBot for automatic screenshot collection, detection, and localization.
  • Evaluation: 85% precision and 84% recall were achieved, while localization reached 90% accuracy against 13 state-of-the-art baselines.The detection results represented more than 17% and 50% gains in recall and precision over the best baseline, respectively.
  • Real-world evaluation: 57 previously undetected UI display issues were found among 329 real-world apps, with 26 confirmed or fixed by developers.The apps came from Google Play and F-Droid.
  • Contributions: The work contributes a systematic investigation, a large-scale labelled dataset, and a heuristics-based augmentation method for UI display issues.The augmentation method generates issue screenshots from bug-free UI images.

2 MOTIVATIONAL STUDY

The motivational study measures how often UI display issues occur, categorizes their forms, and shows why image-based detection is preferable to effort-intensive, platform-specific program analysis.

  • Study design: A pilot study was conducted both to measure issue prevalence and to identify issue types that could guide automated detection design.The study examined crowdtesting screenshots and additional screenshots from the Rico dataset.
  • Prevalence: A manual study found UI display issues in 43.2% of 10,330 crowdtesting screenshots, while 18 screenshots from 16 of 200 Rico apps also contained issues.The Rico estimate is likely underestimated because the dataset does not cover all app pages or device configurations.
  • Categorization: The study groups UI display issues into component occlusion, text overlap, missing image, NULL value, and blurred screen categories.The categories support the design and evaluation of the detection approach.
  • Categorization: Component occlusion covers 47% of categorized issues, text overlap 21%, and missing image 25%, while NULL value and blurred screen account for 6% and 1%.The listed categories differ in appearance and possible causes, including adaptive layouts, loading failures, parameter or database problems, and hardware limitations.
  • Motivation: Program analysis is costly because it requires app instrumentation, platform-specific rules, code rewriting, and compatibility customization across devices.These requirements motivate a more efficient and general detection method.

3 ISSUES DETECTION AND LOCALIZATION APPROACH

OwlEye combines CNN-based screenshot classification with Grad-CAM localization to detect UI display issues and identify their positions for debugging.

  • 3 ISSUES DETECTION AND LOCALIZATION APPROACH: OwlEye first classifies whether a UI screenshot contains display issues, then uses Grad-CAM to localize the detailed issue region.The CNN processes screenshot visual information, while Grad-CAM highlights regions important to the predicted bug category.
  • 3.1 CNN-based UI Display Issues Detection: The CNN converts fixed-size screenshots into learned feature representations through convolution, batch normalization, ReLU, pooling, and fully connected layers.Its fully connected layers encode screenshot features into a K-dimensional vector, and softmax produces category probabilities.
  • 3.2 Grad CAM-based UI Display Issues Localization: Grad-CAM uses class-specific gradients from the final convolutional layer to weight feature maps and produce a class-discriminative localization map.The weighted activation map identifies image regions important to the predicted bug class.
  • 3.1 CNN-based UI Display Issues Detection: The CNN architecture contains 12 convolutional layers, 6 pooling layers, and 4 fully connected layers for UI display issue classification.Convolutional kernels are 3 × 3, with channel counts increasing across successive convolutional-layer groups.

4 HEURISTIC-BASED DATA AUGMENTATION

OwlEye addresses the lack of labelled buggy UI screenshots with heuristic augmentation that transforms bug-free screenshots and their view hierarchies into screenshots containing display issues.

  • 4 HEURISTIC-BASED DATA AUGMENTATION: The augmentation method generates UI screenshots with display issues from bug-free UI images because no suitable open dataset was available.It uses screenshots and associated JSON view hierarchies to create labelled buggy examples.
  • 4 HEURISTIC-BASED DATA AUGMENTATION: The algorithm locates TextView and ImageView components, randomly selects one according to the target category, and alters its position, size, or covering content.Coordinates and dimensions from the JSON hierarchy guide the transformations.
  • 4 HEURISTIC-BASED DATA AUGMENTATION: Blurred-screen examples are collected online rather than generated by the heuristic method because this category is difficult to synthesize.The authors leave improved blurred-screen generation for future work.
  • 4 HEURISTIC-BASED DATA AUGMENTATION: Component occlusion uses a background-colored block to cover part of a TextView, whereas text overlap uses duplicated text offset over the original content.These transformations create visually distinct occlusion and overlap bugs.
  • 4 HEURISTIC-BASED DATA AUGMENTATION: Missing-image screenshots are generated by covering the original image area with a randomly selected image icon while matching the original background color.NULL-value screenshots similarly cover the original TextView with a matching color block containing NULL.

5 EXPERIMENT DESIGN

The experiment evaluates OwlEye for UI display-issue detection and localization using crowdtesting and augmented screenshots, with comparisons against machine-learning and deep-learning baselines.

  • Experimental goals: OwlEye is evaluated for issue detection, localization, and real-world usefulness through DroidBot integration and development-team reporting.The study defines separate evaluation questions for detection, localization, and real-world operation.
  • Dataset construction: The dataset combines crowdtesting screenshots with augmented screenshots and removes visually similar examples to improve experimental diversity.Augmented screenshots are filtered using ORB features and cosine similarity, with balanced bug-free counterparts collected for experimentation.
  • Experimental setup: The experiment separates crowdtesting screenshots into training, validation, and testing sets by application, while adding all augmented screenshots to the dataset.The split uses 6,340 training, 1,000 validation, and 1,600 testing screenshots from distinct app groups.
  • Baselines: OwlEye is compared with 13 baselines spanning handcrafted visual features paired with four machine-learning classifiers and an MLP applied directly to screenshots.The handcrafted-feature baselines use SIFT, SURF, or ORB with SVM, KNN, Naive Bayes, or Random Forests.
  • Evaluation metrics: Detection performance is measured with precision, recall, and F1-score, while localization agreement is assessed with Kendall’s W.Higher detection-metric values indicate better performance, and Kendall’s W approaches 1 with stronger agreement among raters.

6 RESULTS AND ANALYSIS

OwlEye detects UI display issues with strong overall performance, while Grad-CAM localizes buggy regions with high practitioner agreement. Its results improve over baselines and benefit from deeper, batch-normalized CNNs and heuristic data augmentation, but tiny buggy regions remain difficult.

  • 6.1 Issues Detection Performance (RQ1): 85% precision and 84% recall demonstrate OwlEye’s UI display issue detection performance, exceeding the best baselines by 50% in precision and 17% in recall.The comparison is against ORB-NB for precision and MLP for recall.
  • 6.1 Issues Detection Performance (RQ1): All five issue categories achieved at least 0.82 precision and 0.80 recall, with missing-image cases strongest and text-overlap cases weakest.Missing-image detection reached 0.86 precision and 0.90 recall, whereas text overlap reached 0.82 precision and 0.81 recall because its patterns and buggy regions are smaller and more varied.
  • 6.1 Issues Detection Performance (RQ1): Tiny buggy regions cause missed detections and localization errors, especially for text overlap and component occlusion.The authors propose attention mechanisms and image magnification as future directions for difficult detection cases.
  • 6.1 Issues Detection Performance (RQ1): Increasing convolutional layers from 4 to 12 with batch normalization improved precision by 18% and recall by 58%, while adding batch normalization at 12 layers improved them by 18% and 47%.These configuration changes influenced detection performance in the reported experiments.
  • 6.1 Issues Detection Performance (RQ1): Adding 15,640 augmented screenshots improved precision by 13% and recall by 35%, with the larger recall gain indicating broader issue coverage.The authors attribute this to greater screenshot diversity in the augmented training set; the null-value category improved most in F1-score.
  • 6.2 Issues Localization Performance (RQ2): 90% of localization judgments were positive, while six practitioners showed high agreement with Kendall’s W of 0.946.Practitioners strongly agreed or agreed for 90% of screenshots and disagreed for 4%.
  • 6.3 Usefulness Evaluation (RQ3): OwlEye detected 57 UI display issues in 329 explored apps, with 26 fixed or confirmed by developers.The detected issues comprised 24 in F-droid and 33 in Google Play; 6 and 4 were fixed, while 8 and 8 were confirmed, respectively.

7 DISCUSSION

OwlEye shows cross-platform and cross-language potential for detecting UI display issues from screenshots, while its real-world impact depends partly on more effective automated exploration tools.

  • Generality across platforms: 71% of seven buggy iOS screenshots were accurately detected, supporting OwlEye’s potential beyond Android.The authors describe this small-scale result as evidence of cross-platform generality and note that broader experiments remain future work.
  • Generality across languages: 16 of 22 buggy screenshots in German and Korean were accurately detected, demonstrating feasibility across display languages.The experiments also used Chinese and English screenshots.
  • Potential with more effective automatic testing tool: OwlEye’s real-world usefulness is limited when DroidBot cannot run applications or captures only one screenshot, restricting GUI exploration.A more effective automatic testing tool could allow OwlEye to play a bigger role in practice.

8 RELATED WORK

Prior GUI research addresses search, code generation, design consistency, linting, and functional testing, whereas OwlEye targets non-functional visual issues caused mainly by device and Android-version compatibility.

  • GUI design and implementation: GUI research spans image-based search, computer-vision code generation, and visual comparison against original UI designs.These studies assist GUI search, generate GUI code, or check implemented designs for image similarity.
  • Linting and GUI testing: Static linting flags programming, stylistic, and suspicious constructs, whereas automatic GUI testing dynamically explores application interfaces.Android Lint covers more than 260 Android bug types, while StyleLint enforces style conventions.
  • Visual issue detection: OwlEye differs from functional GUI testing by detecting non-functional visual issues that typically do not crash apps but can reduce usability.The targeted bugs are mainly attributed to compatibility differences across devices and Android versions.
  • Platform scope: Existing GUI bug-detection studies are generally platform-specific, while OwlEye uses screenshots to support detection across Android, iOS, and IoT devices.The screenshot-based formulation is presented as enabling broader platform generalization.

9 CONCLUSION

The paper presents OwlEye as a proactive screenshot-based approach for improving mobile-app quality, reports substantial real-world findings and benchmark gains, and identifies further model and repair work.

  • Conclusion: 26 previously undetected UI display issues in popular Android apps were confirmed or fixed, demonstrating OwlEye’s practical effectiveness.The paper also reports 57 previously undetected issues uncovered in real-world evaluation.
  • Conclusion: OwlEye improved recall by more than 17% and precision by 50% over the best baseline.These are the paper’s reported comparative gains for UI display issue detection.
  • Future work: Future work will improve classification, locate issue root causes, and recommend patches for fixing display bugs.The planned extensions move beyond detection toward diagnosis and repair assistance.
Loading 2009.01417v2…