Source-linked AI summary
Object Detection for Graphical User Interface: Old Fashioned or Deep Learning or a Combination?
Jieshan Chen, Mulong Xie, Zhenchang Xing, Chunyang Chen, Xiwei Xu, Liming Zhu, Guoqiang Li
TL;DR
GUI element detection lacks systematic evidence about whether generic old-fashioned and deep-learning object detectors fit GUI-specific visual characteristics and strict localization requirements. The paper evaluates seven representative methods on more than 50,000 GUI images, then combines a GUI-specific coarse-to-fine detector with mature deep-learning components; on 25,000 images, the resulting method significantly advances state-of-the-art performance.
Problem
Existing GUI detectors borrow computer-vision methods without accounting for GUI-specific characteristics and high localization requirements, while their capabilities and design sensitivities lack systematic large-scale study.
Method
The paper conducts a large empirical comparison and combines a top-down coarse-to-fine old-fashioned detector for non-text elements with ResNet50 classification and EAST scene-text detection.
Results
0.573 F1 for all GUI elements, 0.523 for non-text elements, and 0.516 for text elements on 25,000 GUI images, outperforming the best deep-learning model by 19.4%.
Takeaways & Limitations
GUI element detection benefits from combining GUI-specific old-fashioned region detection with mature deep-learning models rather than relying on one generic detection approach.
Takeaways & Limitations
The evaluation focuses on Android app UIs, while generalization to other platforms and GUI designs requires substantial manual labeling and remains future work.
Abstract
from arXiv · showhide
Detecting Graphical User Interface (GUI) elements in GUI images is a domain-specific object detection task. It supports many software engineering tasks, such as GUI animation and testing, GUI search and code generation. Existing studies for GUI element detection directly borrow the mature methods from computer vision (CV) domain, including old fashioned ones that rely on traditional image processing features (e.g., canny edge, contours), and deep learning models that learn to detect from large-scale GUI data. Unfortunately, these CV methods are not originally designed with the awareness of the unique characteristics of GUIs and GUI elements and the high localization accuracy of the GUI element detection task. We conduct the first large-scale empirical study of seven representative GUI element detection methods on over 50k GUI images to understand the capabilities, limitations and effective designs of these methods. This study not only sheds the light on the technical challenges to be addressed but also informs the design of new GUI element detection methods. We accordingly design a new GUI-specific old-fashioned method for non-text GUI element detection which adopts a novel top-down coarse-to-fine strategy, and incorporate it with the mature deep learning model for GUI text detection.Our evaluation on 25,000 GUI images shows that our method significantly advances the start-of-the-art performance in GUI element detection.
1 INTRODUCTION
GUI element detection is a domain-specific object-detection problem supporting software engineering tasks, but existing computer-vision methods were not systematically evaluated against GUI-specific characteristics and strict localization needs. The paper studies representative methods at scale and combines GUI-specific traditional detection with mature deep-learning models.
- Motivation: Pixel-based GUI recognition detects element regions and classes directly from images, providing a non-intrusive and generic alternative to instrumentation-based methods.It supports applications including GUI automation, testing, advanced interaction, search, code generation, and cross-platform robotic testing.
- Research gap: Existing studies had not systematically examined how GUI characteristics, model designs, training data, and anchor-box settings affect detection performance.Earlier evaluations used only dozens to hundreds of GUIs, while the only large-scale evaluation tested one default anchor-box setting.
- Empirical study: 50,524 GUI screenshots from 8,018 Android applications support a comprehensive comparison of two old-fashioned methods and three deep-learning detectors.The study covers major method styles, including REMAUI, Xianyu, Faster RCNN, YOLOv3, and CenterNet.
- Findings: At IoU>0.9, old-fashioned methods achieve F1=0.201 and F1=0.154, whereas Faster RCNN reaches F1=0.438 for non-text GUI element detection.Deep-learning methods perform much better than the old-fashioned methods, although Faster RCNN still struggles to balance coverage and bounding-box accuracy.
- Findings: The study finds that anchor-box settings are robust, merging settings can improve performance, and dedicated text and non-text models outperform a single model.It also treats GUI text as scene text and finds that pretrained EAST can accurately detect it without fine-tuning.
- Proposed approach: The proposed combination achieves 0.573 in F1 for all GUI elements, 0.523 for non-text elements, and 0.516 for text elements on 25,000 GUI images.It uses a GUI-specific top-down coarse-to-fine old-fashioned detector with ResNet50 classification and EAST text detection, outperforming the best deep-learning model by 19.4%.
2 PROBLEM SCOPE AND SOLUTION SPACE
GUI elements differ from ordinary physical objects through visual variability, cross-class similarity, dense layouts, heterogeneous content, and stricter localization requirements. These properties expose limitations in traditional features and generic detection designs.
- GUI characteristics: GUI elements exhibit large in-class variance and high cross-class similarity, making their visual boundaries and categories difficult to distinguish.Different classes can share shapes and text, while one class can vary in size, aspect ratio, texture, or style.
- GUI characteristics: GUI screens are packed with close-by widgets, images, and text rather than sparsely arranged physical objects.In the dataset, 77% of GUIs contain more than seven GUI elements, and neighboring elements may be separated by only small padding.
- Existing methods: Physical-world features such as Canny edges and contours do not directly match GUI shape and composition, making their aggregation error-prone.GUI images also contain image views whose contents should not be mistaken for separate GUI elements.
- Localization requirements: GUI detection requires highly accurate region localization because inaccurate boxes can impair classification and downstream testing or generated-code layouts.Generic object detection often accepts IoU>0.5, whereas GUI applications may require substantially tighter localization.
- Research questions: The study questions whether statistical bounding-box regression, anchor-box settings, training-data size, and generic architectures satisfy GUI detection requirements.It also asks whether document-oriented OCR is appropriate for cluttered GUI text or whether GUI text should be treated as scene text.
3 EMPIRICAL STUDY
The empirical study uses a large Rico-derived Android GUI dataset to compare representative old-fashioned and deep-learning detection methods.
- Dataset and methods: The study uses 50,524 Rico GUI screenshots extracted from 8,018 Android applications across 27 application categories.It systematically compares two old-fashioned methods, including REMAUI and Xianyu, with representative deep-learning approaches.
3.1 Research Questions
The study frames GUI detection around region-detection performance, sensitivity to deep-learning settings and data, and appropriate GUI text-detection methods.
- Research Questions: RQ1 asks how effectively methods detect non-text GUI regions in bounding-box accuracy and GUI-element coverage.The question focuses on the quality and completeness of predicted regions.
- Research Questions: RQ2 asks how sensitive deep-learning techniques are to anchor-box settings and training-data amount.This targets design choices that may affect model performance under GUI-specific variation.
- Research Questions: RQ3 examines the appropriate approach for detecting GUI text.The surrounding study compares alternatives for text detection because GUI text differs from ordinary document text.
3.2 Experiment Setup
The experiment uses Rico-derived Android GUI screenshots to compare old-fashioned, deep-learning, OCR, and scene-text detection methods under GUI-specific evaluation settings.
- The dataset contains 50,524 GUI screenshots from 8,018 Android applications, covering 15 commonly used GUI element types.
- The baselines include REMAUI and Xianyu for old-fashioned detection, Faster RCNN, YOLOv3, and CenterNet for deep learning, plus Tesseract and EAST for text detection.
- Faster RCNN, YOLOv3, and CenterNet are initialized from COCO-pretrained models and fine-tuned on the GUI training data.
- Region detection is evaluated with precision, recall, and F1-score at IoU thresholds, with each ground-truth box matched at most once using NMS.
3.3 Results - RQ1 Performance
RQ1 compares five methods for detecting non-text GUI regions, emphasizing the trade-off between coverage and high bounding-box accuracy across IoU thresholds.
- The five methods are trained on 40k GUI images and evaluated on 5k images using five-fold cross-validation.
- 31%, 45%, and 28% are the F1-score decreases for Faster RCNN, YOLOv3, and CenterNet respectively as IoU rises from 0.5 to 0.9.
- Old-fashioned methods decline less as IoU increases, but their overall F1-scores remain much lower than those of deep learning models.
- Deep learning models locate more elements in noisy backgrounds, while their statistically regressed bounding boxes can be less precise.
3.4 Results - RQ2 Sensitivity
RQ2 examines anchor-box choices and training-data size, finding limited sensitivity to anchor settings but substantial dependence on sufficient training data and model architecture.
- Anchor-Box Settings: Adding anchor-box scales and aspect ratios produces only a small F1 increase for Faster RCNN and YOLOv3.
- Anchor-Box Settings: 55% of Faster RCNN true positives and 67% of YOLOv3 true positives overlap across their two anchor-box settings.
- Anchor-Box Settings: Merging detections from different anchor-box settings can improve performance because the settings detect complementary bounding boxes.
- Amount of Training Data: All models lose performance as training data decreases from 40k to 10k or 2k images, while Faster RCNN remains comparatively stronger than the one-stage models.
- Amount of Training Data: Two-stage anchor-box models can achieve comparable performance with one magnitude less training data than one-stage models, whereas one-stage anchor-box models are hardest to train.
3.5 Results - RQ3 Text Detection
RQ3 evaluates unified versus separated text and non-text detection and compares OCR with scene-text detection for GUI text.
- Separated versus Unified Detection: Training one model on text and non-text elements degrades non-text detection compared with training on non-text elements alone.
- Separated versus Unified Detection: GUI text and non-text elements should be detected separately because their visual characteristics interfere when learned together.
- OCR versus Scene Text: The study concludes that GUI text resembles scene text, allowing a pretrained scene-text model to detect it without fine-tuning.
- OCR versus Scene Text: EAST achieves 0.402 precision, 0.720 recall, and 0.516 F1, exceeding Tesseract’s 0.291 precision, 0.518 recall, and 0.372 F1.
- OCR versus Scene Text: EAST detects GUI text in background images and low-contrast settings more reliably than Tesseract.
4 A NOVEL APPROACH
The approach combines GUI-specific old-fashioned detection for non-text elements with deep learning for classification and GUI text detection. Its top-down, coarse-to-fine design achieves stronger overall detection performance than the evaluated baselines, while remaining subject to widget-disambiguation and dense-layout failures.
- 4 A NOVEL APPROACH: The approach combines GUI-specific old-fashioned non-text region detection with deep learning for region classification and GUI text detection.It separates text and non-text detection, using a top-down coarse-to-fine strategy for non-text elements and mature deep learning models for classification and text detection.
- 4.1.1 Region Detection for Non-Text GUI Elements.: The method detects layout blocks through flood filling and rectangle recognition rather than assuming clear horizontal and vertical lines.Each rectangle region is treated as a block before subsequent binary segmentation and element-region detection.
- 4.1.1 Region Detection for Non-Text GUI Elements.: Connected component labeling identifies GUI element regions in binarized block segments and encloses each detected region with its smallest covering bounding box.The method uses two-pass scanning to label connected pixels and supports GUI elements of arbitrary shape.
- 4.2 Evaluation: 0.573 versus 0.388 in F1: the approach outperforms CenterNet for combined text and non-text element detection.For non-text elements, it also outperforms Faster RCNN, 0.523 versus 0.438 in F1; text performance is overall the same as EAST.
- 4.2 Evaluation: The method detects more GUI elements, fewer noisy regions, and more accurate, less-overlapping bounding boxes than the evaluated old-fashioned and deep learning baselines.The reported explanation emphasizes robust coarse-to-fine processing, GUI-specific image processing, and pixel-level analysis rather than statistical regression.
- 4.2 Evaluation: The model fails when similar-looking regions represent different widget types, when dense interfaces contain repetitive regions, and when text belongs to a widget rather than a label.The paper states that these challenges affect all methods and leaves them for future work.
- 4.2 Evaluation: 0.449 versus 0.315 in F1 for non-text elements and 0.524 versus 0.282 for all elements: the approach exceeds the strongest baseline results in overall object detection.These results measure true-positive bounding boxes with correct region classification over all detected element regions.
5 RELATED WORK
Related work applies GUI element detection across software engineering tasks and platforms, but existing approaches have important scope and representation limitations. The paper focuses on Android because the Rico dataset supports large-scale experimentation and labeling remains costly elsewhere.
- 5 RELATED WORK: GUI element detection supports GUI code generation, search, design examination, dataset construction, accessibility, testing, and security.Many of these software engineering tasks require locating GUI elements precisely.
- 5 RELATED WORK: Prior work includes GUI-specific and generic object detectors, plus Tesseract and EAST for GUI text detection.The empirical study compares representative old-fashioned methods, generic deep learning models, and text detectors.
- 5 RELATED WORK: Image captioning predicts which elements appear in a UI but not their bounding boxes, motivating pix2code's assumptions about fonts, shapes, sizes, and layouts.Those assumptions limit performance on real-app UIs with more diverse visual designs.
- 5 RELATED WORK: The experiments use Android app UIs because Rico enables large-scale data, while extending evaluation to other platforms or designs requires substantial manual labeling.The authors believe the model could generalize because it makes no Android-specific assumptions, but leave that evaluation for future work.
6 CONCLUSION
The paper finds that GUI element detection is challenging because GUIs combine distinctive visual characteristics with stringent localization requirements. Its large-scale study exposes weaknesses in borrowed computer-vision methods and motivates a GUI-aware approach that achieves state-of-the-art performance.
- 6 CONCLUSION: GUI element detection is challenging because GUIs exhibit large in-class variance, high cross-class similarity, packed elements, and heterogeneous objects.These characteristics make accurate detection difficult for both old-fashioned and deep learning methods.
- 6 CONCLUSION: The empirical study reveals underperformance in existing computer-vision methods and identifies effective designs for GUI element detection.The findings inform a new approach that accounts for GUI characteristics while incorporating effective existing designs.
- 6 CONCLUSION: The new GUI element detection method achieves state-of-the-art performance in the largest-ever evaluation of GUI element detection methods.The conclusion presents this result as the paper's primary outcome.