Vision-Based Mobile App GUI Testing: A Survey

Motivation

• GUI quality affects user experience, app behavior, and security, making GUI testing a significant part of mobile app quality assurance.GUI testing checks functionality, appearance, and interaction under complex components, fragmented environments, customized widgets, and user-operation simulation.Source: §1 · p. 2

Limitations of Traditional Testing

• Traditional code- and layout-based testing can fail across fragmented environments and miss GUI elements absent from runtime layout files.Canvas-contained widgets and refreshed contents can produce inaccurate identification and unreliable user-operation simulation.Source: §1 · p. 2

Vision-Based Shift

• Vision-based testing analyzes screenshots and rendered UI states to improve cross-environment understanding, GUI-element detection, and operation simulation.It uses visual information corresponding more closely to what users see rather than relying solely on structural or code representations.Source: §1 · pp. 3–4

Survey Scope and Taxonomy

• The survey compares mobile GUI-testing techniques by source-code requirement and approach basis while covering generation, replay, frameworks, maintenance, reporting, detection, usability, accessibility, and evaluation.Its taxonomy combines SWEBOK-informed top-down structuring with bottom-up analysis of recurring topics and challenges in the literature.Source: §1 · pp. 3–5

Contribution

• The survey focuses on vision-based GUI testing as an under-explored perspective and compares it systematically with traditional approaches across core dimensions.It aims to characterize the field’s evolution and identify future directions driven by visual reasoning.Source: §1 · pp. 3–4

Taxonomy Dimensions

• The survey classifies studies by black-box, gray-box, or white-box source-code requirements and by code-, layout-, image-, or hybrid approach basis.These dimensions capture differing testing needs, source access, and technological routes.Source: §2 · p. 5

Taxonomy Construction

• The hybrid taxonomy combines SWEBOK-informed top-down categories with bottom-up analysis of primary studies and secondary surveys.This approach aligns the classification with established software-engineering frameworks while reflecting recurring mobile GUI-testing themes.Source: §2 · p. 6

Classification Reliability

• Three authors independently labeled all 271 papers, senior authors resolved disagreements, and Fleiss kappa reached 0.95, indicating almost perfect agreement.The procedure was used to support the correctness of the paper-classification results.Source: §2 · p. 6

Observed Evolution

• Black-box techniques are used more often than white-box techniques, while layout-based methods followed early code-based approaches in the field’s development.Black-box testing fits APK-based integration testing, whereas layout files are accessible but can overlook specially designed widgets.Source: §2 · p. 6
• Image-based approaches are becoming a strong new force across mobile GUI-testing directions as computer vision improves widget identification from screenshots.CNNs and OCR can identify widget attributes such as type, attached text, and containment or relative relationships.Source: §2 · p. 6

3 AUTOMATED GUI TESTING

• Automated GUI testing covers test generation, record and replay, and frameworks that support test execution, management, and analysis.Test generation is a focused sub-function, whereas frameworks provide a broader end-to-end testing environment.Source: §3 · p. 7, §3.3 · p. 11

3.1 GUI Test Generation

• GUI test generation includes random-, model-, system-, and learning-based strategies, each trading off exploration, structure, runtime realism, adaptability, and resource requirements.Random testing can expose unforeseen problems but may generate redundant, unstructured, and inaccurate cases; learning-based methods require substantial data and suitable models.Source: Figure 2 · p. 7, §3.1 · pp. 7–8
• UI exploration remains an efficiency challenge because automated generation must cover many app functions within reasonable time while accounting for rendering delays and runtime overhead.Proposed directions include identifying exploration tarpits and scheduling events after the GUI is fully rendered.Source: §3.1 · pp. 8–9

3.2 GUI Test Record & Replay (Migration)

• Record-and-replay research has expanded from same-app regression testing to cross-device, cross-platform, and cross-app migration to address fragmentation and reduce test creation costs.Visual and learning-based techniques capture interface actions, transform event sequences, reuse tests across related apps, and convert screen recordings into replayable scenarios.Source: §3.2 · pp. 9–10

3.3 GUI Testing Framework

• GUI testing frameworks provide integrated environments for creating, executing, managing, and reporting tests, evolving from traditional data- or keyword-driven designs toward more adaptable visual technologies.This broader scope distinguishes frameworks from test generation, which focuses specifically on automatically producing effective tests.Source: §3.3 · pp. 11–12

4 AUTOMATION-ASSISTED MANUAL GUI TESTING

• Automation-assisted manual GUI testing supports complex scenarios requiring human intervention through script maintenance and automated test-report analysis.The two topics address adapting tests to evolving apps and extracting actionable information from results.Source: §4 · p. 12

4.1 GUI Test Script Maintenance

• Script generation, refactoring, updating, and repair improve test reuse, maintainability, synchronization, and reliability as mobile apps evolve.Vision-based scripts can use component images and user-visible feedback, while update and repair methods model GUI changes or widget structure and appearance.Source: §4.1 · p. 13

4.2 GUI Test Report Analysis

• Test-report analysis reduces manual effort by examining coverage, errors, and other results through deduplication, clustering, prioritization, enhancement, dialogue-based reporting, and crash reproduction.Recent methods combine screenshots, screen recordings, textual descriptions, computer vision, and language processing.Source: §4.2 · pp. 14–15
• Crowdsourced GUI testing broadens coverage across users, devices, and environments, but report processing remains a significant issue requiring specialized analysis.The survey emphasizes crowdsourced reports while also considering reports from other resources.Source: §4.2 · pp. 14–15

5.1 GUI Element Detection

• GUI element detection supports test generation, record and replay, and interaction by identifying and locating buttons, text fields, menus, and other interface components.Vision-based detection also extends testing to games, aerospace control apps, sensitive icons, and other specialized scenarios.Source: §5.1 · pp. 15–17
• Visual techniques classify screenshot components, detect display issues, and identify GUI changes across app releases when automatic localization is challenging.Examples use CNNs, screenshot modeling, computer vision, and natural-language generation.Source: Figure 6 · p. 16

5.2 GUI Usability and Accessibility Testing

• Usability and accessibility testing must account for diverse devices, environments, impairments, display conditions, and interaction preferences that affect GUI perception and use.Reported approaches detect animation and rendering problems, scaling inconsistencies, image-button labels, and text-accessibility issues.Source: §5.2 · p. 17

5.3 GUI Testing Evaluation Criteria

• GUI testing evaluation research develops comparison frameworks, standardized metrics, component-level units, multi-level criteria, industrial assessments, and user-experience evaluations.These efforts aim to make testing quality and performance measurable, comparable, and useful for method selection.Source: §5.3 · pp. 17–18

6.1 GUI Element Detection and Semantics Understanding

• GUI element recognition must handle heterogeneous, dynamic, asynchronous, overlapping, and resolution-varying widgets while integrating reliably with end-to-end testing tools.Image-based methods can suffer mismatches from quality, similarity, and deformation, whereas AI-based methods require labeled data and computational resources.Source: §6.1 · p. 18
• Multimodal fusion, knowledge graphs, and cloud computing are proposed to improve GUI element semantics, training efficiency, scalability, and adaptive testing.These techniques are intended to support automated test generation, execution, and script maintenance across complex GUI environments.Source: §6.1 · p. 19

6.2 Oracle Generation for GUI Testing

• Visual oracles struggle with subtle GUI bugs because recognition models often emphasize prominent content and filter small defects as noise.Semantic, context-aware, adaptive, user-informed, and cross-platform oracle generation is proposed to improve expected-result accuracy as apps change.Source: §6.2 · pp. 19–20

6.3 Expert Domain Knowledge Integration for GUI Testing

• Expert-knowledge integration remains limited by incomplete or inconsistent knowledge, difficult maintenance, cross-domain coordination, and conflicts among multiple testing goals.Knowledge graphs, collaborative platforms, and version control are proposed to improve coordination, traceability, and adaptability.Source: §6.3 · pp. 20–21

6.4 App Scenario Identification, Segmentation, Combination, and Understanding

• Complex app scenarios combine diverse actions, device states, network conditions, and user paths, causing exponential combinations and difficult coverage.Proposed directions include CV-NLP scenario discovery, data-mining-based navigation analysis, dynamic segmentation, reinforcement learning, and cross-scenario test suites.Source: §6.4 · pp. 21–22

6.5 Large Language Model Utilization in GUI Testing

• LLMs can generate test cases, scripts, and reports, potentially connecting GUI test generation with record-and-replay and scenario-based testing.Generated scripts still require maintenance and optimization in practical use.Source: §6.5 · p. 22
• LLM multimodal fusion can combine visual and language information for broader GUI testing, while GPT-4(Vision) is described as precisely captioning app GUI screenshots.The section presents Fig. 7 as illustrating GPT-4(Vision)'s caption-generation capability compared with non-LLM captioning models.Source: §6.5 · pp. 22–23, Figure 7 · p. 23
• LLMs are presented as having broad prospects for more intelligent and efficient vision-based GUI testing, but their application remains an area for further research.The survey also identifies challenges in interpreting outputs, adapting models to domains, matching generated tests to expected behavior, and achieving high coverage.Source: §6.5 · pp. 22–23

• The survey covers eight GUI-testing topics spanning generation, execution, maintenance, and analysis, while examining how computer vision benefits vision-based techniques.It compares vision-based approaches with code- and layout-based methods across the broader mobile GUI-testing landscape.Source: §7 · p. 23
• Vision-based testing directly detects GUI elements from screenshots, reduces dependence on source code or layout files, adapts across environments and devices, and supports user-action execution.Source: §7 · p. 23
• The survey also discusses remaining effectiveness and efficiency limitations and proposes research directions using emerging techniques.Source: §7 · p. 23

Paper collection

• The collection process uses explicit inclusion criteria, database searches, automated metadata retrieval, deduplication, filtering, validation, and snowballing to build a representative corpus.Searches cover DBLP and Google Scholar, while snowballing targets relevant studies missed by keyword search.Source: A SURVEY METHODOLOGY · pp. 36–37

Corpus construction

• The final dataset contains 271 research papers, with its topical distribution illustrated in Fig. 9 and the included-study list released publicly.Source: A SURVEY METHODOLOGY · p. 37, Figure 9 · p. 37

Vision-based classification

• Vision-based studies are identified during screening by their reliance on visual inputs, allowing them to be analyzed within the broader mobile GUI-testing corpus.The search deliberately avoids requiring the inconsistently used term “vision-based.”Source: A SURVEY METHODOLOGY · pp. 37–38

Research categorization

• The taxonomy organizes eight topics into automated testing, automation-assisted manual testing, and fundamental GUI-testing techniques.Automated testing includes generation, record and replay, and frameworks; assisted testing includes script maintenance and report analysis; fundamental techniques include element detection, usability and accessibility, and evaluation criteria.Source: A SURVEY METHODOLOGY · p. 38

Topic distribution

• GUI test generation has the most studies at 62, while record and replay and testing frameworks have 34 and 56 papers, respectively.Element detection has many vision-based studies, whereas script maintenance, usability and accessibility, and evaluation criteria have 16, 11, and 19 papers, respectively, with few vision-based studies.Source: A SURVEY METHODOLOGY · p. 38

Temporal trends

• The number of GUI-testing studies rose sharply after around 2012, and vision-based studies began increasing around 2016 alongside advanced deep-learning networks.Source: A SURVEY METHODOLOGY · p. 39, Figure 10 · p. 39

Venue distribution

• Most publications appear in software-engineering venues, with a high percentage of vision-based studies in top-tier venues such as TSE, ICSE, and ASE.The Venue-Year matrix uses colors for 1–2, 3–4, and more than 4 papers, and vision-based studies cluster in recent top-tier software-engineering venues.Source: A SURVEY METHODOLOGY · pp. 39–40, Table 2 · p. 40

Scope boundary

• The corpus is bounded by a collection deadline at the end of September 2024 because some venues had not yet published their 2024 proceedings.Source: A SURVEY METHODOLOGY · p. 39

Topic taxonomy

• The classification lists topic-specific approach clusters for GUI test generation, record and replay, testing frameworks, script maintenance, report analysis, element detection, usability and accessibility, and evaluation criteria.Source: B PAPER CLASSIFICATION OF DIFFERENT… · p. 40

GUI test generation

• GUI test-generation studies are grouped into random-, model-, system-, and learning-based approaches, including work addressing testing-efficiency challenges.Source: B PAPER CLASSIFICATION OF DIFFERENT… · p. 40, Table 3 · p. 41

GUI test record & replay

• GUI test record-and-replay studies address migration within an app and across devices, platforms, and apps, including event matching.Source: B PAPER CLASSIFICATION OF DIFFERENT… · p. 40, Table 4 · p. 41

GUI testing framework

• Framework studies are divided between traditional-technology-based approaches and innovation-approach-driven approaches.Source: B PAPER CLASSIFICATION OF DIFFERENT… · p. 40

Assisted manual testing

• Script-maintenance studies cover test-script generation, refactoring, repair, and updating, while report-analysis studies cover augmentation, captioning, clustering, duplication detection, prioritization, assistance, quality detection, and reproduction.Source: B PAPER CLASSIFICATION OF DIFFERENT… · p. 40

Fundamental techniques

• Element-detection studies include pure detection tasks and domain applications, usability and accessibility studies receive dedicated coverage, and evaluation studies address comparisons, criteria, and other evaluation topics.Source: B PAPER CLASSIFICATION OF DIFFERENT… · pp. 40–41

GUI Test Generation

• LLMs can support GUI test generation by exploring app pathways adaptively, simulating user behaviors, discovering untested scenarios, and reducing manual exploratory-test creation.Their exploration can respond to unexpected behavior or GUI changes, improving coverage for complex applications.Source: C LLM APPLICATION IN DIFFERENT GUI… · p. 43

GUI Test Record & Replay

• LLMs can make GUI test record and replay more robust by interpreting interactions, generating readable scripts, and adapting to repositioned buttons or updated labels.The approach preserves test-script usefulness after GUI layout or workflow changes.Source: C LLM APPLICATION IN DIFFERENT GUI… · pp. 43–44

GUI Testing Framework

• LLMs can improve GUI testing frameworks by dynamically adapting execution to interface changes and streamlining management through reports, result categorization, and coverage insights.This combines interface-aware execution with automated test-management outputs.Source: C LLM APPLICATION IN DIFFERENT GUI… · p. 44

GUI Test Script Maintenance

• LLMs can reduce GUI test-script maintenance by detecting interface changes, updating scripts, and suggesting fixes from previous failures, lowering manual effort and maintenance costs.The stated goal is greater reliability and longevity as mobile apps evolve.Source: C LLM APPLICATION IN DIFFERENT GUI… · p. 44

GUI Test Report Analysis

• LLMs can analyze GUI test reports by identifying recurring failures, prioritizing issues, correlating failures with recent app changes, and suggesting troubleshooting actions.They can also summarize large automated-test outputs in natural language.Source: C LLM APPLICATION IN DIFFERENT GUI… · p. 44

GUI Element Detection

• LLMs can detect GUI elements more flexibly by combining visual and semantic cues, handling changes in layout, style, or functionality without predefined identifiers.This is intended to reduce false positives and missed elements in automated testing and interaction simulation.Source: C LLM APPLICATION IN DIFFERENT GUI… · p. 44

GUI Usability / Accessibility Testing

• For usability and accessibility testing, LLMs can assess clarity, consistency, navigation, contrast, alt text, font sizes, and interactions from users with disabilities.The described analyses are grounded in established usability and accessibility guidelines.Source: Table 9 · p. 43, C LLM APPLICATION IN DIFFERENT GUI… · p. 44

GUI Testing Evaluation Criteria

• LLMs can evaluate test effectiveness through coverage analysis and additional-test recommendations, while interpreting failures across device configurations and monitoring responsiveness and GUI quality.These capabilities aim to help testers locate defects faster and more accurately.Source: C LLM APPLICATION IN DIFFERENT GUI… · pp. 44–45

D.1 Approach Basis Classification

• Vision-based studies primarily use image processing, computer vision, or screenshot analysis, whereas non-vision-based studies rely on traditional methods such as code analysis or event-flow testing.Three authors classified papers independently, two others reviewed them, disagreements were discussed, and a consistency test assessed agreement.Source: §D.1 · p. 45

D.2 Paper Selection Process

• The paper expands an initial database search through snowballing related-work sections to identify relevant studies beyond keyword results.The two-phase process uses defined inclusion criteria followed by iterative expansion.Source: §D.2 · p. 45
• The corpus focuses on papers from six journals and fourteen top-tier conferences in software engineering, programming languages, and human-computer interaction.These venues are described as major and visible forums for the target research area.Source: §D.2 · p. 45
• The survey excludes general-purpose AI, machine-learning, and deep-learning studies unless they directly address mobile GUI testing challenges.Applied methods such as screenshot classification and visual bug detection remain in scope.Source: §D.2 · p. 45
• Systematic search, iterative expansion, and documented inclusion boundaries are used to support a diverse, representative, and focused survey corpus.The stated design balances broad coverage with a clear scope.Source: §D.2 · p. 45