Scenario-Guided LLM-based Mobile App GUI Testing

Motivation

• Existing approaches struggle to generate logically coherent event sequences, contextually appropriate inputs, and semantically complete tests aligned with realistic user behavior.Their emphasis on event diversity, state transitions, or code coverage does not adequately represent functional or experiential depth.Source: §1 · pp. 2–3

ScenGen Framework

• ScenGen bridges high-level testing scenarios and low-level GUI actions by interpreting visual semantics and contextual testing intent with multimodal LLMs.The framework is designed around scenario-guided tasks rather than unconstrained exploration.Source: §1 · pp. 3, 5

Multi-Agent Architecture

• The Observer identifies GUI widgets, layouts, attributes, and semantic roles to provide structured perceptual information for subsequent testing decisions.Its perception combines computer vision and multimodal LLM techniques.Source: §1 · p. 4
• The Decider uses recognized GUI elements and scenario goals to select target widgets and generate an action plan instead of relying on random or purely structural exploration.This separates semantic reasoning from later operation execution.Source: §1 · p. 4
• The Executor follows the Decider’s plan for consistent, traceable, and repeatable interaction, while the Supervisor checks intermediate and final states against scenario requirements.This separation mirrors manual testing and supports verification of scenario completion.Source: §1 · p. 4

Contribution

• ScenGen integrates scenario-driven reasoning, localization refinement, iterative verification, and structured memory into a loop that produces reproducible GUI test artifacts.Its multi-agent collaboration simulates observing, interpreting, deciding, acting, and learning during testing.Source: §1 · pp. 5–6

2.1 Testing Scenario

• A testing scenario is a self-consistent business task expressed as a coherent sequence of GUI operations toward a concrete user goal.Examples include logging in, completing a purchase, or submitting a form.Source: §2.1 · p. 6
• Existing automated GUI testing often neglects scenarios, producing fragmented or redundant sequences that do not explicitly model app business logic or user goals.Scenario-aware methods are needed to reason about why interactions matter for specific business objectives.Source: §2.1 · pp. 6–7

2.2 Motivating Examples

• Hidden widgets, such as note-creation controls behind expandable menus, challenge methods that rely on surface-level GUI exploration.The relevant control may appear only after an intermediate interaction and disappear when the user clicks elsewhere.Source: §2.2.1 · p. 7
• Meaningful GUI testing may require domain knowledge, because calculator correctness depends on constraints such as division by zero and large-number handling.Conventional methods may struggle to generate inputs that exercise such domain-specific conditions.Source: §2.2.2 · p. 7
• Multi-step workflows such as login impose temporal and semantic dependencies, including filling valid fields before submitting.Random or model-based exploration may fail to preserve these logical dependencies.Source: §2.2.3 · p. 8

3 Methodology

• ScenGen coordinates context memory, Observer, Decider, Executor, Supervisor, and Recorder in an iterative process for scenario-based GUI testing.The agents integrate perception, reasoning, execution, verification, feedback, and recording.Source: §3 · pp. 8–9
• Persistent context memory supports controlled stepwise reasoning by retaining environment information, session history, recent GUI states, and widget-recognition results.Long-term memory stores stable app and scenario context, working memory tracks the evolving session, and short-term memory captures current and preceding GUI states.Source: §3 · p. 8, §3.1 · pp. 9–10
• A high-level scenario is transformed into reproducible GUI action sequences through explicit action traces, widget selections, execution, verification, and state recording.The framework preserves scenario intent and intermediate execution information rather than treating each interaction as an isolated prompt.Source: §3 · p. 8
• ScenGen externalizes agent interactions through publicly available prompt templates to improve testing reproducibility and transparency.The framework progressively refines its understanding of app functionality and testing intent through collaboration between LLM-driven and rule-based components.Source: §3 · p. 9
• GUI state comparison uses perceptual hashing and structural similarity to identify equivalent states and skip redundant exploration.States differing only in trivial animations, timestamps, or non-functional layout shifts are treated as identical.Source: §3.1 · p. 10
• The Observer combines deterministic computer-vision detection with multimodal semantic interpretation to produce precise, structured GUI representations.Its recognition pipeline identifies candidate widgets, merges related text and graphical regions, and filters duplicate, oversized, undersized, or irrelevant detections.Source: §3.2 · pp. 10–11

3.4 Executor & Recorder

• The Executor parses structured decisions and performs click, input, scroll, back, and related operations through system interfaces such as Android ADB.Input actions require target, text, and positional information, while scroll actions require direction and optionally a target region.Source: §3.4 · p. 15
• The Recorder captures each operation’s intent, action type, resolved widget attributes, execution parameters, screenshots, and resulting GUI state.These semantically enriched entries preserve traceability between high-level scenario intent and low-level execution.Source: §3.4 · pp. 15–16
• Completed action entries are serialized into JSON scripts with device metadata, enabling deterministic replay without the original LLM reasoning process.This externalizes generated behavior into inspectable artifacts that can be replayed, shared, and validated independently.Source: §3.4 · p. 16
• Deterministic execution and recording improve repeatability, traceability, and post-analysis of failures and system behavior.Separating rule-based execution from LLM-based reasoning helps prevent uncertain outputs from propagating into execution.Source: §3.4 · pp. 15–16
• The Supervisor adaptively waits for loading completion by analyzing GUI images for loading indicators and transitional states.This avoids relying on a fixed waiting time that may be inefficient or difficult to generalize.Source: §3.5 · pp. 16–17
• After execution, the Supervisor compares current and previous GUI states against the latest operation and target scenario to detect ineffective or unintended transitions.It can trigger self-correction for non-effective actions and contextual rollback for incorrect navigation or unwanted dialogs.Source: §3.5 · p. 17

4.1 Experiment Settings

• The evaluation defines five research questions covering scenario-guided test generation, bug detection, widget localization, logical decision-making, and efficiency.Together, these questions assess overall test quality, practical bug-finding utility, visual grounding, action validity, and computational or temporal cost.Source: §4.1 · pp. 18–19
• ScenGen uses GPT-4V through the gpt-4-vision-preview API for visual reasoning about GUI layouts and testing-scenario semantics.The model is selected for multimodal reasoning over screenshots and textual prompts.Source: §4.1 · p. 19
• Testing scenarios are derived from prevalent Google Play Store app categories and essential business functionalities, with experienced testers defining high-level descriptions.The framework autonomously derives operational sequences, so manual effort is limited to short textual scenario specifications.Source: §4.1 · pp. 19–20
• The study selects ten representative scenarios, 92 apps, and 99 app-scenario tasks, including both open-source and commercial apps.Seven apps are used in two testing scenarios, and the scenarios cover varied functional complexities.Source: Table 1 · p. 19, §4.1 · p. 20
• Baselines are selected by research question, including GPTDroid, ScenTest, and ASOT1 for scenario-guided generation and efficiency comparisons.Bug-detection evaluation uses representative automated GUI testing tools, including Monkey and Stoat.Source: §4.1 · p. 20

Evaluation Metrics

• Coverage Rate counts complete scenario coverage regardless of termination, whereas Success Rate requires both full coverage and successful completion.The evaluation classifies generations into four cases based on coverage and termination status.Source: §4.2 · p. 20

Results

• 86.87% Coverage Rate and 84.85% Success Rate across 99 tasks exceed all compared methods.GPTDroid, ScenTest, and ASOT achieve Coverage Rates of 74.75%, 77.78%, and 84.85%, and Success Rates of 68.69%, 69.70%, and 73.74%, respectively.Source: §4.2 · p. 21, Table 2 · p. 21
• ScenGen has fewer abnormal terminations and incomplete-coverage cases, indicating more stable and logically coherent scenario execution than baseline tools.The reported advantage is attributed to multi-modal LLM reasoning, Supervisor feedback, scenario understanding, and context memory.Source: §4.2 · p. 21

Failure Analysis

• Failure analysis identifies premature termination as the dominant failure type, accounting for 6 cases and 40% of observed failures.Other failures include inaccurate completion detection, widget localization errors, app-specific design variations, and unconventional processes.Source: §4.2 · p. 21
• Premature termination occurs when completion is judged before required operations finish, especially under implicit confirmations or visually similar intermediate states.The mechanism relies on screenshot comparison and semantic reasoning, which may miss implicit business-logic constraints; one translation test stopped before the final action.Source: §4.2 · p. 22
• Remaining limitations include ambiguous visual feedback, dense interfaces, heterogeneous implicit operations, and unconventional flows that challenge completion judgment and localization.The framework remains challenged by nuanced scenario completion reasoning and complex UI variability despite verification and localization refinements.Source: §4.2 · pp. 22–23

• Complex scenarios such as Translation, Shopping, and Login yield more bugs because multi-step interactions and asynchronous GUI transitions are more error-prone.Simple, heavily tested functions such as Calculation and Note produce fewer discoveries, while manual revalidation confirms the scenario findings.Source: §4.3 · pp. 23–24
• Traditional tools detect more bugs across entire apps, but ScenGen achieves stronger coverage and precision within the predefined testing scenarios.Random- and model-based methods explore broadly without prioritizing target workflows, making them less consistent on scenario paths.Source: §4.3 · p. 24
• LLM semantic understanding helps ScenGen open hidden or collapsed menus and reach target interaction paths that traditional tools often abandon.The framework’s multi-agent design also detects and corrects erroneous operations through the Supervisor, preserving scenario alignment and traceability.Source: §4.3 · p. 24
• ScenGen detects every bug found by baseline approaches while uncovering distinct, meaningful bugs across diverse apps and testing scenarios.Its scenario-guided testing systematically exercises realistic business-critical flows, although the detected bugs are mainly crash-related.Source: §4.3 · p. 24

• ScenGen integrates pixel-level computer vision with multimodal LLM semantic reasoning to localize widgets in visually complex, context-rich interfaces.Localization attempts are judged successful when the subsequent operation executes correctly and produces the expected GUI state transition.Source: §4.4 · p. 25
• 80.79% initial localization accuracy rises to 97.76% after self-correction, with every scenario exceeding 94% final accuracy.Final accuracy ranges from 94.29% to 100%, reaching 100% in four scenarios.Source: Table 4 · p. 25, §4.4 · p. 26
• In nine of ten scenarios, self-correction improves localization by 3.13% to 83.33%, demonstrating recovery from dynamic layouts and ambiguous widget boundaries.The mechanism combines CV feedback with LLM semantic re-analysis to correct initial misalignments.Source: §4.4 · p. 26
• Most initial localization errors involve occluded widgets or visually similar components, which self-correction resolves through contextual re-evaluation and refined coordinate predictions.Examples include drop-down or scrollable menus and repeated icons.Source: §4.4 · pp. 25–26

• Decision correctness requires an operation to advance or appropriately terminate the scenario, with unanimous agreement from three experienced evaluators.Evaluators judge only the scenario, GUI state, and generated decision, while normal operations and completion decisions are both treated as decision units.Source: §4.5 · pp. 26–27
• 96.82% average initial decision accuracy increases to 97.59% after self-correction across ten testing scenarios.Initial accuracy ranges from 90.14% to 100.00%, while final accuracy ranges from 92.96% to 100%; four scenarios are perfect initially.Source: §4.5 · p. 27, Table 5 · p. 27
• Self-correction compensates for transient decision errors caused by dynamic GUI updates or ambiguous widget states, improving reliability in three scenarios.Four operational errors are successfully corrected through iterative re-evaluation of the app state.Source: §4.5 · p. 27
• Among 12 incorrect decisions, premature termination is the most common error, followed by redundant continuation, widget-semantic confusion, missing prerequisites, and rare alternative paths.The errors include stopping before submission, confusing “Next” with “Submit,” and failing to open a required menu.Source: §4.5 · p. 27
• The Decider reliably captures scenario intent and translates it into executable GUI operations while remaining adaptable to reasoning inconsistencies.This conclusion follows the observed high accuracy and correction behavior across diverse app testing scenarios.Source: §4.5 · pp. 27–28

Token Overhead Analysis

• Token consumption ranges from 17,184 to 84,111 tokens and increases with interaction depth and operational complexity.Shopping and Login consume more tokens than simpler Photo and Alarm-clock scenarios because they require more steps and self-correction.Source: §4.6.1 · p. 28, §4.6.2 · p. 29
• GUI-state verification and test-completion verification generally consume the most tokens because they combine multi-page visual comparison with semantic reasoning.Logical decision-making and self-correction analysis consume fewer tokens because they typically process less visual information.Source: §4.6.1 · p. 28
• Structured visual inputs, including screenshots and annotated screenshots, reduce reliance on lengthy textual GUI descriptions and help control token growth.The modular workflow concentrates resources on information-intensive reasoning tasks rather than uncontrolled context accumulation.Source: §4.6.1 · p. 29, §4.6.2 · p. 30

4.6.2 Time Overhead Analysis

• 4.71 minutes is ScenGen’s average test-generation time, slower than ScenTest’s 1.72 minutes but faster than ASOT’s 10.38 and GPTDroid’s 7.48 minutes.The comparison reflects a trade-off between additional LLM and vision-language inference and scenario reasoning capability.Source: §4.6.2 · p. 29, Table 8 · p. 30
• ScenGen maintains more stable time consumption across application domains, completing complex Shopping and Login tasks within 4–5 minutes while baselines exceed 11 minutes.The reported stability is associated with adaptive decision control and dynamic context management.Source: §4.6.2 · pp. 29–30

Selection of Testing Scenarios and Apps

• Scenario and app selection may be limited or arbitrary, so the study samples diverse domains, interaction complexities, interface designs, and difficulty levels.Shopping and Login represent high-complexity scenarios with branching and state-dependent interactions, while other scenarios cover lower and medium difficulty levels.Source: §5 · pp. 30–31

Instability of LLMs

• LLM stochasticity can produce variable outputs even at fixed temperature, motivating ten repeated runs per scenario and aggregated rounded results.The repetition strategy is intended to filter random variation and highlight stable behavioral patterns.Source: §5 · p. 31

LLM Hallucinations

• LLM hallucinations may yield invalid actions such as targeting nonexistent widgets, so the Supervisor checks post-execution states and can initiate rollback.This verification mechanism is designed to detect deviations from the intended testing scenario.Source: §5 · p. 31

Context Management

• ScenGen limits context growth by treating each LLM invocation independently and selectively supplying structured memory, recent actions, and current GUI state.The authors report that current context windows were sufficient for longer tasks and that long-horizon limits were not a bottleneck in experiments.Source: §6.2 · pp. 31–32

Multi-Stage Reasoning

• Each LLM stage has a distinct role, while dependencies between decision, localization, execution, and verification make trivial merging undesirable.The architecture separates semantic intent resolution, pixel-level targeting, state validation, and completion judgment.Source: §6.3 · p. 32

Efficiency–Robustness Trade-off

• The modular design favors stable, verifiable test generation over minimizing query count, although its precise efficiency–robustness tradeoff remains future work.The authors state that semantic reasoning and state comparison dominate token use, while modular context scoping reduces reasoning-collapse risk.Source: §6.3 · p. 32

Testing-Oriented Design

• ScenGen is a testing-oriented multi-agent framework whose objective is executable, traceable scenario alignment rather than ordinary GUI task completion.Its mechanisms preserve scenario validity and support bug exposure along business-critical execution paths.Source: §6.4 · p. 33

Testing Artifacts

• Unlike typical GUI agents, ScenGen records actions, widgets, screenshots, and replayable commands while monitoring runtime logs for scenario-triggered faults.These artifacts support reuse and analysis as GUI tests rather than only producing a task-completion trajectory.Source: §6.4 · p. 33

LLM-Powered GUI Agents

• GUI-agent research has progressed from single-step interaction toward autonomous exploration, cross-environment generalization, and long-term stability, but primarily targets task execution.Examples include action-history reasoning, region-focused perception, unified action spaces, reinforcement learning, and history-aware reasoning.Source: §7.1 · pp. 33–34

Visual GUI Testing

• Visual GUI-testing methods improve screenshot understanding, widget recognition, replay robustness, cross-platform testing, and maintenance, yet mostly remain focused on perception and pattern recognition.The literature includes defect detection, widget identification, visual replay, script repair, duplicate-report handling, and bug inference.Source: §7.2 · pp. 34–35

LLMs for Software Testing

• LLMs have been applied to software testing through mutation-enhanced test generation, unit-test creation, contextual GUI inputs, question-answering, rare-value generation, crash reproduction, and program repair.These studies demonstrate applications across test generation, GUI testing, bug detection, and repair.Source: §7.3 · p. 35

Research Gap

• ScenGen builds on multimodal LLM capabilities to combine visual perception with testing-scenario reasoning and move beyond static event exploration toward context-aware test generation.Its motivation is the unresolved lack of scenario and business-logic reasoning in many existing approaches.Source: §7.3 · pp. 35–36, §7.2 · p. 35

Conclusion

• ScenGen aligns GUI interactions with testing scenarios and app business logic through semantic perception, scenario-aware decisions, execution, and iterative verification.Structured context management coordinates these functions across multiple GUI interactions.Source: §8 · p. 36
• The framework provides a practical foundation for scenario-guided GUI testing by combining LLM reasoning with testing-specific verification and execution mechanisms.The authors identify automatic scenario extraction, richer scenario modeling, and broader application scope as future directions.Source: §8 · p. 36