PaperlayerView exact revision on arXiv

Source-linked AI summary

Make LLM a Testing Expert: Bringing Human-like Interaction to Mobile GUI Testing via Functionality-aware Decisions

Zhe Liu, Chunyang Chen, Junjie Wang, Mengzhuo Chen, Boyu Wu, Xing Che, Dandan Wang, Qing Wang

arXiv:2310.15780v1cs.SE

TL;DR

Mobile GUI testing methods can have low coverage, limited generalization, and substantial training-data requirements. GPTDroid treats testing as an interactive Q&A process, using GUI context, app feedback, and functionality-aware memory to guide an LLM. Across 93 apps, it achieves 32% higher activity coverage and detects 31% more bugs than the best baseline, while finding 53 new Google Play crash bugs.

  • Problem

    Existing GUI testing approaches suffer from low coverage, limited generalization to unseen situations, and heavy reliance on training data, risking missed app issues.

  • Method

    GPTDroid prompts an LLM with GUI information, executes its generated operations, feeds back app responses, and uses functionality-aware memory for long-term testing decisions.

  • Results

    32% higher activity coverage than the best baseline was achieved across 93 apps, alongside 31% more bugs detected faster and 53 new Google Play crash bugs.

  • Takeaways & Limitations

    GPTDroid demonstrates that an LLM-based, functionality-aware interaction process can improve mobile GUI coverage and bug detection in the evaluated apps.

  • Takeaways & Limitations

    Some uncovered cases involve widgets lacking meaningful text or resource IDs, while database connections, long presses, and fixed-location drags are difficult to automate.

Abstract

from arXiv · show

Automated Graphical User Interface (GUI) testing plays a crucial role in ensuring app quality, especially as mobile applications have become an integral part of our daily lives. Despite the growing popularity of learning-based techniques in automated GUI testing due to their ability to generate human-like interactions, they still suffer from several limitations, such as low testing coverage, inadequate generalization capabilities, and heavy reliance on training data. Inspired by the success of Large Language Models (LLMs) like ChatGPT in natural language understanding and question answering, we formulate the mobile GUI testing problem as a Q&A task. We propose GPTDroid, asking LLM to chat with the mobile apps by passing the GUI page information to LLM to elicit testing scripts, and executing them to keep passing the app feedback to LLM, iterating the whole process. Within this framework, we have also introduced a functionality-aware memory prompting mechanism that equips the LLM with the ability to retain testing knowledge of the whole process and conduct long-term, functionality-based reasoning to guide exploration. We evaluate it on 93 apps from Google Play and demonstrate that it outperforms the best baseline by 32% in activity coverage, and detects 31% more bugs at a faster rate. Moreover, GPTDroid identify 53 new bugs on Google Play, of which 35 have been confirmed and fixed.

1 INTRODUCTION

GPTDroid reframes mobile GUI testing as an interactive Q&A task in which an LLM generates and revises operations using app feedback and functionality-aware memory. On 93 Android apps, it improves coverage and bug detection over established baselines while revealing practical constraints in GUI understanding and specialized interactions.

  • Existing GUI testing methods can suffer from low testing coverage because mobile apps contain many dynamic screens and generated traces differ from real-user interactions.
  • DL- and RL-based methods also require substantial training data, may generalize poorly to unseen app states, and face nondeterministic app behavior.
  • GPTDroid formulates GUI testing as interactive Q&A, passing GUI information to an LLM, executing generated scripts, and feeding app feedback back for iterative decisions.
  • Functionality-aware memory records explored activities, widgets, and function-level progress so the LLM can reason globally and choose meaningful operation sequences.
  • 32% higher activity coverage and 20% higher code coverage than the best baseline yielded 75% activity coverage and 66% code coverage across 93 apps.
  • 31% more bugs were detected faster than by the best baseline, while 53 new Google Play crash bugs were found and 35 were confirmed and fixed.

2 APPROACH

GPTDroid uses nested loops to translate current GUI context and accumulated testing knowledge into LLM prompts, then execute the LLM’s next operation. Its inner memory loop preserves detailed interaction history and functionality-level progress for long-term reasoning.

  • GPTDroid models GUI testing as a Q&A problem in which the LLM acts as a human tester interacting with the app.
  • Outer loop: The outer loop extracts the current GUI context, prompts the LLM, converts its response into executable operation scripts, and repeats after app interaction.
  • Inner loop: The inner loop stores detailed activities, widgets, and functionality-level testing progress in a testing sequence memorizer.
  • Functionality-aware memory: Functionality-aware memory prompts combine with GUI prompts to help the LLM retain whole-test knowledge and query the next operation.

2.1 GUI Context Extraction

GPTDroid extracts GUI context at app, page, and widget levels to give the LLM both global perspective and actionable details about the current interface.

  • GUI context: GUI context combines app information, current-page information, and all page widgets extracted from AndroidManifest.xml and view hierarchy files.
  • App information: App information supplies macro-level semantics, including the app name and all activity names, for a general view of available functions.
  • Page GUI information: Page GUI information describes the current activity, its widgets, and widget positions to represent the present testing snapshot.
  • Widget information: Widget information captures text, hint-text, resource-id, class, clickability, and nearby parent or sibling text to support actionable operations.

2.2 GUI Prompting and Executive Command Generation

GPTDroid converts extracted GUI and testing information into structured prompt patterns, then maps LLM responses to a small set of executable mobile operations. Separate prompts support starting, routine testing, and error recovery.

  • 2.2 GUI Prompting and Executive Command Generation: Six linguistic patterns describe app, page, widget, operation, and feedback information for GUI testing prompts.
  • 2.2.1 Linguistic Patterns of GUI Prompt: GUI prompt patterns separately represent app information, page information, and widget information from different viewpoints.
  • 2.2.1 Linguistic Patterns of GUI Prompt: Operation prompts ask what action is required, while feedback prompts report invalid widget correspondence and request a regenerated operation.
  • 2.2.2 Prompt Generation Rules: Start prompts provide all activities for a global overview, whereas test prompts describe the current page and widgets to query the next operation.
  • 2.2.3 Executive Command Generation: In-context demonstrations and output templates help map natural-language LLM responses to executable commands.
  • 2.2.3 Executive Command Generation: GPTDroid supports click, double-click, long press, scroll, and input, dividing them into action and input categories.

2.3 Functionality-aware Memory Prompting

GPTDroid augments LLM-driven GUI testing with functionality-aware memory, preserving detailed interaction history and function-level progress to guide meaningful operation sequences.

  • 2.3 Functionality-aware Memory Prompting: GPTDroid combines context extraction, GUI prompting, command generation, and functionality-aware memory prompting for automated GUI testing.
  • 2.3.1 Functionality-level Progress: The system queries the LLM about the currently tested functionality and completion status, representing progress as <Function name> + <Status>.
  • 2.3.1 Functionality-level Progress: GPTDroid seeds the app’s functionality list from its description file and activity names, then refines that information throughout iterative testing.
  • 2.3.2 Testing Sequence Memorizer: The testing sequence memorizer records tested functions, activity paths, activity visits, and current-page widget visits.
  • 2.3.2 Testing Sequence Memorizer: During each operation, the memorizer stores function status, widget operations, and activity transitions, updating widget visits through text and resource-id matching.
  • 2.3.3 Functionality-aware Memory Prompt: The memory prompt combines explored functionalities, covered activities, and recently tested operations, retrieving updated information from the memorizer at every iteration.
  • 2.3.3 Functionality-aware Memory Prompt: The explored-functionality pattern reports explored functions, exploration counts, and completion status, adding high-level information absent from the current-page GUI prompt.
  • 2.3.3 Functionality-aware Memory Prompt: The covered-activities pattern summarizes activity sequences and visits, while the recent-operations pattern supplies widget visits and the latest five operation pages.

2.4 Implementation

GPTDroid is implemented as an automated GUI testing tool using Android virtualization, UI hierarchy extraction, and app interaction components, with ChatGPT as its LLM.

  • 2.4 Implementation: GPTDroid uses VirtualBox and pyvbox to run and control Androidx86, Android UIAutomator to extract view hierarchies, and ADB to interact with apps.
  • 2.4 Implementation: The implementation uses the pre-trained ChatGPT model for its LLM component.
  • 2.4 Implementation: The basic ChatGPT model identified for the implementation is gpt-3.5-turbo.

3 EFFECTIVENESS EVALUATION

GPTDroid is evaluated on 93 Android apps using coverage, bug detection, and ablation analyses against established automated GUI testing baselines. It achieves substantially higher coverage and bug detection, while its GUI context and functionality-aware memory contribute strongly to performance.

  • 3.1 Experimental Setup: The evaluation uses 93 apps, combining 20 Themis apps with 34 bugs and 73 additional apps with 109 bugs.The additional apps were selected from Google Play and retained confirmed crash bugs for effectiveness evaluation.
  • 3.1 Experimental Setup: GPTDroid is compared with 10 random-, rule-, model-, and learning-based GUI testing techniques plus integrated enhancement baselines.The baselines include Monkey, Droidbot, TimeMachine, WCTester, Stoat, Ape, Fastbot, ComboDroid, Humanoid, and Q-testing, alongside integrations such as Droidbot+QT and Stoat+TO.
  • 3.3 Results and Analysis: 32% higher activity coverage and 20% higher code coverage than the best baseline yield 75% activity coverage and 66% code coverage across 93 apps.Against Ape with QTypist, activity coverage is 0.75 versus 0.57.
  • 3.3.1 Performance of Coverage (RQ1): GPTDroid maintains higher activity coverage at every measured time point and reaches high coverage within about 24 minutes.This supports its effectiveness under a limited testing budget.
  • 3.3.2 Bug Detection (RQ2): 31% more bugs are detected than by the best baseline, with 95 bugs versus 66, while the highest value is reached in about 27 minutes.The reported testing time is 17 versus 26 minutes compared with Stoat with Toller.
  • 3.3.3 Ablation Study (RQ3): Removing GUI context reduces activity coverage by 77%, while removing Function Memory reduces it by 55%.The results associate GUI context with semantic page understanding and Function Memory with retaining testing knowledge and reaching uncovered areas.
  • 3.3.3 Ablation Study (RQ3): Removing explored functionalities causes the largest ablation decline, reducing activity coverage by 51% from 0.75 to 0.37.The authors link this module to functionality-level progress awareness and exploration planning.
  • 3.3.3 Ablation Study (RQ3): Coverage increases through five latest tested pages and operations, then gradually decreases as more pages are included.This supports the selected five-step history length.

4 USEFULNESS EVALUATION

GPTDroid is tested for usefulness in finding previously unseen crash bugs across 223 Google Play apps. It detects 53 new bugs, including 35 confirmed or fixed by developers, with many involving deep, compound, or business-logic behavior.

  • 4.1 Experimental Setup: The usefulness evaluation filters 317 popular, recently updated apps to 223 apps with available bug-reporting channels.Unlike the effectiveness evaluation, issue records or pull requests are not required.
  • 4.2 Results: GPTDroid detects 53 newly detected bugs involving 41 apps, and only 9 of these bugs are also detected by three baselines.Overall, it detects 135 bugs involving 115 apps.
  • 4.2 Results: 35 of the 53 submitted new bugs have been confirmed or fixed by developers, comprising 20 fixed and 15 confirmed bugs.The remaining bugs are pending, and none has been rejected.
  • 4.2 Results: 17 bugs involve multiple text inputs or compound operations, 11 require more than 20 operations, and at least 28 concern main business logic.These findings characterize the depth and functional complexity of the detected bugs.

5 INSIGHTS FROM EXPERIMENT RESULTS

The experiments attribute GPTDroid’s performance to functionality-aware exploration, prioritization, valid text input, and compound actions. These capabilities help it preserve long-term testing context, target important functions, and execute semantically meaningful interactions.

  • Functionality-aware exploration through the long meaningful testing trace: Long meaningful test traces can complete business-logic sequences whose only execution path reveals a crash.The SmartMeter example requires adding, starting, turning off, long-pressing, and deleting equipment.
  • Functionality-aware exploration through the long meaningful testing trace: Functionality-aware memory captures long-term dependencies among GUI pages to guide exploration across multi-step features.The mechanism supports tracing a function across the sequence of pages and operations.
  • Function-aware prioritization: GPTDroid prioritizes important functions, helping reach more key activities with relatively less time.The Moni example contrasts selecting AddIncome first with baseline exploration that enters a Setting-page cycle.
  • Valid text inputs: GPTDroid generates valid semantic text for single and correlated multiple input widgets, such as departure and arrival cities and dates.Valid inputs allow testing to pass pages that require meaningful values.
  • Compound actions: GPTDroid executes compound operations by combining text entry, date selection, button adjustment, and submission.Few-shot learning and output templates map the LLM output to widget actions.

6 RELATED WORK

Automated GUI testing has progressed from random and model-based exploration toward learning-based and LLM-assisted approaches, but coverage and generality remain central concerns. GPTDroid differs from text-only assistance by using an LLM to propose diverse app interactions as a complete testing tool.

  • Automated GUI testing: Monkey emits pseudo-random UI and system events but cannot formulate testing paths according to app characteristics, resulting in low coverage.
  • Automated GUI testing: Model-based tools improve test coverage, yet remain limited because they do not consider semantic information from the app’s GUI and pages.
  • LLM for Software Engineering: QTypist generates text inputs to pass GUI pages, whereas GPTDroid proposes broader interactions as a complete GUI testing tool.
  • LLM for Software Engineering: GPTDroid supports actions including button clicks, text filling, and more complicated compound interactions for wild mobile app testing.

7 CONCLUSION

The paper formulates mobile GUI testing as an iterative Q&A process between an LLM and an app, using GUI context and functionality-aware memory to guide actions. On 93 popular apps, GPTDroid achieved higher activity coverage, detected more bugs faster, and uncovered new Google Play bugs, many later confirmed or fixed.

  • GPTDroid encodes GUI context and functionality-aware memory into prompts, decodes LLM feedback into operations, executes them, and iterates.
  • 75% activity coverage was achieved on 93 popular apps, 32% higher than the best baseline.
  • GPTDroid detected 31% more bugs with faster speed than the best baseline.
  • 53 new Google Play bugs were detected, with 35 confirmed and fixed.
Loading 2310.15780v1…