PaperlayerView exact revision on arXiv

Source-linked AI summary

Fill in the Blank: Context-aware Automated Text Input Generation for Mobile GUI Testing

Zhe Liu, Chunyang Chen, Junjie Wang, Xing Che, Yuekai Huang, Jun Hu, Qing Wang

arXiv:2212.04732v1cs.SE

TL;DR

Automated GUI testing is limited by the challenge of generating semantically appropriate text inputs needed to advance through mobile app interfaces. QTypist uses contextual prompts, a pre-trained LLM, and prompt-based data construction and tuning; across 106 Google Play apps, it achieved an 87% passing rate and improved testing coverage and bug detection when integrated with GUI testing tools.

  • Problem

    Generating semantically appropriate text inputs remains challenging for automated GUI testing and can prevent tools from proceeding to subsequent UI pages.

  • Method

    QTypist uses local and global GUI context to construct prompts for a pre-trained LLM, with prompt-based data construction and tuning for semantic text input generation.

  • Results

    87% passing rate on 106 Google Play apps, 93% higher than the best baseline; integrated tools covered up to 42% more activities, 52% more pages, and detected 122% more bugs.

  • Takeaways & Limitations

    Integrating QTypist with automated GUI testing tools increased covered activities and pages and helped reveal more confirmed crash bugs.

  • Takeaways & Limitations

    The study's external validity is constrained by the representativeness and generality of its experimental apps.

Abstract

from arXiv · show

Automated GUI testing is widely used to help ensure the quality of mobile apps. However, many GUIs require appropriate text inputs to proceed to the next page which remains a prominent obstacle for testing coverage. Considering the diversity and semantic requirement of valid inputs (e.g., flight departure, movie name), it is challenging to automate the text input generation. Inspired by the fact that the pre-trained Large Language Model (LLM) has made outstanding progress in text generation, we propose an approach named QTypist based on LLM for intelligently generating semantic input text according to the GUI context. To boost the performance of LLM in the mobile testing scenario, we develop a prompt-based data construction and tuning method which automatically extracts the prompts and answers for model tuning. We evaluate QTypist on 106 apps from Google Play and the result shows that the passing rate of QTypist is 87%, which is 93% higher than the best baseline. We also integrate QTypist with the automated GUI testing tools and it can cover 42% more app activities, 52% more pages, and subsequently help reveal 122% more bugs compared with the raw tool.

I. INTRODUCTION

Mobile GUI testing often fails at semantically constrained text fields, because incorrect or context-free inputs prevent apps from reaching subsequent pages. QTypist addresses this gap by using a context-aware LLM workflow and improves input validity and testing coverage.

  • Motivation: Text input generation is challenging because fields require specific values and may impose relationships with other inputs.Examples include addresses for navigation apps, movie names for media apps, and minimum values smaller than maximum values.
  • Motivation: Without valid text inputs, automated GUI testing tools cannot proceed to subsequent pages, reducing testing adequacy and missing potential bugs.
  • Existing approaches: Most existing methods use fixed strings, random words, heuristics, or manually labeled RNN-based generation that does not generalize well or use contextual information.
  • QTypist: QTypist extracts prompts and answers automatically for prompt-based model tuning and generates prompts from GUI context for LLM input.
  • Results: 87% passing rate was achieved by QTypist, representing a more than 93% boost over the best baseline in the reported experiment.The evaluation used 168 text inputs from 106 popular Android apps and compared QTypist with 11 baselines.
  • QTypist: QTypist formulates text input generation as a cloze-style fill-in-blank task and uses a pre-trained LLM to infer semantic inputs from GUI context.

II. MOTIVATIONAL STUDY AND BACKGROUND

The study examines real-world mobile-app text inputs and constructs an experimental basis from Android UI data. Its dataset spans many apps and pages, with filtering and manual checks used to identify candidate text-input apps.

  • Study design: The pilot study investigates how prevalent text inputs are and identifies input categories to guide information extraction and text generation.
  • Dataset: The source dataset is Rico, an Android UI dataset containing screenshots and corresponding view hierarchy files from diverse app categories.
  • Dataset: The researchers filter 7,000 apps using text-input keywords, obtaining 5,663 candidate apps with potential text inputs before manual verification.

2) The Categories of Text Inputs:

The study categorizes mobile text inputs into five content types and highlights both semantic requirements and cross-field constraints. About 73% of apps contain pages requiring text input to proceed.

  • Categorization: The researchers manually categorize text inputs from 2,600 apps using open coding and researcher review of app, activity, input type, and input content.
  • Identity: Identity inputs account for 27% and commonly represent personal information such as usernames, family details, email, or jobs.
  • Geography: Geography inputs account for 17% and include locations, addresses, and countries, with constraints such as distinct departure and arrival addresses.
  • Numeric: Numeric inputs account for 12% and include values such as years, weights, income, and blood glucose, often with ordering constraints.
  • Query: Query inputs account for 33% and require meaningful entities such as games, shopping items, music, or films.
  • Comment: Comment inputs account for 11% and include reviews, remarks, notes, diaries, and code, whose content depends on the app category.
  • Implications: About 73% of apps contain pages requiring text input, making appropriate inputs an obstacle to reaching app activities.

B. Background of Large Language Model

Large language models provide a basis for generating text from prompts, but mobile input generation requires prompts that encode GUI context. QTypist constructs such prompts from widget, local, and global information.

  • LLM background: Pre-trained LLMs such as BERT, GPT-3, and T5 encode broad information from billions of Internet samples and can support language tasks through prompts.
  • LLM background: GPT-3 uses transformer components including embeddings, masked multi-self attention, normalization, and feed-forward layers to process sentences.
  • QTypist: QTypist uses a pre-trained LLM to generate semantic input text from prompts built from GUI-page context and tunes the model with automatically constructed prompt-answer data.
  • Context extraction: QTypist extracts widget information from hint-text, resource-id, or text fields, selecting the first non-empty source.
  • Context extraction: Local context combines nearby widgets and page fragments, while global context includes the activity name, app name, and number of input widgets.
  • Prompt generation: The extracted information is preprocessed and organized into linguistic patterns that become prompts for the LLM.

2) Prompt Generation:

QTypist generates context-aware prompts by combining widget, local-page, and global-app information, while adapting patterns to input categories and multi-widget pages.

  • Prompt Generation: Input categories guide prompt design because diversified content affects what the LLM should generate.Five glossaries, including location and search, are matched against activity and app names to derive categories.
  • Prompt Generation: Authors developed 14 linguistic patterns from widget, local, and global context, with eight examples presented in Table I.The patterns were tested by generating content and checking whether apps transitioned to new activities.
  • Prompt Generation: Widget patterns specify the required content using noun, verb, and preposition keywords in continuation or mask expressions.Examples include “the game name is:” and “your weight is [MASK] kg”.
  • Prompt Generation: Local and global patterns enhance prompts with whole-page and whole-app context, including app name, activity name, and input category.This context supplements the widget description when generating input text.
  • Prompt Generation: Prompt rules combine the three information sources for single-widget pages and account for shared global context on multi-widget pages.The rules distinguish between pages with one input widget and pages with multiple input widgets.

B. Prompt-based Data Construction and Tuning

QTypist constructs tuning data from explicit input widgets whose candidate contents are available in search lists, popup menus, or crowdsourced prefilled fields.

  • B. Prompt-based Data Construction and Tuning: Domain-specific prompt tuning lacks an open dataset of input prompts and answers, while manual collection is time-consuming and biased.The method addresses this gap through automated data construction.
  • B. Prompt-based Data Construction and Tuning: Three cases provide candidate answers: associated search lists, popup menus, and human-prefilled widget content.These cases are illustrated in Fig. 6.
  • B. Prompt-based Data Construction and Tuning: Explicit input widgets are widgets with candidate contents available through lists, menus, or crowdsourced prefilled content.Rico is cited as a source of crowdsourced mobile UI data containing prefilled fields.
  • 1) Widget Extraction:: The construction challenge is locating explicit input widgets and mapping their candidate contents to the corresponding widget.This mapping defines the core extraction problem.
  • 1) Widget Extraction:: For search lists, an EditText directly above a ListView is treated as explicit, and list text or item titles become candidate contents.News-like items contribute the title identified by the smallest ordinate among relevant TextViews.
  • 1) Widget Extraction:: For popup menus, a Spinner is located and nearby TextViews are retrieved; the Spinner hint-text supplies the candidate content.The extraction uses relative coordinates of the Spinner and TextViews.
  • 1) Widget Extraction:: For filled content, human-entered hint-text is retained after filtering likely developer presets using terms such as “search,” “add,” and “enter.”The retained EditText and its hint-text provide the explicit widget and input content.

2) Prompt and Answer Generation:

QTypist turns explicit widget contexts into prompt–answer pairs, fine-tunes GPT-3 on them, and evaluates generated text by whether it passes input-requiring UI pages.

  • 2) Prompt and Answer Generation:: QTypist converts each explicit widget and its context into an input prompt, using the candidate content as its answer.This creates prompt–answer pairs for model tuning.
  • 2) Prompt and Answer Generation:: The model is fine-tuned with prompts as inputs and candidate answers as outputs.The learning objective is the same as that of the pretrained model.
  • 2) Prompt and Answer Generation:: Prompt tuning trains the pretrained model to recognize the prompt syntax and usage patterns represented in the tuning data.The paper presents this as more effective than general fine-tuning for understanding the questions.
  • C. Text Input Generation Model: GPT-3 receives the generated prompt and produces input content used to enhance automated GUI testing.The chosen GPT-3 model has 175 billion parameters and was trained on billions of tokens.
  • D. Implementation: QTypist uses GPT-3 Curie with a 64 batch size, 100 epochs, and a 0.01 learning-rate multiplier.Its tuning dataset contains 7,000 prompt–answer pairs from Rico and FrontMatter, trained for about 12 hours.
  • D. Implementation: QTypist can replace a testing tool’s text-input module, extracting widgets from UIAutomator view hierarchies and generating corresponding text.This integration is intended to improve activity coverage.
  • IV. EFFECTIVENESS EVALUATION: Effectiveness is measured by whether generated text passes UI pages requiring text input, followed by an evaluation of usefulness in integrated GUI testing.The paper compares QTypist with 11 commonly used and state-of-the-art approaches.
  • IV. EFFECTIVENESS EVALUATION: Twenty testers manually evaluate the quality of QTypist’s generated input text.This user study addresses the quality of generated content separately from page-passing effectiveness.

A. Experimental Setup

The effectiveness study evaluates QTypist on selected Android apps and text inputs using page-passing rate against 11 baselines, while a separate user study assesses output quality.

  • A. Experimental Setup: 106 apps with 168 text inputs remained after filtering 637 Google Play apps using tool, stability, baseline, and input-page criteria.The initial crawl covered the 100 most popular apps in each of 12 categories, retaining apps updated after March 2022.
  • A. Experimental Setup: Passing rate measures the proportion of UI pages requiring text inputs that are successfully passed.For each app, one text-input page is randomly selected for evaluation.
  • A. Experimental Setup: Table II reports the dataset used for the effectiveness evaluation.The table is identified as the effectiveness-evaluation dataset.
  • A. Experimental Setup: Twenty computer-science software testers with more than three years of testing experience independently assess generated inputs from QTypist and the baselines.Each tester sees the app UI page and the competing generated inputs.
  • A. Experimental Setup: QTypist is compared with 11 common-used and state-of-the-art baselines divided into random-/rule-based, constraint-based, and learning-based methods.The baseline grouping facilitates comparison across method families.
  • A. Experimental Setup: The random-/rule-based baselines include eight automated GUI testing methods, while three app-specific methods form the remaining comparison group.The listed random-/rule-based tools include Stoat, Droidbot, Ape, Fastbot, ComboDroid, TimeMachine, Humanoid, and Q-testing.
  • A. Experimental Setup: Constraint-based baselines include Mobolic and TextExerciser, which use input constraints or feedback hints to generate text.Mobolic combines online testing with input constraints, while TextExerciser exploits hints for malformed inputs.

C. Result

QTypist achieves a high passing rate across evaluated UI pages, outperforming the best baseline and benefiting from prompt-tuning. Its generated inputs are generally judged accurate and diverse, though failures remain for pages lacking context or requiring unique values.

  • 0.87 passing rate across 106 UI pages demonstrates that QTypist passes most text-input transitions.
  • 93% higher passing rate than RNNInput reflects QTypist’s use of contextual information and a pre-trained Large Language Model.
  • QTypist can generate meaningful inputs such as “Blood pressure record,” but struggles without contextual widgets or when apps require unique values such as server addresses.
  • 43% improvement, from 0.61 to 0.87, results when prompt-tuning data is used.
  • Practitioners rated QTypist’s generated input diversity and accuracy 4.4 on average, with Kendall’s W of 0.8 indicating high inter-agreement.

2) Input Text Generation Quality (RQ2):

QTypist-generated text is perceived as more semantically meaningful than baseline outputs, and integrating it with GUI testing tools increases explored activities, pages, and confirmed crash bugs.

  • QTypist generates semantically meaningful text, whereas random inputs may pass by chance without practical meaning.
  • QTypist was evaluated by integrating it with three automated GUI testing tools on 30 Android apps using activities, pages, and revealed bugs as metrics.
  • 42% more activities and 52% more pages are covered when Ape integrates QTypist.
  • 122% more confirmed crash bugs are found by Ape with QTypist than by Ape alone, detecting 51 versus 23 bugs.

A. Discussion

QTypist shows preliminary generality beyond Android and across app languages, while the evaluation remains constrained by app representativeness and manual judgments.

  • QTypist passed UI pages in 90% of sampled iOS apps and 85% of sampled Web apps, based on experiments with 20 apps per platform.
  • QTypist passed text-input UI pages in 86% of 50 apps using German, French, Italian, Chinese, or Korean, despite English prompt-tuning.
  • The approach may also assist manual testing by suggesting diversified input test cases for crowdtesters.
  • The experimental apps’ representativeness and generality are an external validity threat, despite selection criteria agreed with four practitioners.
  • Manual inspection and tagging introduce potential subjectivity into evaluation results, mitigated by an inspection team seeking agreement.

VII. RELATED WORK

Existing mobile GUI testing methods often rely on predefined candidates, templates, symbolic constraints, or random events for text input. QTypist instead uses contextual LLM generation and reports broader testing coverage.

  • Existing tools commonly use predefined candidates, while Monkey and Dynodroid cannot generate input content.
  • Template-based methods and symbolic-execution approaches require predefined templates or source-code constraints and may not directly apply to complex Android rendering.
  • QTypist generates input content from input widgets and contextual information using a pre-trained Large Language Model and prompt-based tuning.
  • QTypist covers 42% more app activities and 52% more pages than the raw automated GUI testing tool.
  • Future work targets extracting context from nearby UI pages and extending QTypist beyond app testing.
Loading 2212.04732v1…