PaperlayerView exact revision on arXiv

Source-linked AI summary

Prompting Is All You Need: Automated Android Bug Replay with Large Language Models

Sidong Feng, Chunyang Chen

arXiv:2306.01987v3cs.SE

TL;DR

Automated Android bug replay is difficult because reproduction steps can be ambiguous, incomplete, and poorly handled by pattern- and vocabulary-based methods. AdbGPT uses prompt engineering with few-shot examples and chain-of-thought reasoning to extract replay entities and guide interactions from GUI states. It reproduces 81.3% of bug reports in 253.6 seconds and is also evaluated through extraction experiments and a user study.

  • Problem

    Existing automated bug-replay approaches struggle with unclear, ambiguous, and incomplete reproduction steps and rely on manually crafted patterns or predefined vocabularies.

  • Method

    AdbGPT uses prompt engineering, few-shot learning, and developers’ chain-of-thought reasoning to extract S2R entities and dynamically guide replay using current GUI screens.

  • Results

    81.3% of bugs were reproduced in 253.6 seconds, while extraction accuracy reached 90.4% for steps and 90.8% for entities.

  • Takeaways & Limitations

    The experiments and user study demonstrate AdbGPT’s effectiveness, efficiency, and usefulness for automated Android bug replay.

  • Takeaways & Limitations

    The main external validity threat is whether the testing dataset adequately represents broader bug-replay settings.

Abstract

from arXiv · show

Bug reports are vital for software maintenance that allow users to inform developers of the problems encountered while using the software. As such, researchers have committed considerable resources toward automating bug replay to expedite the process of software maintenance. Nonetheless, the success of current automated approaches is largely dictated by the characteristics and quality of bug reports, as they are constrained by the limitations of manually-crafted patterns and pre-defined vocabulary lists. Inspired by the success of Large Language Models (LLMs) in natural language understanding, we propose AdbGPT, a new lightweight approach to automatically reproduce the bugs from bug reports through prompt engineering, without any training and hard-coding effort. AdbGPT leverages few-shot learning and chain-of-thought reasoning to elicit human knowledge and logical reasoning from LLMs to accomplish the bug replay in a manner similar to a developer. Our evaluations demonstrate the effectiveness and efficiency of our AdbGPT to reproduce 81.3% of bug reports in 253.6 seconds, outperforming the state-of-the-art baselines and ablation studies. We also conduct a small-scale user study to confirm the usefulness of AdbGPT in enhancing developers' bug replay capabilities.

1 INTRODUCTION

AdbGPT addresses the difficulty of reproducing Android bugs from unclear, incomplete, and semantically complex reproduction steps by using LLM-based prompt engineering. It combines few-shot examples and developer reasoning to extract steps and guide replay, achieving strong extraction and replay results.

  • Motivation: Previous automated approaches struggle with unclear, ambiguous, and incomplete steps to reproduce bugs, limiting reliable bug replay.These steps may require semantic interpretation, resolving multiple sub-steps, and understanding temporal relations beyond fixed patterns or vocabulary lists.
  • Approach: AdbGPT uses prompt engineering, few-shot learning, and chain-of-thought reasoning to extract reproduction entities and dynamically guide replay from GUI screens.The approach supplies entity specifications, representative examples, and developers’ step-by-step reasoning before using current GUI information for replay guidance.
  • Evaluation: 90.4% and 90.8% accuracy were achieved for step extraction and entity extraction, respectively, while 81.3% of bugs were successfully reproduced.The evaluation compares AdbGPT with two state-of-the-art baselines and two ablation studies.
  • Evaluation: AdbGPT also saved an average of 1105.17 seconds per bug report compared with the baselines.A user study additionally assessed the perceived usefulness of the tool.
  • Contribution: The work presents LLM-based bug report analysis and GUI guidance as a new direction for software engineering tasks.The authors describe AdbGPT as a lightweight approach that avoids training and hard-coded patterns.

2 BACKGROUND

The background explains how prompting adapts pretrained LLMs to tasks without additional training, and why few-shot examples and chain-of-thought reasoning are useful for AdbGPT’s complex bug-replay tasks.

  • Prompt Engineering: Prompt engineering invokes pretrained LLMs through natural-language instructions rather than additional training or hard coding.The input prompt contains task-specific instructions that are encoded and used to generate an output.
  • In-context Learning: Few-shot learning augments a prompt with example inputs and outputs so LLMs can recognize task syntax and output patterns.AdbGPT uses this in-context paradigm for S2R entity extraction and bug-replay guidance.
  • Chain-of-thought Reasoning: Chain-of-thought reasoning adds intermediate rationales to examples for tasks requiring logical thinking and multiple steps.The paper represents this paradigm as <input, reasons, output> and applies it to extraction and replay guidance.

3 ADBGPT APPROACH

AdbGPT automatically extracts the entities needed to reproduce each bug step and then matches those entities against current GUI states to replay the bug.

  • Overview: AdbGPT has two phases: S2R Entity Extraction and Guided Replay.The first defines action types, target components, input values, or scroll directions; the second matches them with GUI states.
  • S2R Entity Extraction: S2R Entity Extraction identifies the entities that define each step required to reproduce a bug report.The extracted entities include action types, target components, input values, and scroll directions.
  • Guided Replay: Guided Replay uses the extracted entities and current GUI states to repeat the bug-reproduction steps.The phase matches the S2R entities with GUI states to guide interaction with the application.

3.1 S2R Entity Extraction

The extraction phase uses LLM knowledge, action specifications, representative examples, and developer-authored reasoning to convert varied S2R text into structured replay entities.

  • Entity Specifications: AdbGPT provides LLMs with entity specifications and action primitives to interpret varied expressions in S2R text.The extracted entities are intended to map bug-report language to device-replay instructions.
  • Action Primitives: The approach focuses on five standard actions: Tap, Scroll, Input, Double-tap, and Long-tap.Customized gestures such as pinch and multi-handed gestures are outside the paper’s focus.
  • In-context Learning: Representative examples are selected to expose the complexities of the extraction task rather than relying on random bug reports.The paper notes that random examples may not cover all task complexities and describes recruiting professional developers for selection.
  • Chain-of-thought Reasoning: Developer-authored chain-of-thought reasoning explains action mappings, multiple sub-steps, and altered temporal order.For example, reasoning can map an unavailable action to a semantically closest action and clarify conjunctions that change execution order.
  • Prompt Construction: The extraction prompt combines available actions, action primitives, example inputs and outputs, and chain-of-thought reasoning.The test bug report is then queried so the LLM generates a numeric list of extracted S2R entities.

3.2 Guided Replay

Guided Replay maps extracted S2R entities to GUI events by encoding the current screen as HTML-like text and prompting an LLM to select target components, including when steps are missing. Few-shot examples and chain-of-thought reasoning support semantic matching and missing-step guidance.

  • Guided Replay: Guided Replay explores the app to map extracted S2R entities onto a sequence of GUI events for bug reproduction.The approach addresses component-mapping failures caused by missing steps on the current screen.
  • GUI encoding: The current GUI screen is represented as HTML-like text so its component information can be provided to the LLM.The encoding converts the view hierarchy heuristically rather than performing a perfect one-to-one HTML conversion.
  • GUI encoding: GUI encoding includes component resource IDs, native classes, visible text, content descriptions, and traversal-order numeric IDs.Common Android classes are mapped to HTML tags with similar functionality, such as Button to <button> and ImageView to <img>.
  • In-context learning: Five representative examples teach the LLM to handle S2Rs with missing steps through developer-validated examples and chain-of-thought reasoning.Examples are created by randomly omitting 0 to 2 steps and selecting challenging cases with three developers.
  • In-context learning: Given an extracted step and current GUI encoding, the LLM predicts a target component ID or a missing-step flag for replay.The prompting process uses GUI encoding, example inputs, chain-of-thought, and example outputs; target components are inferred using their IDs.

3.3 Implementation

AdbGPT uses ChatGPT with a small number of few-shot examples and extracts structured predictions from verbose responses to automate Android-device interaction and bug replay.

  • LLM configuration: AdbGPT uses ChatGPT with 1 to 3 few-shot learning examples, chosen to fit within its 4,096-token input limit.The range was selected through a small-scale pilot study.
  • Output handling: The implementation extracts entity, target-component-ID, and missing-step predictions from verbose LLM responses using square brackets.This reduces responses to the specific outputs needed by the extraction and replay phases.
  • Input handling: When a bug report gives no explicit input value, AdbGPT assigns the value “test” to the input action.This handles vague descriptions such as “enter the name.”
  • System implementation: The automated tool runs a virtual Android device with Genymotion, dumps GUI hierarchies through UIAutomator, and replays steps through ADB.These components provide device control, interface observation, and action execution.

4 EVALUATION

The evaluation measures AdbGPT’s accuracy, replay success, efficiency, and usefulness against baselines and ablations. AdbGPT improves S2R extraction and guided replay while reducing replay time and receiving strong participant support, though ambiguous or incomplete reports remain challenging.

  • RQ1: S2R Entity Extraction: AdbGPT is 39.3% more accurate in step extraction and 42.2% more accurate in entity extraction than the best baseline, MaCa.Few-shot learning and chain-of-thought increased performance from 84.6% and 87.6% to 90.8%.
  • RQ1–RQ2: Qualitative Analysis: AdbGPT’s qualitative advantages address inconsistent formats, dependent context, missing steps, and component mismatches through semantic interpretation of reports and GUIs.The approach also handles morphological forms including abbreviations, synonyms, misspellings, and different languages.
  • Limitations: Ambiguous S2Rs and unspecified input values remain limitations because they can cause extraction errors or require invalid random inputs.The paper notes missing keywords, unclear field values, and fields such as URLs that require valid formats.
  • RQ2: Guided Replay: 81.3% of bugs are reproduced, compared with 45.8% for ReCDroid, 58.3% for AdbGPT w/o Few, and 62.5% for AdbGPT w/o CoT.Chain-of-thought provides an 18.8% improvement in guided replay performance.
  • RQ3: Efficiency: 255.75 seconds is AdbGPT’s average replay time, versus 1360.92 seconds for ReCDroid, saving nearly 5x time per bug replay.Entity extraction takes 2.11 seconds and GUI replay takes 253.64 seconds on average.
  • RQ4: Usefulness: Participants preferred AdbGPT with an average score of 4.5 out of 5.0, while the approach completed all tasks within 7 minutes.Participants’ average reproduction time was 480.7 seconds, compared with 269.4 seconds for AdbGPT; three participant attempts exceeded 10 minutes.

5 THREATS TO VALIDITY

The study identifies internal threats from stochastic LLM outputs and manual ground-truth labeling, and an external threat from the representativeness of the testing dataset.

  • LLM randomness may produce different results across runs of the same prompt, causing metrics to vary.The authors ran LLM-related approaches three times and aggregated the metrics.
  • Manual S2R entity labels may introduce subjectivity or errors into the ground truth.Annotators received training, passed a test, labeled independently, and reached consensus.
  • The testing dataset may not fully represent the intended evaluation population.The authors collected reports from previous research and added five practical reports evaluated with graduate students and developers.

6 RELATED WORK

Related work covers Android bug recording and replay, automated processing of reproduction steps, bug-report quality assistance, and emerging LLM applications in software engineering.

  • Bug record and replay: Prior Android tools record user interactions or support replay through browser-based screen capture, heatmaps, and program-analysis applications.
  • Bug record and replay: Automated bug-replay research has generated executable test cases and used lexical knowledge, but earlier work did not consider temporal order in reproduction steps.
  • Bug-report quality: Other approaches detect missing bug-report information, suggest reproduction steps during reporting, or interactively guide users to include essential elements.
  • LLMs for software engineering: LLM research in software engineering has examined code completion, developer productivity, software development, and security vulnerabilities.
  • LLMs for software engineering: This work positions LLM-based GUI understanding as a new direction alongside existing software-engineering applications.

7 CONCLUSION

AdbGPT uses LLM prompting, few-shot learning, and chain-of-thought reasoning to accelerate automated bug reproduction. The authors identify incorporating richer bug-report artifacts as a direction for improving effectiveness.

  • AdbGPT prompts LLMs with entity specifications, representative examples, and developers’ reasoning to extract S2R entities like a developer expert.
  • It then uses each step’s entities, the current GUI screen, few-shot learning, and chain-of-thought reasoning to dynamically guide bug replay.
  • Future improvements could incorporate stack traces, error logs, screenshots, and screen recordings to enhance the LLMs’ understanding of bugs.
Loading 2306.01987v3…