PaperlayerView exact revision on arXiv

Source-linked AI summary

Prompting Is All You Need: Automated Android Bug Replay with Large Language Models

Sidong Feng, Chunyang Chen

arXiv:2306.01987v3cs.SE

TL;DR

Automated bug-replay approaches can struggle with unclear or ambiguous reproduction steps in bug reports. AdbGPT uses prompted LLMs to extract steps and guide replay; it reproduced 81.3% of bugs, outperforming baselines and ablations.

  • Problem

    Automated bug-replay approaches struggle with unclear, imprecise, or ambiguous steps-to-reproduce (S2Rs) in bug reports.

    Source: §1 · p. 1

  • Method

    AdbGPT prompts LLMs with entity specifications and few-shot, chain-of-thought examples to extract S2R entities and guide replay using the current GUI state.

    Source: §1 · p. 2, §3.1 · p. 3, §3 · p. 3

  • Results

    81.3% of bugs were successfully reproduced, outperforming the baselines and ablations.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The experiments and user study support AdbGPT’s effectiveness, efficiency, and usefulness for accelerating bug replay.

    Source: §7 · p. 11

    The main external-validity limitation is whether the selected testing dataset is representative.

    Source: §5 · p. 10

Abstract

from arXiv · show

Bug reports are vital for software maintenance that allow users to inform developers of the problems encountered while using the software. As such, researchers have committed considerable resources toward automating bug replay to expedite the process of software maintenance. Nonetheless, the success of current automated approaches is largely dictated by the characteristics and quality of bug reports, as they are constrained by the limitations of manually-crafted patterns and pre-defined vocabulary lists. Inspired by the success of Large Language Models (LLMs) in natural language understanding, we propose AdbGPT, a new lightweight approach to automatically reproduce the bugs from bug reports through prompt engineering, without any training and hard-coding effort. AdbGPT leverages few-shot learning and chain-of-thought reasoning to elicit human knowledge and logical reasoning from LLMs to accomplish the bug replay in a manner similar to a developer. Our evaluations demonstrate the effectiveness and efficiency of our AdbGPT to reproduce 81.3% of bug reports in 253.6 seconds, outperforming the state-of-the-art baselines and ablation studies. We also conduct a small-scale user study to confirm the usefulness of AdbGPT in enhancing developers' bug replay capabilities.

1 INTRODUCTION

The introduction frames Android bug replay as difficult because bug reports contain unclear, ambiguous, or incomplete reproduction steps, then presents AdbGPT as a lightweight, prompt-engineered LLM approach. It reports improved extraction and replay results and initial evidence of usefulness from a small-scale user study.

1 INTRODUCTION

  • S2Rs challenge automated replay because they can be unclear, ambiguous, semantically complex, multi-step, temporally ordered, or incomplete.More than 1.3k open-source project developers raised concerns about missing S2Rs in a letter to GitHub.Source: §1 · pp. 1–2
  • The authors motivate LLMs as expert developers that can understand bug reports and guide replay through dynamic GUI components.They cite LLMs’ natural-language understanding and reasoning abilities, including ChatGPT’s ability to comprehend questions and act as an assistant.Source: §1 · p. 2
  • AdbGPT’s two phases—S2R entity extraction and GUI-guided replay—use prompt engineering, few-shot examples, developer chain-of-thought, and GUI encoding.The paper presents this as a lightweight alternative to manually crafted patterns or machine learning requiring massive training data.Source: §1 · p. 2
  • AdbGPT reproduced 81.3% of bugs, outperforming baselines and ablations, while saving an average of 1105.17 seconds per bug report.The paper also reports a small-scale study on five real-world bug reports as initial evidence of AdbGPT’s usefulness in facilitating replay.Source: §1 · p. 2
  • AdbGPT achieved 90.4% step extraction accuracy and 90.8% entity extraction accuracy on 88 bug reports, exceeding two state-of-the-art baselines and two ablations.The paper also identifies its work as the first to exploit LLMs for bug report analysis and GUI guidance.Source: §1 · p. 2

2 BACKGROUND

The background explains how AdbGPT uses prompt-based LLMs, few-shot examples, and chain-of-thought reasoning to extract bug-report steps and guide bug replay.

2.1 Large Language Models

  • LLMs can perform tasks from natural-language prompts without additional training or hard coding, and AdbGPT prompts them to extract S2R entities and generate actionable operations for bug replay.The paper describes a Transformer-based process for interpreting prompts and generating answers.Source: §2.1 · p. 2

2.2 In-context Learning

  • Few-shot learning supplies examples of desired inputs and outputs to clarify prompt syntax and output patterns, helping LLMs extract S2R entities and guide replay of bug reports they were not specifically trained to handle.The paper adopts few-shot rather than zero-shot learning because applying zero-shot learning to unseen tasks remains challenging.Source: §2.2 · pp. 2–3

2.3 Chain-of-Thought Reasoning

  • Chain-of-thought reasoning uses intermediate rationales between input and output to help LLMs handle the logical thinking and multiple steps needed to understand reports and trigger bugs.AdbGPT uses this approach to reproduce bugs through intermediate steps modeled on expert developers’ reasoning.Source: §2.3 · p. 3

3 ADBGPT APPROACH

AdbGPT uses prompt engineering with a single LLM to extract steps-to-reproduce entities and guide bug replay by matching those entities to current GUI states.

3 ADBGPT APPROACH

  • AdbGPT addresses S2R entity extraction and guided replay through prompt engineering with a single LLM, rather than ad-hoc methods for the two phases.The approach is presented as lightweight compared with prior approaches using methods such as NLP and greedy algorithms.Source: §3 · p. 3
  • S2R entity extraction identifies each step’s action type, target component, input value, or scroll direction.Source: §3 · p. 3
  • Guided replay matches extracted S2R entities with current GUI states to repeat the bug reproduction steps.Source: §3 · p. 3

3.1 S2R Entity Extraction

AdbGPT prompts LLMs to extract S2R entities by supplying action specifications, representative examples, and step-by-step reasoning. The resulting prompt is applied to test bug reports to generate structured entities for replay.

3.1 S2R Entity Extraction

  • AdbGPT conditions LLMs with action specifications, example S2Rs, chain-of-thought reasoning, and example outputs, then uses few-shot prompting to extract entities from test reports.The extracted entities are intended to map report steps to device replay instructions.Source: §3.1 · p. 3

3.1.1 Available actions.

  • The approach focuses on five common actions—Tap, Scroll, Input, Double-tap, and Long-tap—and represents each with action-specific primitives for components, directions, or values.For example, Tap uses [Tap] [Component], Scroll uses [Scroll] [Direction], and Input uses [Input] [Component] [Value].Source: §3.1.2 · p. 3, §3.1.1 · p. 3

3.1.2 Action primitives.

  • Representative examples are preferred to random samples because a random bug report may not cover the task’s complexities and give LLMs comprehensive understanding.The authors recruited professional developers to identify challenging reports for representative examples.Source: §3.1.3 · p. 3

3.1.3 In-context learning.

  • Three developers reviewed prior-work bug reports [29], selecting challenging cases and reaching consensus on a 15-report representative dataset.After 1.5 hours, each developer had examined an average of 110 reports and selected 7 as challenging.Source: §3.1.3 · p. 4

3.1.4 Chain-of-thought reasoning.

  • Developers wrote and refined chain-of-thoughts that explain action mapping, conjunctions, temporal order, and step relationships leading to extracted entities.For example, “Open” is mapped to the available action “Tap.”Source: §3.1.4 · p. 4

3.1.5 Prompt construction.

  • The prompt combines available actions, action primitives, an example input, its reasoning, and output; test reports then yield numeric entity lists parsed with regular expressions.The prompt’s wording need not strictly follow grammar rules.Source: §3.1.5 · p. 4

3.2 Guided Replay

AdbGPT uses few-shot learning and chain-of-thought reasoning to guide GUI actions, including when reported steps are missing. It encodes screens as HTML-like text and iteratively selects target components to reproduce each step.

3.2 Guided Replay

  • AdbGPT uses LLM-generated GUI guidance to map each reported step to a target component, including when no matching component is visible.This addresses lexical component matching’s inability to handle missing steps.Source: §3.2 · p. 4

3.2.1 GUI encoding.

  • The GUI encoder heuristically converts view hierarchies into HTML-like text using depth-first traversal and a selected set of component properties.It maps common classes to functionally similar tags, represents special inputs with input and label elements, and encodes resource IDs, text, image descriptions, and traversal-order indexes. The conversion is not one-to-one; for example, resource IDs become class attributes and image descriptions become alt attributes.Source: §3.2.1 · p. 5

3.2.2 In-context learning.

  • The in-context examples include five developer-selected cases built by randomly omitting zero to two reported steps.Developers also wrote and discussed chain-of-thought examples covering semantic mismatches such as “Sign in” versus “Log in,” and missing steps with a potential target and missing tag; the examples address the effort of manually collecting representative cases.Source: §3.2.2 · p. 5, §3.2.3 · p. 5

3.2.3 Chain-of-thought reasoning.

  • At replay time, the prompt combines GUI encoding, example inputs, chain-of-thought, and outputs; the model selects targets by ID and can mark missing steps.If a missing-step prediction lacks an ID, the system returns to the prior step and prompts for another component to explore.Source: §3.2.4 · p. 5

3.3 Implementation

AdbGPT uses ChatGPT with a small few-shot example set, prediction parsing and fallback input values, and Android tools to automate bug replay.

3.3 Implementation

  • ChatGPT uses 1–3 few-shot examples, a range chosen to fit its 4,096-token input limit based on a pilot study.The examples are selected to fit ChatGPT’s implicit maximum input tokens.Source: §3.3 · p. 5
  • AdbGPT uses “[]” to extract predictions from verbose LLM outputs and supplies a “test” value when an action lacks an explicit input value.The prediction may be an entity, target component ID attribute, or missing-step flag, depending on the task.Source: §3.3 · p. 6
  • The automated replay tool uses Genymotion to control a virtual Android device, UIAutomator to dump its GUI hierarchy, and ADB to replay steps.Source: §3.3 · p. 6

4 EVALUATION

The evaluation tested AdbGPT’s S2R entity extraction, guided replay, runtime, and perceived usefulness to developers. AdbGPT outperformed the reported baselines on extraction and replay, and the user study found it faster than participants’ manual replay.

4.1 RQ1: Performance of S2R Entity Extraction

  • Against the best baseline, MaCa, AdbGPT was on average 39.3% more accurate in step extraction and 42.2% in entity extraction; few-shot learning and chain-of-thought boosted performance from 84.6% and 87.6% to 90.8%.The experimental dataset comprised 88 bug reports containing textual S2Rs.Source: §4.1 · pp. 6–7
  • The extraction analysis attributed baseline errors to inconsistent step formats, context-dependent instructions, and varied wording for the same action.AdbGPT also failed on ambiguous S2Rs, such as an input description missing the word “to.”Source: §4.1 · p. 7

4.2 RQ2: Performance of Guided Replay

  • AdbGPT reproduced 81.3% of bugs, compared with 45.8% for ReCDroid, 58.3% for AdbGPT w/o Few, and 62.5% for AdbGPT w/o CoT.The study evaluated 48 reproducible S2Rs; chain-of-thought improved AdbGPT’s performance by 18.8%.Source: §4.2 · pp. 7–8, Table 4 · p. 8
  • For guided replay, AdbGPT can infer omitted steps from later instructions and the current GUI, and match components despite imprecise descriptions or language differences.Unclear input values remain a limitation: randomly generated text may be invalid for fields such as URLs.Source: §4.2 · pp. 8–9

4.3 RQ3: Efficiency Performance

  • AdbGPT took 255.75 seconds per bug report on average—2.11 seconds for extraction and 253.64 seconds for replay—versus 1360.92 seconds for ReCDroid.The paper reports nearly 5x time savings per replay and 435% acceleration in GUI replay, attributing the difference primarily to AdbGPT’s ability to bridge missing steps.Source: Table 5 · p. 9, §4.3 · p. 9

4.4 RQ4: Usefulness of AdbGPT

  • AdbGPT’s average replay time was 269.4 seconds versus 480.7 seconds for participants, and it completed all tasks within 7 minutes.The small-scale study involved eight participants; three failed to reproduce bugs within the 10-minute limit, so the participants’ average time was underestimated.Source: Table 6 · p. 9, §4.4 · p. 9
  • All participants favored using AdbGPT in practice, giving it an average preference score of 4.5 out of 5.0.Participants described manual S2R interpretation and trial-and-error around missing steps as time-consuming; one suggested developer confirmation when model confidence is low.Source: §4.4 · p. 10

5 THREATS TO VALIDITY

The authors mitigate internal-validity risks from LLM variability and manual labeling, and external-validity risks from dataset representativeness and graduate-student participation in the user study.

Internal validity

  • Three runs aggregated LLM-related metrics, while annotator training, independent labeling, and consensus addressed randomness and subjectivity in ground-truth creation.The ground truth was for S2R entity extraction in RQ1.Source: §5 · p. 10

External validity

  • The study used reports from prior research and five randomly collected practical bug reports; five Android-experienced graduate students were treated as developer substitutes.The practical reports were evaluated with five graduate students and three developers in real-world development environments; each student had at least 1.5 years of Android app development and bug reproduction experience.Source: §5 · p. 10

6 RELATED WORK

Related work covers Android bug recording and replay, and the use of large language models in software engineering. AdbGPT is positioned as a lightweight, prompt-based alternative that applies LLMs to understand reproduction steps and guide GUI-based replay.

6.1 Bug Record and Replay

  • Unlike Android record-and-replay tools requiring heavy frameworks or app instrumentation, AdbGPT needs only natural-language reproduction steps.Nurmuradov et al. captured interactions and generated heatmaps; ECHO, Reran, Barista, and Android Bot Maker also recorded and replayed interactions.Source: §6.1 · p. 10
  • Yakusu, ReCDroid, and MaCa automate replay from S2Rs, but depend on report wording and omit temporal order; AdbGPT uses prompt engineering instead.Prior approaches are constrained by formatting, word usage, and granularity; AdbGPT uses an LLM to extract S2R entities and reproduce bugs without manually crafted patterns or predefined vocabularies.Source: §6.1 · p. 10
  • Bug-reporting aids such as DeMIBuD, EBug [26], and BURT complement AdbGPT by improving report quality and potentially LLM understanding.These approaches detect missing information, suggest reproduction steps, or guide users to report essential elements.Source: §6.1 · p. 10

6.2 Large Language Models for Software Engineering

  • LLM software-engineering research spans code completion [39] [41] [51], Codex code generation, and program patching [3] [44].Codex, employed by GitHub Copilot, generates code from natural-language comments, though minor issues remain.Source: §6.2 · p. 10
  • AlphaRepair encodes patch-context relations; other work studies software development [57], productivity, and security vulnerabilities [38].AdbGPT explores integrating LLMs with GUI understanding to interpret S2Rs and guide dynamic screens for automated bug reproduction.Source: §6.2 · p. 11

7 CONCLUSION

AdbGPT uses prompt engineering, few-shot examples, and chain-of-thought reasoning to reproduce bugs from bug-report steps, and the paper reports that experiments and a user study demonstrate its effectiveness, efficiency, and usefulness. The authors identify incorporating bug-report artifacts such as logs, screenshots, and recordings as a future direction.

7 CONCLUSION

  • AdbGPT prompts LLMs with entity specifications, exemplars, and developer reasoning to extract step entities, then uses GUI screens and few-shot chain-of-thought guidance to reproduce bugs.The authors report that experiments and a user study demonstrate the approach’s effectiveness, efficiency, and usefulness in accelerating bug reproduction.Source: §7 · p. 11
  • Future work will improve AdbGPT’s effectiveness by incorporating bug-report information such as stack traces, error logs, screenshots, and screen recordings to help LLMs understand bugs.Source: §7 · p. 11

Related papers

Loading 2306.01987v3…