Source-linked AI summary
Automatically Discovering, Reporting and Reproducing Android Application Crashes
Kevin Moran, Mario Linares-Vásquez, Carlos Bernal-Cárdenas, Christopher Vendome, Denys Poshyvanyk
TL;DR
Android crashes are difficult to detect, report, and reproduce because mobile apps combine event-driven interfaces with contextual inputs and existing reports often lack actionable detail. CRASHSCOPE systematically explores Android apps using multiple strategies and generates augmented reports with reproduction steps, screenshots, stack traces, and replayable scripts. It detects about as many crashes as current tools while producing more detailed reports, and its reports are readable and reliably reproducible, though its coverage is limited by unsupported swipe gestures and highly specialized text input.
Problem
Android crash detection and reporting are difficult because event-driven interfaces, contextual inputs, and inadequate existing reports hinder reliable reproduction.
Method
CRASHSCOPE systematically explores Android apps with static- and dynamic-analysis-informed strategies, then generates natural-language crash reports and replayable scripts.
Results
CRASHSCOPE detected about as many crashes as other Android input-generation tools, while its reports were more readable and useful and achieved 94% reproducibility versus 92% for human-written reports.
Takeaways & Limitations
CRASHSCOPE provides automated crash discovery together with detailed fault information and reports that support reliable reproduction.
Takeaways & Limitations
CRASHSCOPE cannot execute swipe gestures and does not support highly specialized text input, limiting exploration of some applications and GUI components.
Abstract
from arXiv · showhide
Mobile developers face unique challenges when detecting and reporting crashes in apps due to their prevailing GUI event-driven nature and additional sources of inputs (e.g., sensor readings). To support developers in these tasks, we introduce a novel, automated approach called CRASHSCOPE. This tool explores a given Android app using systematic input generation, according to several strategies informed by static and dynamic analyses, with the intrinsic goal of triggering crashes. When a crash is detected, CRASHSCOPE generates an augmented crash report containing screenshots, detailed crash reproduction steps, the captured exception stack trace, and a fully replayable script that automatically reproduces the crash on a target device(s). We evaluated CRASHSCOPE's effectiveness in discovering crashes as compared to five state-of-the-art Android input generation tools on 61 applications. The results demonstrate that CRASHSCOPE performs about as well as current tools for detecting crashes and provides more detailed fault information. Additionally, in a study analyzing eight real-world Android app crashes, we found that CRASHSCOPE's reports are easily readable and allow for reliable reproduction of crashes by presenting more explicit information than human written reports.
I. INTRODUCTION
CRASHSCOPE addresses the difficulty of detecting, reporting, and reproducing Android crashes by automatically exploring apps and generating detailed, replayable crash reports. Its evaluations show crash detection comparable to existing tools and reports that are more detailed, readable, and useful than human-written reports.
- Motivation: Android crash reports often lack reproduction steps and contain only stack traces, overly detailed logs, or loosely structured natural language.The absence of useful reproduction information is identified as a major cause of developers failing to reproduce reports.
- Approach: CRASHSCOPE automatically discovers, reports, and reproduces Android application crashes through systematic input generation.It requires only an .apk and an Android emulator or device, without instrumenting the subject app or Android OS.
- Contribution: CRASHSCOPE generates expressive crash reports with screenshots, augmented natural-language reproduction steps, and replayable scripts.The approach is presented as a fully automatic way to provide detailed fault information for mobile applications.
- Evaluation: On 61 Android applications, CRASHSCOPE detected crashes about as effectively as five state-of-the-art input-generation tools while automatically producing detailed reports and replayable scripts.The comparison included Dynodroid, Gui-Ripper, PUMA, A3E, and Monkey.
- Evaluation: For eight open-source applications, CRASHSCOPE reports provided more detail while being at least as useful as human-written crash reports.The user study evaluated report reproducibility and readability against human-written reports.
- Resources: The experimental dataset and crash reports were made available in an online appendix.
II. RELATED WORK & MOTIVATION
Prior Android testing approaches use random, systematic, or model-based input generation, but commonly provide limited reporting, require burdensome instrumentation or setup, or lack integrated contextual and text-input strategies. CRASHSCOPE is motivated as an approach that combines automated crash discovery with expressive reporting and replay.
- Input-generation approaches: Automated mobile input-generation approaches are commonly categorized as random-based, systematic, or model-based techniques.Random approaches choose or bias events, systematic approaches follow heuristics such as BFS or DFS, and model-based approaches construct app-state models.
- Bug reporting: CRASHSCOPE differs from VanarSena by requiring no instrumentation and generating screenshots, natural-language reproduction steps, and replayable scripts.
- Input-generation approaches: Systematic approaches execute input events according to predefined heuristics, while model-based approaches construct models of screens and event flows for exploration.
- Other mobile testing approaches: The related-work landscape includes tools that expose programmable UI events, instrument app binaries, or systematically test adverse contextual conditions using existing test cases.
- Bug reporting: Earlier crash-reporting work either relies on pre-existing stack traces and manually annotated traces or requires users to create reports for functional problems.CRASHDROID translates call stacks into reproduction steps, whereas FUSION assists users in creating expressive reports.
B. Previous Studies on Mobile App Bug/Crashes
Previous mobile-app studies informed CRASHSCOPE’s focus on targeted contextual-state testing, crash resilience, and app-agnostic fault triggers. These studies also motivated its use of uncaught exceptions and crashes as detection oracles.
- Crash studies: A study of 25 million Windows Phone crash reports found that a small number of root causes cover many crashes and can affect diverse execution paths.The study also identified externally inducible faults, such as HTTP errors caused by network connectivity issues.
- Design implications: The Windows Phone study motivated targeted testing of different contextual states while remaining resilient to crashes encountered during exploration.
- Design implications: CRASHSCOPE uses uncaught exceptions and app crashes as well-defined fault oracles and incorporates rotation, activity life-cycle, and gesture categories to trigger them.The approach specifically implements a targeted version of double rotation.
C. Limitations of Mobile Testing Approaches
Existing mobile testing tools are limited by weak fault reporting, impractical instrumentation or setup requirements, and incomplete coverage of text-input and contextual-state strategies. CRASHSCOPE addresses these gaps through automatic reporting, accessible execution, contextual exploration, and crash-resilient testing.
- Existing limitations: Previous approaches often leave developers to interpret stack traces, log files, and non-expressive event sequences rather than providing clear testing results.
- Existing limitations: Many automated input-generation approaches are impractical because they require instrumentation or difficult setup procedures.Instrumentation can also create false-positive crashes that increase the effort needed to interpret results.
- Existing limitations: Few approaches combine multiple text-input methodologies with contextual-state testing, such as testing wifi on and off, in one holistic system.
- CRASHSCOPE design: CRASHSCOPE requires only an .apk and an Android 4.3-or-newer emulator or physical device, and emulator execution can be parallelized across configurations.The supported Android versions represented 55% of the Android OS install base at the time reported.
- CRASHSCOPE design: CRASHSCOPE explores varying contextual states using static analysis to target relevant locations and continues testing unvisited GUI components and states after handling a crash.
- Workflow: Its workflow statically identifies context-related activities, systematically executes GUI strategies, stores crash execution data, and generates natural-language reports and crash scripts.
A. Extracting Activity and App-Level Contextual Features
CRASHSCOPE uses static and dynamic analyses to identify contextual features and systematically exercise Android GUI inputs, text fields, and navigation orders.
- CRASHSCOPE uses AST-based analysis to extract API-call chains associated with contextual-feature invocations.
- Call-graph analysis links contextual API calls to Activities; unlinked calls are tested at app level.Activity-level testing changes the feature state when the Activity is foreground, whereas app-level testing changes it at app startup.
- CRASHSCOPE dynamically extracts each screen’s GUI hierarchy and systematically executes clickable, long-clickable, and text-input components.
- Text inputs use expected and unexpected generation strategies selected after detecting the field’s expected keyboard type.
- GUI hierarchies are traversed top-down or bottom-up, and newly reached screens are explored after recorded transitions.
C. Testing Apps in Different Contextual States
CRASHSCOPE tests contextual behavior by changing app- or Activity-level conditions and combining contextual-state testing with GUI traversal and text-input strategies.
- The GUI Ripping Engine first checks app-level contextual features and then tests Activity-level features when the current Activity is suitable.
- Sensor-related adverse states use unexpected sensor values, while network connectivity is disabled during emulator-based contextual testing.Examples include nonphysical GPS coordinates and other sensor values unlikely under normal conditions.
- CRASHSCOPE combines GUI traversal direction, text-input generation, and adverse-context testing into multiple exploration strategies.
E. Generating Expressive, Natural Language Crash Reports
CRASHSCOPE generates crash reports that combine contextual execution details, reproduction steps, screen-flow information, and pruned exception traces.
- Each crash report includes general app, operating-system, device, contextual-state, orientation, and resolution information.
- Reports describe GUI events and contextual states in natural-language reproduction steps and show the interacted component within the app’s screen flow.
- The reproduction template identifies an action, component text, component type, and relative screen location.
- Text-entry steps replace the generic action with the specific input string entered by the user.
- Crash scripts contain adb commands for touch and text inputs plus markers for contextual-state changes.
- The first study evaluates CRASHSCOPE’s crash-detection effectiveness against other Android testing approaches.
A. Methodology
The comparative study evaluates CRASHSCOPE against Android input-generation tools under controlled emulator conditions, while accounting for runtime and crash-reporting differences.
- The study compares CRASHSCOPE with a subset of Androtest tools selected because some alternatives produced inconsistent runtime behavior.
- CRASHSCOPE runs each app on an emulator configured with a 1200x1920 display, 2GB RAM, a 200 MB virtual SD card, and Android 4.4.2.
- Monkey uses 100–700 event sequences, seven configurations, five repetitions per app, and different random seeds to align event counts with other tools.
- CRASHSCOPE detects crashes through Android’s standard crash dialog, whereas the other tools lack reliable runtime crash identification.
- Crash results report unique crashes by differing stack traces and separately identify instrumentation-caused crashes for tools other than CRASHSCOPE.
- Parsing problems in competing tools’ outputs underscore the need for automated crash detection and reporting.
B. Results & Discussion
CRASHSCOPE detected crashes about as effectively as comparable tools while producing detailed reports, and its varied strategies uncovered different crash types. Average statement coverage did not reliably predict crash-discovery capability.
- Crash detection: CRASHSCOPE detected about as many crashes as A3E, GUI-Ripper, and Dynodroid, more than PUMA, and fewer than Monkey, without instrumentation false positives.Unlike Monkey, CRASHSCOPE also provided detailed bug reports and replayable scripts.
- Crash diversity: CRASHSCOPE detected crashes that differed from those found by other approaches, partly because its exploration strategies exercised unexpected inputs.In the zooborns app, unexpected text triggered a formatting-related crash whose AsyncTask exception could mislead developers.
- Strategy effectiveness: The unexpected text heuristic contributed to all crashes listed for CRASHSCOPE, while forward GUI traversal found 8 crashes versus 7 for backward traversal.Seven crashes overlapped between the forward and backward strategies; the strongest overall combination was contextual features enabled, forward, unexpected.
- Coverage and discovery: Higher average statement coverage did not necessarily imply better fault discovery, because CRASHSCOPE detected about as many crashes with lower coverage than PUMA, Monkey, and Dynodroid.The results motivate considering metrics beyond code coverage when evaluating automated mobile-app testing.
V. STUDY 2: REPRODUCIBILITY & READABILITY
The second study evaluates whether CRASHSCOPE-generated reports improve crash reproducibility and readability relative to human-written reports using eight real-world Android crashes.
- Study scope: The study compares CRASHSCOPE reports with original human-written reports from online issue trackers for eight real-world Android app crashes.Its focus is developers’ ability to reproduce the reported bugs.
- Research questions: RQ5 asks whether CRASHSCOPE-generated reports are more reproducible than original human-written reports.
- Research questions: RQ6 asks whether CRASHSCOPE-generated reports are more readable than original human-written reports.
A. Methodology
The study manually selected eight reported crashes, evaluated reproduction with 16 graduate students, and collected user-experience responses after reproduction. CRASHSCOPE reports achieved similar reproducibility to human-written reports and were reported as more readable and useful.
- Study setup: Eight crashes were manually selected from F-droid issue trackers, then CRASHSCOPE was run against the corresponding app versions and emulator configuration.The study was designed to evaluate report reproducibility and readability, not crash-discovery effectiveness.
- Participants and procedure: Sixteen computer-science graduate students reproduced eight crashes using either human-written or CRASHSCOPE reports.Each crash-report type was evaluated by four participants, with report order balanced across participants.
- Measures: Participants completed a survey covering user preferences and usability after reproducing the crashes.The study also collected programming experience and Android familiarity.
- Reproducibility results: 94% of CRASHSCOPE reports were successfully reproduced, compared with 92% of original reports.These correspond to 60 of 64 CRASHSCOPE reports and 59 of 64 original reports.
- Readability results: Participants found CRASHSCOPE reports more readable and useful than original reports.
VI. LIMITATIONS & THREATS TO VALIDITY
CRASHSCOPE’s evaluation is bounded by tool capabilities and study scope. It cannot cover some Android interactions, and its results cannot be generalized to Android apps broadly.
- Tool limitations: CRASHSCOPE cannot execute swipe gestures, so it may miss crashes involving list components that extend beyond the device screen.It also lacks support for highly specialized text input, limiting exploration in some apps.
- External validity: CRASHSCOPE’s evaluation used 61 open-source applications in the first study and eight crashes from eight open-source applications in the second.These subject-app samples constrain generalization to Android apps overall.
- Internal validity: The second study approximated experienced Android developers with graduate Computer Science students who reported extensive programming and moderate Android experience.The authors identify this participant choice as a potential internal-validity threat, mitigated by prior evidence supporting experienced graduate students as developer proxies.
- Scope of conclusions: The authors conclude that CRASHSCOPE detects about as many crashes as other automated tools while providing more detailed, readable, and replayable reports.These conclusions remain bounded by the evaluated applications and study settings.