PaperlayerView exact revision on arXiv

Source-linked AI summary

How do Developers Test Android Applications?

Mario Linares Vasquez, Carlos Bernal-Cardenas, Kevin Moran, Denys Poshyvanyk

arXiv:1801.06268v1cs.SE

TL;DR

Automated mobile testing tools have not persuaded developers to replace much manual testing, and developers’ needs remain insufficiently understood. The paper surveys open-source Android contributors about their testing practices and preferences. Responses indicate reliance on use-case-based test design and a preference for expressive generated cases, while the authors caution that the findings may not generalize beyond the sampled developers.

  • Problem

    Automated mobile-testing approaches have limited adoption, and researchers lack a clear understanding of real developers’ testing needs and preferences.

    Source: §I · p. 1

  • Method

    The authors survey open-source Android developers about testing practices, automated-testing preferences, tool use, and testing-quality measures.

    Source: §I · p. 1, §II · p. 2

  • Results

    Developers rely on usage models when designing tests and prefer high-level, expressive generated cases organized around use cases.

    Source: §VI · p. 9

  • Takeaways & Limitations

    The authors recommend that automated mobile-testing approaches account for developer preferences and workflows, including use-case-oriented tests that co-evolve with apps and features.

    Source: §VI · p. 9

    The findings may not generalize beyond open-source GitHub developers or across mobile platforms, and the sample may not represent Android developers globally.

    Source: §IV · p. 8

Abstract

from arXiv · show

Enabling fully automated testing of mobile applications has recently become an important topic of study for both researchers and practitioners. A plethora of tools and approaches have been proposed to aid mobile developers both by augmenting manual testing practices and by automating various parts of the testing process. However, current approaches for automated testing fall short in convincing developers about their benefits, leading to a majority of mobile testing being performed manually. With the goal of helping researchers and practitioners - who design approaches supporting mobile testing - to understand developer's needs, we analyzed survey responses from 102 open source contributors to Android projects about their practices when performing testing. The survey focused on questions regarding practices and preferences of developers/testers in-the-wild for (i) designing and generating test cases, (ii) automated testing practices, and (iii) perceptions of quality metrics such as code coverage for determining test quality. Analyzing the information gleaned from this survey, we compile a body of knowledge to help guide researchers and professionals toward tailoring new automated testing approaches to the need of a diverse set of open source developers.

I. INTRODUCTION

Mobile GUI testing is costly and often manual, while developers lack clear guidance on what automated testing should provide. This study surveys open-source developers to identify their testing preferences and guide future approaches.

I. INTRODUCTION

  • Automated mobile-testing tools have seen limited practical adoption, and reported barriers include irreproducible cases, cross-run side effects, and weak debugging support.The paper argues that approaches misaligned with developer needs and preferences are unlikely to have meaningful impact.Source: §I · p. 1
  • The survey examines open-source developers’ testing preferences to provide actionable guidance for researchers and practitioners developing mobile-testing automation.The authors describe the work as complementing earlier studies that identified or speculated about reasons for low adoption.Source: §I · pp. 1–2
  • Developers report preferring manual testing over automation because of test-case representation and reproducibility issues.The study also reports that developers rely heavily on application usage models when designing cases and prefer expressive generated cases organized around use cases.Source: §I · p. 2
  • About 64% of participants did not consider code coverage an important measure of test quality, citing feature coverage or fault detection as more useful.Automated input-generation techniques commonly target code coverage or fault discovery.Source: §I · pp. 1–2

II. DESIGN OF THE EMPIRICAL STUDY

The study surveys open-source Android contributors to characterize manual test-design practices and preferences for automated test cases and testing strategies. It addresses details of test design that earlier studies had not examined.

II. DESIGN OF THE EMPIRICAL STUDY

  • The study examines developers’ practices in documenting requirements and designing tests, preferences for automated testing, tool use, and testing-quality measures.Its intended audience is researchers and practitioners designing mobile-app testing approaches and tools.Source: §II · p. 2
  • Earlier studies addressed testing challenges and tools but had not analyzed manual test-design strategies or preferences for generated test cases.The authors argue that understanding manual practices can inform automated techniques that better match developer needs.Source: §II · p. 2
  • The study extends a prior survey’s tool findings with a list of 55 tools reported by participants and a question about experiences with random testing.The prior survey listed 10 tools.Source: §II · p. 2
  • The survey covers test-design practices, reasons for manual testing, preferred generated-test features, and use of random testing and coverage analysis.Table I begins with SQ1, asking what documentation participants use to specify app requirements.Source: §II · p. 2, Table I · p. 2

1) Research Questions:

The research questions cover how mobile developers design test cases and what they prefer in automatically generated cases. They also ask which automated-testing tools developers use and whether they value code coverage as a measure of effectiveness.

2) Research Questions:

  • RQ1 asks how mobile developers design test cases, while RQ2 asks what they prefer in automatically generated test cases.These questions address developers’ testing practices and desired characteristics of automation.Source: 1) Research Questions · p. 2
  • RQ3 asks which automated-testing tools developers use, and RQ4 asks whether they consider code coverage useful for evaluating test effectiveness.Together, these questions address tool use and a specific testing-quality metric.Source: 1) Research Questions · p. 2

2) Data Collection:

The researchers collected survey responses from GitHub contributors to open-source Android apps and analyzed both selected-response and free-text answers. After filtering, 102 valid responses remained.

2) Data Collection:

  • The survey questions map to the study’s research questions: SQ1–SQ5 address RQ1, SQ6–SQ7 RQ2, SQ8–SQ9 RQ3, and SQ10 RQ4.Demographic questions were also collected to filter participants and measure sample diversity.Source: 2) Data Collection · p. 2
  • Researchers emailed 10,000 GitHub contributor addresses and received 485 responses; after filtering, 102 valid responses remained.The filtering excluded responses with no Android-programming experience, invalid answers, and unfinished surveys.Source: 2) Data Collection · pp. 2–3
  • Participants’ claimed experience varied across general programming, Android programming, and industrial experience.The paper presents participant demographics in Figure 1.Source: 2) Data Collection · p. 3, Figure 1 · p. 3
  • Multiple- and single-choice answers were analyzed with descriptive statistics, while free-text answers were manually categorized using grounded-theory-based open coding.Three authors coded independently, standardized codes, and marked disagreements as “Unclear.”Source: 2) Data Collection · p. 3

III. RESULTS AND DISCUSSION

The section reports participant responses using descriptive statistics and summaries and discussion of free-text answers.

III. RESULTS AND DISCUSSION

  • Participant responses are summarized using descriptive statistics.The section also discusses examples from free-text answers.Source: §III · p. 3
  • Examples of free-text answers are included in the discussion.Source: §III · p. 3
  • The section presents responses to the study’s research questions.Source: §III · p. 3

A. RQ1: What are the strategies used by MDs to design test cases?

Surveyed developers relied on app usage models to design tests, devoted the most testing effort to manual testing, and preferred expressive, use-case-oriented generated tests.

A. RQ1: What are the strategies used by MDs to design test cases?

  • In this multiple-choice survey, feature lists were the top-used requirements-documentation artifact, although there was no large tendency toward a preferred artifact; six participants did not document requirements.Among single-artifact practices, user stories were reported by 14 participants, feature lists by 12, informal natural-language descriptions by 10, and use cases by 9.Source: A. RQ1: What are the strategies used… · p. 3
  • Manual testing averaged 58.18% of participants’ testing effort (IQR [25%, 97.25%], maximum 100%), compared with 18.96% for JUnit-based testing and 8.29% for API-based automated testing.Source: A. RQ1: What are the strategies used… · pp. 3–4
  • In-the-wild use was limited: only 11 participants reported spending more than 10% of testing effort with Android Monkey (averaging 16.36%, maximum 35%), and only two used automated ripping tools (5% and 10%).Source: A. RQ1: What are the strategies used… · p. 4
  • Participants attributed manual-testing preference to changing requirements, time and process constraints, app size, limited tool knowledge or usability, and maintenance costs.The cited rationale also includes the time and cost of creating automated tests and maintaining their artifacts.Source: A. RQ1: What are the strategies used… · pp. 4–5
  • The most common test-design strategy was following the app’s usage model (30 answers), ahead of unit tests (10) and negative or edge-case testing (9).Only 15 participants reported mixed strategies; non-functional requirements were mentioned infrequently.Source: A. RQ1: What are the strategies used… · p. 5
  • Test cases most often targeted individual use cases or features (77 answers) or combinations of them (49), while random events were selected by 16 participants.Source: A. RQ1: What are the strategies used… · p. 5, Figure 4 · p. 5
  • For automatically generated tests, participants preferred natural language (33 of 102) and automated testing APIs over low-level ADB input commands.The prose characterizes the combined natural-language and API preference as favoring expressive cases over low-level scripts.Source: A. RQ1: What are the strategies used… · p. 6, Figure 5 · p. 6
  • For generated tests, 82 of 102 participants (80.39%) preferred expected outputs or reproduction steps grouped by use case or feature; 21.7% favored screenshots.Source: A. RQ1: What are the strategies used… · p. 6, Figure 6 · p. 6

C. RQ3: What tools are used by MDs for automated testing?

Survey participants reported a wide range of automated-testing tools, but many had no experience with random testing. They saw possible uses for random testing, alongside concerns about low-value results, reproducibility, and expressiveness.

C. RQ3: What tools are used by MDs for automated testing?

  • 55 different tools were reported, led by JUnit (45 participants), Roboelectric (16), and Robotium (11), while 28 reported using none.39 of the 55 tools were mentioned by only one participant.Source: C. RQ3: What tools are used by MDs for… · p. 6
  • 60.78% (62 out of 102) reported no experience with Monkey or random-testing tools; 12.75% (13) considered them useful for corner cases or stress testing.Some also cited performance issues as a possible use.Source: C. RQ3: What tools are used by MDs for… · p. 6
  • Participants cited random testing's possible value for corner cases and stress testing, but eight of 15 noted low-impact bugs or reproducibility limitations.Examples of reported uses include finding memory leaks and minor stability issues.Source: C. RQ3: What tools are used by MDs for… · pp. 6–7
  • Monkey and random-testing tools drew complaints of noise, little practical signal, limited usefulness on higher-quality code, and hard-to-reproduce bugs.Eleven participants cited reproducibility, script-maintenance costs, or a lack of tangible benefits.Source: C. RQ3: What tools are used by MDs for… · p. 7
  • Monkey can replay seeded event streams but lacks logs for higher-level representations; participants also cited poor expressiveness and reproducibility as adoption barriers.The authors call for more expressive generated streams and tools that are easier for industry to adopt.Source: C. RQ3: What tools are used by MDs for… · p. 7

D. RQ4: Do MDs consider code coverage as a useful metric

Most surveyed participants did not use or consider code coverage useful for measuring test-case quality, though some valued it for finding untested code. Participants also cited other quality criteria and differing uses for coverage.

D. RQ4: Do MDs consider code coverage as a useful metric

  • Of 51 respondents answering “No” to whether code coverage is useful, 19 said it was not a good test-case quality metric because alternatives included reviews, fault detection, feature coverage, or “works for me” criteria.One response described coverage as an indication that something is tested, not that the test is correct.Source: D. RQ4: Do MDs consider code coverage… · pp. 7–8
  • Coverage can identify untested code, but one respondent argued it says nothing about code already under test.The respondent distinguished coverage of code from the quality of tests exercising that code.Source: D. RQ4: Do MDs consider code coverage… · p. 8
  • 29 out of 102 participants considered coverage useful, particularly for identifying code entities that had not been tested.Some described it as a guide to areas needing more tests rather than a measure of test quality.Source: D. RQ4: Do MDs consider code coverage… · p. 8
  • Some proponents used minimum coverage requirements of 75-80% or valued coverage as an easy-to-measure metric, while noting it was not the only consideration.One respondent said being measured could encourage developers to write more tests.Source: D. RQ4: Do MDs consider code coverage… · p. 8
  • 63.73% (65 out of 102) did not use or consider code coverage useful for measuring test-case quality.The authors report this as the overall finding for the surveyed participants.Source: D. RQ4: Do MDs consider code coverage… · p. 8

IV. THREATS TO VALIDITY

The study mitigated construct-validity risks through survey filtering and a grounded-theory coding process. Its findings are bounded by the open-source GitHub Android sample and 102 respondents, which may not represent broader developer populations.

IV. THREATS TO VALIDITY

  • To reduce construct-validity threats, the researchers excluded incomplete surveys, participants with no experience, and surveys with invalid answers.Three authors independently open-coded responses, standardized categories, and marked disagreements as “Unclear.”Source: §IV · p. 8
  • The findings may not generalize to developers on other platforms or to mobile developers beyond open-source GitHub projects.The authors also note they cannot guarantee that all participants are commercial developers.Source: §IV · p. 8
  • 102 respondents may not represent the global community of Android developers.The authors compare this respondent count with other studies,,.Source: §IV · p. 8

V. RELATED WORK

Prior work examined mobile-development challenges, testing tools, and testing practices from different perspectives. This study distinguishes itself by surveying open-source Android developers about their practices and preferences across manual and automated testing.

V. RELATED WORK

  • Erfani et al. surveyed 12 experts and 188 community members on mobile-development challenges, including testing practices and testing support.They identified multiple platforms as a major challenge and emphasized the importance of performance-monitoring tools and testing frameworks.Source: §V · p. 8
  • Kochhar et al. surveyed Android and Windows developers, reporting that about 86% of Android apps lacked test cases and that automated tools could be difficult to use.Their study also found limited code reach by existing tools and low developer awareness; this paper examines manual and automated practices and preferences among Android developers.Source: §V · pp. 8–9
  • Choudhary et al. compared Android input-generation techniques on 60 applications and concluded that random testing, specifically Android Monkey, surpassed the other techniques.This paper instead surveys developers about automated-tool usage and experiences with random testing.Source: §V · p. 9
  • Linares-Vásquez et al. surveyed mobile-testing tools and approaches but did not survey developers about current testing trends.Their work outlined Continuous, Evolutionary, and Large-Scale principles for future mobile testing.Source: §V · p. 9
  • Aho et al. evaluated Murphy with three systems and three industrial test engineers, finding it reduced the time and effort of generating model-based test cases.Unlike that evaluation of one tool, this study surveys open-source developers about tools and preferences for automated testing.Source: §V · p. 9

VI. CONCLUSION AND LEARNED LESSONS

A survey of 102 GitHub open-source mobile-app contributors examined test design, automated-testing preferences, and test-suite quality. Respondents favored use-case-centered, expressive generated tests, while code coverage was not viewed as an important quality measure.

VI. CONCLUSION AND LEARNED LESSONS

  • The survey gathered in-the-wild practices and preferences from 102 GitHub contributors on test design, automated approaches, and test-suite quality.The study focused on open-source mobile apps.Source: §VI · p. 9
  • Developers primarily use application usage models, such as use cases and user stories, to design test cases, and prefer expressive generated tests organized around use cases.The authors identify usage models and expressive test cases as goals for automated tools and note support for multi-model model-based testing.Source: §VI · p. 9
  • Code coverage was not perceived as an important measure of test-case quality by survey participants.The authors suggest evaluating other criteria, including relevant fault detection and feature coverage.Source: §VI · p. 9
  • Automation APIs with manually written test cases represent the reported state of practice, and those cases are fragile to GUI changes; Monkey use is low.The authors recommend aligning research tools with developer workflows, including co-evolving tests, low overhead, and support for debugging and traceability.Source: §VI · p. 9
Loading 1801.06268v1…