PaperlayerView exact revision on arXiv

Source-linked AI summary

Continuous, Evolutionary and Large-Scale: A New Perspective for Automated Mobile App Testing

Mario Linares Vasquez, Kevin Moran, Denys Poshyvanyk

arXiv:1801.06267v1cs.SE

TL;DR

Mobile app testing remains largely manual, while existing tools do not resolve key challenges to comprehensive automation. The paper surveys current practice and challenges, then proposes the CEL framework and a research agenda. Its supported conclusion is a vision for automated testing organized around Continuous, Evolutionary, and Large-scale principles.

Abstract

from arXiv · show

Mobile app development involves a unique set of challenges including device fragmentation and rapidly evolving platforms, making testing a difficult task. The design space for a comprehensive mobile testing strategy includes features, inputs, potential contextual app states, and large combinations of devices and underlying platforms. Therefore, automated testing is an essential activity of the development process. However, current state of the art of automated testing tools for mobile apps poses limitations that has driven a preference for manual testing in practice. As of today, there is no comprehensive automated solution for mobile testing that overcomes fundamental issues such as automated oracles, history awareness in test cases, or automated evolution of test cases. In this perspective paper we survey the current state of the art in terms of the frameworks, tools, and services available to developers to aid in mobile testing, highlighting present shortcomings. Next, we provide commentary on current key challenges that restrict the possibility of a comprehensive, effective, and practical automated testing solution. Finally, we offer our vision of a comprehensive mobile app testing framework, complete with research agenda, that is succinctly summarized along three principles: Continuous, Evolutionary and Large-scale (CEL).

I. INTRODUCTION

Mobile testing is challenging, and current limitations leave much of it manual despite its cost to developers and industry. The paper surveys available practice and frames a CEL vision for automated testing.

I. INTRODUCTION

  • Mobile app testing remains mostly manual because current manual and automated solutions have limitations that impede a comprehensive, practical automated approach.The paper notes that this work costs developers and industry significant effort, time, and money.Source: §I · p. 1
  • The paper presents CEL testing—Continuous, Evolutionary, and Large-scale—along with a survey, challenge summary, proposed framework vision, and research agenda.The authors draw on research experience, industrial collaborations, and conversations with industry experts.Source: §I · p. 1

II. STATE-OF-THE-ART AND PRACTICE

  • The state-of-the-art overview covers available mobile-testing frameworks, tools, and services, but is not exhaustive and focuses on functional testing and popular commercial offerings.Its seven tool categories combine prior-work classifications with categories derived from commercial offerings.Source: §II · pp. 1–2

A. Automation APIs/Frameworks

GUI automation APIs expose screen information and let testers script interactions, but scripts can be brittle and costly to maintain. Record-and-replay tools ease script creation while trading event accuracy against portability.

A. Automation APIs/Frameworks

  • GUI automation frameworks expose screen-component information and simulate user interactions, supporting GUI-level tests written as action sequences with state assertions.Scripts may be hand-written or recorded and identify GUI components using varying attributes.Source: A. Automation APIs/Frameworks · p. 2
  • These frameworks may fail on edge cases involving different app states or GUI attributes, and often lack support for complex actions, contextual states, and multiple testing objectives.Maintaining GUI-level tests as an app evolves is expensive and can discourage adoption.Source: A. Automation APIs/Frameworks · p. 2

B. Record and Replay Tools

  • Record-and-replay tools ease test-script creation, but typically trade event timing accuracy against script representativeness and portability.Low-level event-stream recording can be accurate but tied to screen dimensions; higher-level GUI-action recording is more flexible but less accurate in timing.Source: B. Record and Replay Tools · p. 2
  • Record-and-replay tools require developers to define test oracles, either by adding assertions to recorded scripts or using tool wizards.The paper describes an ideal approach as combining accurate and portable scripts.Source: B. Record and Replay Tools · p. 2

C. Automated Test Input Generation Techniques

Automated input generation seeks to reduce manual test creation, but important test-generation challenges remain. The section also describes reporting and monitoring tools that provide richer failure information while automated monitoring remains focused on crashes.

C. Automated Test Input Generation Techniques

  • Automated input generation aims to ease test creation and includes random-based, systematic, model-based, search-based, and symbolic approaches.Approaches may target coverage, bug discovery, shorter scenarios, or typical app use cases.Source: C. Automated Test Input Generation… · p. 2
  • Despite progress, input-generation tools still face challenges including system-event generation, app restart costs, complex inputs, flaky tests, fragmentation, and limited support for diverse testing goals and developer feedback.Other identified needs include reproducibility, service and inter-app communication mocking, and cross-device scenario generation.Source: C. Automated Test Input Generation… · pp. 2, 4
  • Table I surveys tools, frameworks, and services related to mobile testing from academic and industrial backgrounds.The overview presents categories of automated test-input generation among the available approaches.Source: C. Automated Test Input Generation… · p. 2, Table I · p. 3

D. Bug and Error Reporting/Monitoring Tools

  • Bug-reporting and runtime-monitoring tools can provide failure videos, scripts, or stack traces, but automated error monitoring is limited to crash reporting.Collecting this information may require apps to include service-provided API calls.Source: D. Bug and Error Reporting/Monitoring… · p. 4

E. Mobile Testing Services

Crowd-based mobile testing services cover several testing goals but can be costly and too slow for agile development. Device diversity and testing constraints remain central challenges for a comprehensive automated approach.

E. Mobile Testing Services

  • Mobile testing services offer crowd-sourced functional, usability, security, and localization testing, but are paid, not open source, and often too slow for rapidly changing apps.The authors also note limits to collecting field usage data and addressing test flakiness or history agnosticism.Source: E. Mobile Testing Services · p. 4

F. Device Streaming Tools

  • Device-streaming tools range from locally mirroring a connected device to remote low-latency streaming, and can support controlled studies or crowdsourced testing.Examples named include Vysor, OpenSTF, and Appetize.io.Source: F. Device Streaming Tools · p. 4

III. CHALLENGES FOR ENABLING CEL MOBILE TESTING

  • Mobile testing challenges arise from event-driven apps, diverse sensor and hardware inputs, and development time and budget constraints that limit testing across representative configurations.The challenges summarized in the paper are drawn from research experience, industry collaborations, and conversations with professionals.Source: §III · pp. 4–5

A. Fragmentation

  • Device and OS diversity creates a nontrivial testing matrix, while paid cloud or crowd services can be too slow for agile and DevOps practices.Cloud/crowd services offer access to farms of virtual or physical devices or users with varied devices.Source: A. Fragmentation · p. 5

B. Test Flakiness

Backend dependence and variable runtime conditions make mobile tests vulnerable to flaky outcomes. The paper also identifies the lack of mobile-specific fault models in current automated testing approaches.

B. Test Flakiness

  • Backend service failures, timing races, and data-integrity problems can make test outcomes nondeterministic and affect assertions.Variable device states, including garbage collection and resulting GUI lag or ANR errors, are another source of flakiness.Source: B. Test Flakiness · p. 5
  • Assumptions in scripts and sequences, plus missed unexpected backend scenarios, contribute to flaky tests; waiting for GUI idleness reduces but does not eliminate the challenge.The challenge remains across both GUI front ends and service back ends.Source: B. Test Flakiness · p. 5

C. Mobile-Specific Fault Model and its Application

  • Mobile fault models describe platform-specific recurring or uncommon bugs and can inform test cases, mutation testing, and static detection, yet current automated mobile testing approaches do not use them.The paper notes that object-oriented fault models describe only a subset of mobile-app bugs.Source: C. Mobile-Specific Fault Model and its… · pp. 5–6

D. Lack of History Awareness in Test Cases

History-aware testing must preserve meaningful event sequences, but existing models can generate invalid sequences without identifying the use cases those sequences exercise. GUI scripts and models also need to evolve with app changes without discarding useful prior knowledge.

D. Lack of History Awareness in Test Cases

  • System and acceptance tests need event sequences with memory, since later actions may depend on earlier use cases such as creating a task before updating it.Source: D. Lack of History Awareness in Test… · p. 6
  • Event-flow graphs and language models represent history differently, but both generate event sequences without recognizing the high-level use cases they belong to.Event-flow graphs encode GUI states and transitions; language models condition the next event on earlier events,.Source: D. Lack of History Awareness in Test… · p. 6

E. Difficulties Evolving and Maintaining GUI Scripts/Models

  • Recorded or written GUI scripts are vulnerable to device fragmentation and app evolution: they may depend on display locations or change-prone component ids.Automation APIs partially decouple events from device characteristics, but their scripts can still depend on component ids.Source: E. Difficulties Evolving and… · p. 6
  • Runtime GUI ripping can rebuild a model just in time, but discards potentially useful knowledge from prior models; the paper calls for intelligent continuous model evolution.Source: E. Difficulties Evolving and… · p. 6

F. Absence of Mobile-Specific Testing Oracles

Mobile testing oracles remain difficult to automate robustly: exception-based checks miss some GUI errors, while screenshot comparisons face variability across devices and contexts. Automated input generation also covers few testing goals beyond destructive testing.

F. Absence of Mobile-Specific Testing Oracles

  • Manually codified oracles rely on testers to specify expected results, while exceptions-as-oracle and GUI-state-as-oracle offer alternative approaches.GUI-state oracles compare later test states with a baseline, assuming the GUI states remain unchanged.Source: F. Absence of Mobile-Specific Testing… · p. 6
  • Exception-based oracles detect crashes and unexpected errors but can miss GUI-level errors without thrown exceptions; robust automatic oracle generation remains an open issue.API-based testing also requires developers or testers to write and maintain tests and oracles.Source: F. Absence of Mobile-Specific Testing… · p. 6
  • Screenshot-based GUI oracles require app- and window-specific similarity thresholds, and matching can be affected by screen size, device, orientation, responsive layouts, themes, localization, and dynamic content.Source: F. Absence of Mobile-Specific Testing… · p. 7

G. Missing Support for Multi-Goal Automated Testing

  • Automated input generation has largely focused on destructive testing, while functional, regression, security, performance, localization, and play testing remain less supported.Functional and regression testing require links between high-level requirements or use cases and event sequences; play testing still relies on users for games involving specialized interaction and cognition.Source: G. Missing Support for Multi-Goal… · p. 7

IV. VISION: HOW TO ENABLE CEL MOBILE TESTING

The paper proposes CEL—Continuous, Evolutionary, and Large-scale—as principles for a comprehensive mobile-testing solution, paired with a conceptual architecture and research agenda. The vision specifies how testing should adapt to app and environment changes and scale across diverse devices and test conditions.

IV. VISION: HOW TO ENABLE CEL MOBILE TESTING

  • The CEL vision combines three principles with a proposed system architecture and research agenda to address the identified mobile-testing challenges.Source: §IV · p. 7, Figure 1 · p. 8

A. The CEL Testing Principles

  • Continuous testing should run across goals and environmental conditions after code or environment changes, focusing each iteration on the changes' impact set and producing expressive, reproducible bug reports.Source: A. The CEL Testing Principles · p. 7
  • Testing artifacts should evolve with the app, usage patterns, and devices or operating systems, using GUI, domain, usage, fault, and contextual models informed by historical and current data.Source: A. The CEL Testing Principles · p. 8
  • Large-scale execution should support parallel testing on physical or virtual devices, with physical devices or highly accurate simulations required for performance testing.The proposed engine should support cloud and on-premise hardware.Source: A. The CEL Testing Principles · p. 8

B. Proposed Architecture

The proposed CEL architecture combines change monitoring, test-artifact generation, and large-scale execution. Monitoring events trigger targeted testing, while history-aware models and automatically derived oracles support evolving test cases.

1) Change-monitoring subsystem:

2) Testing artifacts generation module:

  • The testing-artifact module combines GUI, Usage, Domain, Contextual, and Fault models into a history-aware Multi-Model representation for generating test cases.The models can be extracted from APIs, app usage, and user reviews, and express scenarios independently of device characteristics.Source: 2) Testing artifacts generation module · p. 8
  • Models and tests should co-evolve with code and external changes, while automatically derived oracles enable fully automated testing and error reporting.A small refactoring can trigger regression testing only on its impact set, whereas an API change can trigger tests for all features using the modified APIs.Source: 2) Testing artifacts generation module · p. 9

3) Large scale execution engine:

The vision pairs scalable execution across diverse devices with a research agenda for stronger app models. That agenda addresses multi-model representations, including GUI, domain, usage, context, and fault information.

3) Large scale execution engine:

  • Large-scale execution across physical or virtual device farms is intended to mitigate fragmentation and reduce test time, but current farms lack needed flexibility and customization.Device-independent tests can reduce GUI-event flakiness across screen sizes; bug reports should also provide actionable detail.Source: 3) Large scale execution engine · p. 9

C. Research Agenda

  • The proposed research agenda has six topics, reflecting that many components of the CEL vision remain unexplored.The authors present the agenda as a way to make the vision tractable.Source: C. Research Agenda · p. 9

1) Improved Model-Based Representations of Mobile Apps:

  • Multi-model app representations remain underdeveloped: GUI models are widely studied, but domain models receive little attention and could help tests exercise complex inputs and behaviors.Proposed directions include combining static and dynamic analysis and automatically extracting domain models from code, storage, and similarities among apps.Source: 1) Improved Model-Based… · p. 9
  • Contextual and usage models are rarely considered together with fault models, and history-aware updates are needed to keep these representations viable as apps evolve.Context models can capture network and sensor conditions, while usage models describe functional use cases as GUI-event combinations.Source: 1) Improved Model-Based… · p. 9

2) Goal-Oriented Automated Test Case Generation:

The research agenda calls for test generation across mobile-specific goals, more flexible scalable testing infrastructure, and stronger automated oracles. It highlights that existing approaches do not cover these needs comprehensively.

2) Goal-Oriented Automated Test Case Generation:

  • Automated input generation has largely focused on destructive testing, leaving security, localization, energy, performance, and play-testing as additional goals to address.Because mobile apps are GUI- and event-driven, device-level GUI tests are needed to exercise functionality that unit tests miss.Source: 2) Goal-Oriented Automated Test Case… · p. 9

3) Flexible Open Source Solutions for Large Scale and CrowdSourced Testing:

  • Customizable virtual devices could support scalable CI testing, while crowdsourcing better fits usability testing and usage-data collection; neither approach covers every testing goal.The passage identifies a need for flexible open-source solutions and privacy-aware execution traces and logs.Source: 3) Flexible Open Source Solutions for… · p. 10

4) Derivation of Scalable, Precise Automated Oracles:

  • Automated oracles remain open-ended; proposed directions include combining GUI images and text for state indications and using platform-specific fault models.Existing examples such as screen-rotation actions and GUI screenshots are described as simple and not comprehensive.Source: 4) Derivation of Scalable, Precise… · p. 10

5) Mining Software Repositories and User Reviews to Drive Testing:

The research agenda proposes mining software histories and user reviews to inform testing, and improving how results and feedback reach developers. The paper concludes with CEL as a proposed framework and roadmap for future work.

5) Mining Software Repositories and User Reviews to Drive Testing:

  • Repository histories and user reviews could inform testing, but little work uses informative reviews to prioritize or generate functional and regression tests.The passage describes mining individual-app or cross-app histories and linking informative reviews to affected code areas.Source: 5) Mining Software Repositories and… · p. 10

6) Derivation of Methods to Provide Useful Feedback for Developers:

  • Useful developer feedback requires actionable visualizations, traceability to app code, and improved ways for users to report issues.The passage points to empirical studies of developer information needs and on-device bug reporting and monitoring.Source: 6) Derivation of Methods to Provide… · p. 10

V. CONCLUSIONS

  • The paper presents CEL as a proposed architecture for automated mobile testing and a six-topic research agenda toward fully automated solutions.The authors describe the paper as a roadmap for the mobile testing community.Source: §V · p. 10
Loading 1801.06267v1…