Source-linked AI summary
Understanding Flaky Tests: The Developer's Perspective
Moritz Eck, Fabio Palomba, Marco Castelluccio, Alberto Bacchelli
TL;DR
Flaky-test research has proposed automated fixes, but scientific knowledge about the phenomenon remains limited. This study examines developers’ perspectives through an empirical analysis of 200 flaky tests and a survey, finding four previously unreported causes, substantial practical relevance, and reproduction and diagnosis as major challenges.
Problem
Despite proposed techniques to automatically fix some flaky tests, scientific knowledge about their causes, significance, and developer-facing challenges remains limited.
Method
The study analyzed 200 flaky tests fixed by professional developers, developed a taxonomy, and surveyed developers about flakiness relevance, challenges, and information needs.
Results
Four previously unreported flakiness categories were identified, three requiring the most fixing effort; 79% of surveyed developers viewed flaky tests as a moderate to serious problem.
Takeaways & Limitations
Understanding developers’ perceptions can guide research and solutions intended to better support diagnosing and fixing test flakiness.
Takeaways & Limitations
Participants may not have recalled the fixing effort precisely, limiting confidence in the dataset’s effort assessments.
Abstract
from arXiv · showhide
Flaky tests are software tests that exhibit a seemingly random outcome (pass or fail) when run against the same, identical code. Previous work has examined fixes to flaky tests and has proposed automated solutions to locate as well as fix flaky tests--we complement it by examining the perceptions of software developers about the nature, relevance, and challenges of this phenomenon. We asked 21 professional developers to classify 200 flaky tests they previously fixed, in terms of the nature of the flakiness, the origin of the flakiness, and the fixing effort. We complement this analysis with information about the fixing strategy. Subsequently, we conducted an online survey with 121 developers with a median industrial programming experience of five years. Our research shows that: The flakiness is due to several different causes, four of which have never been reported before, despite being the most costly to fix; flakiness is perceived as significant by the vast majority of developers, regardless of their team's size and project's domain, and it can have effects on resource allocation, scheduling, and the perceived reliability of the test suite; and the challenges developers report to face regard mostly the reproduction of the flaky behavior and the identification of the cause for the flakiness. Data and materials [https://doi.org/10.5281/zenodo.3265785].
1 INTRODUCTION
This study examines flaky tests from developers’ perspective, addressing limited knowledge about their causes, significance, and fixing challenges. It combines developers’ classifications of real-world flaky tests with a broader survey of developer perceptions.
- 21 professional developers classified 200 previously fixed flaky tests by flakiness nature, origin, fixing effort, and strategy.The study focused on one ecosystem to identify less frequent cases that broader sampling might miss.
- 79% of surveyed developers considered flaky tests a moderate or serious problem, regardless of team size or project domain.40% reported dealing with flaky tests at least weekly.
- Four previously unreported flakiness causes were identified, and they were also judged the most effortful to fix.
- Developers associated flaky tests with consequences for scheduling, resource allocation, and perceived testing-process reliability.
- Reproducing flaky behavior and identifying its cause were perceived as the major developer challenges.
- The study publicly releases its dataset and supporting analysis materials.
2 GOALS AND METHOD’S OVERVIEW
The study uses a mixed-methods design to investigate flaky-test causes, origins, fixing effort, practical relevance, and developer challenges. It combines an annotated dataset of Mozilla flaky tests with developer surveys and a multivocal literature review.
- Study design: Developers classified flaky tests by nature, origin, and fixing effort, while repository data supplied information about fixing strategies.
- Study design: The study combines 200 flaky tests annotated by 21 professional developers with 121 valid online-survey responses.
- Dataset and participants: The researchers selected Mozilla’s fixed flaky tests from Bugzilla and recruited developers with concrete experience fixing multiple flaky tests.
- Dataset and participants: Mozilla’s database records 100 to 150 newly detected flaky tests per week, supporting recruitment of developers experienced with flaky-test diagnosis and repair.
- Dataset and participants: The analysis excluded 24 cases whose developers could not ascertain the cause after rewriting the test and code-under-test.
- Analysis: Researchers used iterative content analysis to consolidate developer-assigned categories into a taxonomy and double-check classifications.
- Survey method: The survey’s multivocal literature review covered white and gray literature and identified information types relevant to diagnosing and fixing flaky tests.
3 RQ1 – FLAKY TESTS: NATURE, ORIGIN, AND FIXING
Developers identified established and four newly reported types of flaky tests, with the latter including several high-effort causes. Flakiness arose in both test and production code, while fixing strategies varied by cause.
- Concurrency affected 26% of flaky tests, had high median fixing effort of 4, and required production-code examination in 34% of cases.This finding indicates that flakiness can originate beyond test code.
- Asynchronous calls without adequate waiting affected 22% of tests, had medium median effort of 3, and were usually fixed by adding a waitFor statement.All cases were attributed to test code, and 86% were fixed with waitFor.
- Test Suite Timeout was rare, appearing in 4 cases, but had median effort 3.5; developers accelerated suites in 75% of cases.Unlike Test Case Timeout, no single test causes the suite-level timeout.
- The dataset confirmed seven prior categories but did not observe Network, I/O, or Unordered Collections, possibly because methodology and system context shaped the findings.The authors present their taxonomy as complementary rather than exclusive of the earlier taxonomy.
- Four newly reported flakiness categories emerged: Too Restrictive Range, Test Case Timeout, Platform Dependency, and Test Suite Timeout.The latter three were associated with higher fixing effort.
4 RQ2 – FLAKY TESTS: RELEVANCE
Survey respondents generally encountered flaky tests frequently and regarded them as a meaningful problem. They linked flakiness to scheduling and resource allocation difficulties and to reduced test-suite reliability.
- 109 respondents encountered flaky tests at least some times a year, and 79% of them considered the problem moderate to serious.Among these respondents, 58% encountered flaky tests at least monthly.
- Developers’ open-text responses were grouped into three major themes concerning problems caused by flaky tests.The supplied passages identify scheduling, test-suite reliability, and developer allocation as themes or related reported consequences.
- Scheduling: 77% of developers described flaky tests as time-consuming because reproducing intermittent failures requires repeated reruns and variable debugging contexts.Their lower priority than permanent failures can also require managerial permission and dedicated refactoring resources.
- Test Suite Reliability: 73% reported that a flaky test is no longer fully reliable, reducing trust in its output and potentially causing actual failures to be ignored.The unpredictability of which tests will fail intermittently adds to this reliability concern.
- Flaky tests were perceived as affecting resource allocation, scheduling, and the reliability of the testing process.The authors summarize these effects as evidence that flaky tests are frequent and non-negligible.
5 RQ3 – FLAKY TESTS: CHALLENGES
Developers identify reproducing failures and determining flakiness causes as the central challenges, while design and fixing practices create additional difficulties across the test life cycle.
- Core fixing challenges: 89% of developers rate identifying the flakiness nature as moderately or extremely important, while 68% find determining its root cause difficult or very difficult.The relevant causes include those documented in this study and prior work.
- Information needs: 83% of developers value identifying involved code elements, but 75% consider obtaining this information not difficult.Perceived difficulty depends partly on source-code complexity and quality.
- Information needs: 79% of developers rate determining whether flakiness originates in test or production code as moderately to extremely important.The study identifies origin information as relevant to diagnosing flaky tests.
- Information needs: 72% of developers consider a test’s past flakiness history poorly relevant during fixing, potentially because different flakiness types require different information.The passage presents this explanation as likely rather than definitive.
- Life-cycle challenges: Flakiness challenges span design, diagnosis, and fixing rather than being confined to one development phase.Design challenges include decoupling tests and mocking dependencies; fixing challenges include uncertainty about whether a change worked and insufficient logs.
- Core fixing challenges: Reproducing the context that triggers failure is the most important and difficult information need for fixing flaky tests.Slow tests and execution environments make verification harder.
6 DISCUSSION AND IMPLICATIONS
The discussion positions flaky tests as a research and organizational problem, extending known causes, highlighting team-level consequences, and motivating prevention, testability, and prioritization research.
- Flaky tests as a relevant research problem: Four previously unreported flakiness types emerged, and Test Case Timeout, Platform Dependency, and Test Suite Timeout require the highest fixing effort.The four types are Too Restrictive Range, Test Case Timeout, Platform Dependency, and Test Suite Timeout.
- An organizational view: Flaky tests create organizational problems involving resource allocation and require research that considers team organization alongside source code.The discussion proposes examining how resolution processes may be optimized.
- Preventing Flaky Tests: Machine learning is proposed as a way to identify tests at high risk of becoming flaky before flakiness is introduced.The approach would use features characterizing each flakiness type.
- Design for Testability: Design patterns for deterministic tests and flakiness-related anti-patterns are proposed as directions for improving testability.Participants identified designing test code to avoid flakiness as an important challenge.
- Flaky Test Prioritization: Ranking flaky tests using effort-related information could help maintain high-quality test suites in large codebases.The discussion argues that flaky tests are not equally problematic.
7 THREATS TO VALIDITY
The authors identify threats involving dataset construction, survey response bias, subjective analysis, incomplete literature coverage, and uncertain generalizability beyond Mozilla.
- Scope of threats: The study’s threats-to-validity discussion covers dataset, relevance, challenge, analysis, and generalizability concerns.The authors frame these as possible influences despite mitigation strategies.
- Construct Validity: Participants may not have recalled fixing effort precisely, and survey responses may overrepresent developers especially involved with flakiness.The survey used a charity donation incentive to mitigate voluntary response bias.
- Conclusion Validity: The taxonomy and challenge themes may contain classification errors or omit challenges absent from the examined sources.The analyses relied on iterative content analysis and further replications are recommended.
- External validity: The taxonomy’s applicability beyond the 200 flaky tests from Mozilla remains uncertain.Further studies are needed to assess generalizability to other systems.
8 RELATED WORK
Prior research established flaky-test causes, consequences, and automated detection or repair techniques; this study complements it by examining developers’ perspectives and practices.
- Prior empirical studies: The three most common previously reported causes involve asynchronous waits, concurrency, and test-order dependency.These findings came from manual analysis of flaky-test fixes.
- Detection and repair: Researchers linked test smells to flakiness and studied test dependency, isolation, shared memory, and automated detection techniques.Examples include refactoring, fault localization, test isolation, shared-object tracking, and DeFlaker.
- Concurrency research: Concurrency has been studied through erroneous patterns, concurrency-bug characteristics, and automated repair methods.The present work is described as complementary to these studies.
9 CONCLUSION
This study contributes evidence on newly identified flaky-test categories, their fixing effort, production-code origins, and organizational consequences. It also highlights needs for improved research and tools to help practitioners manage flakiness.
- Four new flaky-test categories are identified as important targets for researchers and practitioners.
- Flakiness can originate in production code, not only in test code.
- Timeout-related flakiness can require substantial fixing effort, highlighting an unexpected cost of flaky tests.
- Flaky tests can induce organizational problems, including resource-allocation issues.
- The findings motivate improved research techniques and tools to help developers handle flaky tests.