Understanding and Detecting Flaky Builds in GitHub Actions

1 Introduction

• GitHub Actions is described as a leading CI service, whose integration and rerun records enable this study of flaky builds.Source: §1 · pp. 1–2
• CI failures can occur intermittently because of non-deterministic factors, and builds that change outcomes after reruns without code changes are defined as flaky builds.Such unreliability erodes trust, wastes effort and resources, and threatens the validity of CI studies and build-outcome prediction models.Source: §1 · p. 1
• 3.2% of builds were rerun, 67.73% of those were flaky, and the proposed detector improved F1-score by up to 20.3% over the state-of-the-art baseline.The study analyzed rerun data from 1,960 open-source Java projects.Source: §1 · p. 2
• The analysis identified 15 flaky-failure categories, with flaky tests, network issues, and dependency resolution issues the most prevalent.Source: §1 · p. 2
• Prior rerun studies did not specifically target GitHub Actions, while prior flaky-failure detection studies focused on industrial CI systems and features often unavailable in open-source projects.Source: §1 · p. 2
• The paper constructs two GitHub Actions datasets: a build dataset from 1,960 open-source Java projects and a curated flaky-failure dataset from 10 projects.Source: §1 · p. 2

Workflows, Builds, and Jobs

• A workflow comprises one or more jobs, each made of task-oriented steps; the paper calls workflow executions “builds” [13].Jobs in the same build may run in different environments, such as distinct operating systems or runtime versions.Source: §2 · p. 2

Outcomes

• Status tracks execution progress; once a build or job is completed, conclusion records its outcome, while startup_failure, action_required, canceled, and skipped are excluded as incomplete executions.Possible status values include queued, pending, waiting, and in_progress; conclusions include success, failure, and neutral.Source: §2 · pp. 2–3
• A rerun build is flaky if it contains at least one flaky job, and each failure occurrence in such a job is a flaky failure.Organizations may use different terms for flakiness, including “oranges” at Mozilla and “brown build” at Ubisoft.Source: §2 · p. 3

3.1 Data Preparation

• The authors narrowed SEART GitHub Search Engine candidates [12] to 2,042 active Java projects meeting project-quality and GitHub Actions workflow criteria.They excluded forks, selected Java projects, required at least 150 stars and 250 commits, and retained projects with recent commits; 82 candidate repositories lacked collectable build data.Source: §3.1 · pp. 3–4
• The final dataset contains 1,960 repositories, 4,861,768 builds, and 15,382,267 jobs, with rerun and flaky builds identified from attempt metadata and job outcomes.A build is rerun when run_attempt exceeds 1; a job is flaky if it succeeds and fails across attempts, and a build is flaky if it contains at least one flaky job.Source: §3.1 · p. 4

3.2 Cost Measurement

• Waiting time measures elapsed time from the initial build start to the final rerun end, while computational time sums job execution time across the rerun sequence.Computational time is calculated by subtracting each job's start_time from its end_time.Source: §3.2 · pp. 4–5

3.3 Failure Category Labeling

• The failure-labeling pipeline generalized variable tokens into reusable failure patterns, iteratively adding patterns from unmatched logs and grouping related patterns into higher-level categories.The authors began with manual inspection of ten failed jobs, applied patterns to flaky-job logs, then inspected ten unmatched logs and repeated the process until no meaningful new patterns emerged; categories unify different surface forms of the same phenomenon.Source: §3.3 · p. 5

3.4 Flaky Failure Analysis

• Root causes and mitigation patterns were identified through iterative manual sampling of five flaky cases per category, continuing until two consecutive iterations found no new themes.When needed, the analysis also consulted issue reports and documentation.Source: §3.4 · p. 5

4.1 Frequency Analysis (RQ1)

• 3.2% of 4,879,460 builds were rerun; after excluding approval-triggered reruns, 52,610 remained, affecting 60.5% of projects.Of the original rerun builds, 91.6% were rerun exactly once; approval-triggered builds accumulated 84,529 days of waiting, with average and median delays of 20 hours and 1.5 hours.Source: §4.1 · p. 6
• Among 52,610 rerun builds, 35,634 (67.73%) exhibited flaky behavior, affecting 1,005 (51.28%) of the 1,960 projects.Source: §4.1 · p. 6
• 10,406 flaky builds needed multiple reruns before an outcome change, and some required up to 20 reruns.Flakiness prevalence is likely underestimated because the study covered only several months of build histories; reruns can fail to expose flakiness or require multiple attempts.Source: §4.1 · pp. 6–7
• 339.2 years of waiting time and over 31.6 years of computational time were incurred by rerun builds.Rerun builds had 63-times longer average and 60-times longer median waiting times, and 16.7-times longer average and 51-times longer median computational times than non-rerun builds (p<0.0001 for both comparisons).Source: §4.1 · pp. 6–7

4.2 Failure Analysis (RQ2)

• The analysis identified 15 failure categories: flaky tests accounted for 64.99% of flaky failures, network issues 15.8%, and dependency resolution issues 6.32%.The table presents the categories, root causes, and mitigation patterns; the full category list is available on the project website.Source: §4.2 · p. 7, Table 1 · p. 9
• External environment inconsistencies (4.7%), API service unavailability (1.16%), and concurrency issues (0.88%) arose from policy, quota, or shared-resource problems.Mitigations included satisfying workflow policies, refreshing credentials, waiting for quota recovery, and rerunning to change execution timing.Source: §4.2 · p. 8
• Compilation errors (0.82%), execution crashes (0.78%), and memory-limit failures (0.5%) involved caches, runner instability, memory contention, or exhausted disk space.Mitigations included rerunning, refreshing caches, updating runners, uploading missing artifacts, cleaning files, or adding disk capacity.Source: §4.2 · pp. 8, 10
• File-system interaction errors (0.71%) and static-analysis errors (0.66%) included transient downloads, rate limits, upstream changes, and inconsistent external resources.Across categories, reruns were the most common mitigation but often only a temporary workaround; root causes could produce different symptoms and frequently involved external services or upstream repositories.Source: §4.2 · pp. 9–10

5 Flake Detector

• FlakeDetector targets failed jobs because a flaky build can contain jobs that are not themselves flaky.The labeled dataset retains initially failed jobs; developer-triggered reruns that later succeed label a failure flaky, while limited or absent reruns can leave low-probability flaky failures undiscovered.Source: §5 · p. 10, §5.1 · p. 11

5.1 Data Collection

• CI-Rerunner reran failed jobs up to 10 times, finding 83 additional flaky jobs; 99% of flaky jobs succeeded within five reruns.A manual audit of 50 jobs that failed all 10 reruns found 4 additional flaky jobs. The authors describe automated rerunning as a scalable compromise: it improves labeling over developer-triggered reruns but does not eliminate false negatives.Source: §5.1 · p. 11

5.2 Feature Engineering

• The log pipeline filters error-pattern lines, normalizes changing text, embeds logs with all-MiniLM-L6-v2, and retrieves top-K similar historical logs from a training-only vector database.Vectors are stored in FAISS; concatenated neighbor similarity scores and labels feed logistic regression, whose probability is the log confidence score.Source: §5.2 · pp. 11–12

5.3 Structured Feature Analysis

• CI-Miner complements log semantics with structured features spanning developer, code-change, and project/context dimensions, and oversampling balances the failed-job training data.Features include developer experience and trust, change size and AST-level structure, and project history and environment, including recent failures, Dockerfile changes, and concurrent jobs.Source: §5.3 · pp. 12–13
• Mutual Information ranks structured features for top-N selection, and the study evaluates XGBoost, Random Forest, SVM, and MLP to produce a structured flakiness confidence score.The number of selected features is tuned as a hyperparameter; the structured score complements log analysis.Source: §5.3 · pp. 13–14

5.4 Classification

• The classifier combines log and structured confidence scores with a weighted sum, then applies a probability threshold to assign a binary flakiness label.The weight controls the relative contribution of log information, and the weight and threshold are optimized during validation.Source: §5.4 · p. 14

5.5 Validation

• Forward chaining addresses temporal leakage by training on earlier builds and evaluating on subsequent data, with results averaged across 10 data groups.Training grows from 50% to 90% in 10% increments; each round uses the subsequent 5% for validation and testing, and an additional pass swaps those sets.Source: §5.5 · pp. 14–15

6.1 Setup

• The evaluation randomly selected 10 of 674 projects without secrets, whose credentials would prevent successful reruns.Table 3 summarizes their CI-job and flaky-failure statistics.Source: §6.1 · p. 15
• The baseline [25] cleans logs, applies TF-IDF, and combines two XGBoost models; both systems used chronological forward-chaining, grid search, and precision, recall, F1, and AUC.The shared splits and tuning approach were intended to make performance differences reflect model architecture rather than data partitioning or tuning effort.Source: §6.1 · pp. 15–16

6.2 Results

• 20.3% relative improvement: average F1 rose from 0.667 for the baseline to 0.803 for SVM, and the proposed models consistently outperformed the baseline across projects.For eclipse/xtext, baseline F1 was 0.202, compared with 0.745 for MLP and 0.770 for SVM.Source: Table 4 · p. 16, §6.2 · p. 16
• MLP led average Precision (0.766) and AUC (0.852), while Random Forest led Recall (0.915); precision and recall complement AUC under class imbalance.MLP's precision is useful when reducing false alarms matters, while Random Forest's recall suits goals prioritizing detection of as many flaky builds as possible.Source: §6.2 · p. 16
• The committer Bayesian Trust Score ranked highest in Mutual Information across most projects, frequently exceeding 0.6, followed by SLOC and test lines; these are correlations, not causal effects.Committer trust may proxy for code ownership or change complexity, whose individual effects need further investigation.Source: §6.2 · pp. 16–17

7 Threats to Validity

• The study focuses on open-source Java projects using GitHub Actions, which the authors describe as representative of a broad Java and CI-service audience.The authors nevertheless limit the study scope to that project and service setting.Source: §7 · p. 17
• Flaky-failure labeling may be imperfect because flakiness is inherently difficult to observe.The study mitigates this threat with an automated rerun tool combined with manual inspection.Source: §7 · p. 17

• 33% of broken builds were attributed to environmental factors, while prior studies also found frequent flaky tests that erode CI confidence.Source: §8 · p. 17
• 1.72% of Travis CI builds were restarted, and 46.77% of restarted builds changed from failing to passing; failures, network issues, and Travis CI limitations were leading causes [13].Restarted builds also delayed pull-request merging.Source: §8 · p. 17
• Rechecked builds in OpenStack reviews changed outcomes in 42% of rechecked jobs and consumed 187.4 years of computation and 16.8 years of waiting time [24].55% of the 66,932 reviews had at least one rechecked patch set.Source: §8 · p. 17
• 11% of successful jobs in 81 industrial projects were rerun; Aïdasso et al. identified 46 categories of flaky job failures at TELUS [3] [2].Their silent failures frequently involved artifact operations, caching, and ignored exit codes; studies of flaky builds have focused mainly on Travis CI or industrial systems, unlike this GitHub Actions study.Source: §8 · pp. 17–18
• Lampel et al. achieved 73% precision using CI job features, while Olewicki et al. reported a 53% mean F1 score for log-similarity detection; flaky tests are only one cause of flaky builds [18].CPU load, used by Lampel et al., is not readily available in open-source CI environments.Source: §8 · p. 18

9 Conclusions

• The empirical study analyzes rerun data from 1,960 open-source Java projects using GitHub Actions.Source: §9 · p. 18
• The study finds pervasive build flakiness and reports that rerunning builds is costly and often ineffective.It identifies 14 failure categories beyond flaky tests, including network instability, dependency resolution issues, and external service failures.Source: §9 · p. 18
• FlakeDetector detects flaky failures by combining build-log semantics with CI metadata.Source: §9 · p. 18