PaperlayerView exact revision on arXiv

Source-linked AI summary

Test Case Selection and Prioritization Using Machine Learning: A Systematic Literature Review

Rongqi Pan, Mojtaba Bagherzadeh, Taher A. Ghaleb, Lionel Briand

arXiv:2106.13891v2cs.SE

TL;DR

Frequent CI builds make full regression testing costly, creating a need to understand ML-based test case selection and prioritization. The paper conducts a systematic literature review of this area, synthesizing evidence from primary studies across five research questions. It identifies commonly used techniques and features, while highlighting challenges in evaluation, reproducibility, and practical adoption.

  • Problem

    Frequent CI builds and large codebases make running all regression tests time-consuming and resource-intensive, motivating analysis of ML-based test case selection and prioritization.

  • Method

    The paper conducts a systematic literature review of ML-based TSP, examining techniques, features, evaluation, performance, reproducibility, and related limitations.

  • Results

    The review identifies four main ML technique groups, finds that execution history, coverage, complexity, and textual features are used, and reports challenges in comparing performance and reproducing studies.

  • Takeaways & Limitations

    The review provides a high-level summary that can serve as a taxonomy for classifying future TSP studies.

  • Takeaways & Limitations

    The review may miss relevant studies or information because some synthesis and extraction steps were manual and the search covered only selected repositories.

Abstract

from arXiv · show

Regression testing is an essential activity to assure that software code changes do not adversely affect existing functionalities. With the wide adoption of Continuous Integration (CI) in software projects, which increases the frequency of running software builds, running all tests can be time-consuming and resource-intensive. To alleviate that problem, Test case Selection and Prioritization (TSP) techniques have been proposed to improve regression testing by selecting and prioritizing test cases in order to provide early feedback to developers. In recent years, researchers have relied on Machine Learning (ML) techniques to achieve effective TSP (ML-based TSP). Such techniques help combine information about test cases, from partial and imperfect sources, into accurate prediction models. This work conducts a systematic literature review focused on ML-based TSP techniques, aiming to perform an in-depth analysis of the state of the art, thus gaining insights regarding future avenues of research. To that end, we analyze 29 primary studies published from 2006 to 2020, which have been identified through a systematic and documented process. This paper addresses five research questions addressing variations in ML-based TSP techniques and feature sets for training and testing ML models, alternative metrics used for evaluating the techniques, the performance of techniques, and the reproducibility of the published studies. We summarize the results related to our research questions in a high-level summary that can be used as a taxonomy for classifying future TSP studies.

1 Introduction

Regression testing becomes costly in frequent CI builds, motivating TSP techniques that select and prioritize tests. This review focuses on ML-based TSP, analyzing 29 primary studies and organizing findings across five research questions.

  • Motivation: CI frequency and large codebases make running all applicable regression tests time-consuming and resource-intensive.The run-them-all approach may require many servers and hours or days to complete.
  • Motivation: TSP selects tests sufficient to assess changes and prioritizes tests able to detect faults early.The techniques combine coverage information, execution history, and domain-specific heuristics.
  • Review scope: The review addresses ML techniques, feature sets, evaluation, performance, and reproducibility in ML-based TSP.These topics correspond to five research questions.
  • Review scope: Supervised, unsupervised, reinforcement-learning, and natural-language-processing techniques are used for TSP.The review classifies the main ML approaches into these categories.
  • Review scope: Only six of 29 selected studies, or 21%, are considered reproducible.The review links this result to methodological issues and reduced confidence in reported findings.

2 Research Method

The study uses a documented systematic literature review process covering repository search, screening, eligibility assessment, and data extraction. It ultimately includes 29 papers and extracts information aligned with five research questions.

  • SLR process: The SLR process defines research questions, searches repositories, applies inclusion and exclusion criteria, and synthesizes extracted data.The process is depicted in Figure 1 and follows four main steps.
  • Research questions: The review examines ML techniques, feature types, evaluation methodology, performance, and repeatability or reproducibility.These topics define the five research questions.
  • Search strategy: Searches use digital libraries and repositories selected for popularity and relevance to software engineering.Search strings combine TSP terms with terms covering multiple ML techniques.
  • Search strategy: 1,057 papers were found in online repositories before screening and selection.The search results are reported by repository in Table 1.
  • Study selection: The inclusion criteria require long, English, online, final-stage papers related to regression testing and exclude surveys, SLRs, and systematic mappings.The selection process included title and abstract screening followed by full-text manual review.
  • Data extraction: 29 papers were included, and information was extracted according to the research questions.The authors provide a replication package containing search strings and extracted data.

3 Results

The results section addresses five research questions concerning ML-based test case selection and prioritization. These questions cover techniques, features, evaluation, performance, and reproducibility.

  • Scope: The results section analyzes the application of ML techniques to test case selection and prioritization.
  • Organization: The analysis is organized around five research questions.
  • Organization: The section reports results for the research questions defined in the research method.

3.1 RQ1. What ML techniques are used for TSP and what are the reported strengths and weaknesses of using these techniques?

The reviewed ML-based TSP studies use supervised, unsupervised, reinforcement-learning, and NLP-based models, with different strengths and limitations for TSP and CI settings. RL adapts well to changing CI environments, whereas supervised and unsupervised approaches commonly require periodic model reconstruction.

  • Technique categories: ML-based TSP studies classify into supervised learning, unsupervised learning, reinforcement learning, and NLP-based models.Supervised learning uses classification or ranking; unsupervised learning uses clustering; RL uses adaptive policies; NLP-based methods process textual information.
  • Reinforcement learning: RL studies create environments from CI logs and train agents through interactions, supporting adaptation to changing systems and test suites.RL can incorporate new data into existing models without retraining from scratch.
  • Reinforcement learning: Three RL studies evaluated reward functions that prioritize failed test cases, using instant or delayed rewards calculated in different ways.Examples include failure-count, test-case-failure, time-ranked, and weighted rewards.
  • Reinforcement learning: MAB approaches outperformed RL with an ANN policy model in one comparison and were also compared with genetic algorithms for test prioritization.The MAB evaluation covered three time budgets and eleven case studies.
  • Limitations and research directions: Coverage-based TSP can create scalability issues because the underlying test-suite optimization problem is NP-hard.The constraint becomes particularly relevant for large systems with many test cases.
  • Supervised learning: Supervised learning commonly treats TSP as a ranking problem and can achieve good accuracy with established ranking models.Reported models include pointwise, pairwise, and listwise ranking; MART was reported as the most accurate model in one comparison.
  • Limitations and research directions: NLP-based TSP is identified as a promising direction for deriving similarity-based features from textual development artifacts and source code.The review highlights textual artifacts and code vectorization as sources of potentially important features.
  • Limitations and research directions: Supervised and unsupervised techniques are unsuitable for CI contexts when they require rebuilding models from scratch, while RL studies remain incomplete in algorithm and feature coverage.The review identifies unexamined RL algorithms and reliance by nearly all RL studies on execution history alone.

3.2 RQ2. What are the features used by ML-based TSP techniques?

ML-based TSP studies mainly use features that are easy to collect or compute, including execution history, code complexity, textual data, and coverage information. The review finds that potentially useful data remains underused and calls for broader feature sets alongside analysis of their collection costs and tooling.

  • Feature categories: The reviewed feature hierarchy organizes ML-based TSP inputs into groups including code complexity, textual data, coverage information, and historical execution data.The paper presents this hierarchy in Figure 3 and summarizes paper-level feature use in Table 5.
  • Code complexity: Ten studies used code-complexity metrics, but most relied mainly on Lines of Code because broader metric collection is harder.Only five of those ten studies used a comprehensive list of complexity metrics.
  • Textual data: Seven studies used textual data, which the review presents as promising for more accurate TSP, especially when source-code access or analysis is unavailable.Textual artifacts can provide training features without requiring direct source-code analysis.
  • Coverage information: Coverage features appeared in fifteen studies, using dynamic, static, or code-similarity analysis.Dynamic analysis is more precise but time-consuming and platform-dependent; static analysis is easier but can overestimate coverage and change impact.
  • Historical execution data: Historical execution features were used by fifteen studies and are generally easy to collect through CI automation.Examples include execution time, verdicts, test age, average execution time, and failure rate.
  • User inputs: Only three studies used user inputs, whose collection requires manual effort and is less practical for CI and large projects.One approach used customer-assigned priorities based on preferences, time, and execution budget.
  • Data collection and tooling: Most studies did not report practical challenges in collecting coverage data, coverage-analysis complexity, or computation time across case studies.The review also notes gaps in tools for automatically extracting test execution details from popular CI build logs.
  • Research directions: The review concludes that existing techniques underuse potentially relevant data and recommends broader feature sets with explicit benefit-cost analysis.Suggested additions include CI-related features such as build-configuration complexity, runtime environments, and timeouts.

3.3 RQ3. How are ML-based TSP techniques evaluated?

ML-based TSP techniques are evaluated with diverse metrics and software subjects, but inconsistent metric definitions and sparse failures limit meaningful comparisons and practical conclusions.

  • Evaluation metrics: Eight evaluation metrics fall into specific TSP metrics, such as APFD and its extensions, and general classification metrics.APFD measures how early faults are detected, whereas general metrics assess classification performance.
  • Evaluation metrics: APFD measures the area under the curve relating executed test-case proportions to detected-fault proportions, with higher values indicating faster fault detection.APFD ranges from 0 to 1 and can also use test-case failures when detected faults are unavailable.
  • Evaluation metrics: APFDc incorporates test execution cost and fault severity, while NAPFD addresses limited-budget runs where not all faults may be detected.NAPFD uses the ratio of detected to total faults within the prioritized test suite.
  • Evaluation limitations: APFD-family metrics are vulnerable to sparse failures: few faults can cause large value changes, while fault-free builds make calculation impossible and require biased assumptions.All APFD metrics assume faults occur frequently enough for accurate assessment.
  • Comparability: General metrics can be incomparable because measures such as True Positive are defined differently across test selection and prioritization contexts.APFD calculations were consistent across studies using APFD, making those comparisons more meaningful.
  • Used subjects: Most subjects provide execution history or coverage information, but their short regression-testing times often make TSP’s practical value limited.Among 92 subjects, 71% had execution times below 90 seconds and the median was 5.84 seconds.
  • Used subjects: Subject validity also depends on sufficient builds and failed builds, which support randomness assessment and balanced datasets for many ML techniques.The review reports 40 subjects with build data, but half had fewer than 10 failed builds.
  • Conclusion: Overall, metric variation, unequal treatment of test-case verdicts, and extremely low failure rates make results difficult to compare across ML-based TSP studies.The review recommends newer CI subjects with adequate failure data for evaluation.

3.4 RQ4. What is the performance of ML-based TSP techniques?

ML-based TSP performance is difficult to compare because studies use varied metrics, baselines, feature sets, and subjects. Execution history is associated with higher NAPFD, but its effect is confounded with recent reinforcement-learning studies and incomplete feature combinations.

  • Baselines: Nine studies compared proposed techniques with random or cost-only baselines.Random ordering and execution-time ordering were the most common comparison baselines.
  • Feature sets: Code complexity, coverage information, and execution history each had statistically significant effects and large effect sizes on NAPFD.Textual and user-input features showed no significant relationship with NAPFD, likely because they were rarely used.
  • Feature sets: Execution history was associated with median NAPFD values of 0.968 with and 0.830 without the feature set.In contrast, median NAPFD was 0.730 with code complexity and 0.968 without it, an unexpected association examined through feature and model combinations.
  • Confounding factors: Fewer than 7% of subjects combined coverage and complexity with execution history, and none of those subjects used reinforcement learning.This partial feature coverage prevents cleanly separating the effects of feature sets from learning techniques.
  • Future evaluation: Future evaluations should combine comprehensive feature sets with varied learning techniques, including reinforcement learning, across many subjects.Such designs would help investigate combined feature and learning-technique effects without the current confounding.

3.5 RQ5. Are ML-based TSP studies repeatable and reproducible?

The review assesses repeatability and reproducibility through required artifacts and reporting criteria rather than rerunning experiments. Only a small minority of ML-based TSP studies met the review’s reproducibility standards, while many datasets do not reflect current CI practice.

  • Definitions: A repeatable study provides enough artifacts to rerun the same experiments, whereas a reproducible study also supports checking statistical equivalence of results and conclusions.The review defines an experiment as training on previous builds, applying the model to next builds, and running the selected or prioritized tests.
  • Assessment criteria: The assessment checks datasets, scripts, algorithm and environment details, experimental descriptions, complete per-subject results, and statistical-analysis information.The criteria include M1–M7, covering artifacts needed to execute experiments and evaluate whether reported conclusions can be checked.
  • Findings: Only five of 12 papers that provided datasets were reproducible, while only two were repeatable.The review did not rerun experiments; it assessed the degree to which papers supported the required criteria.
  • Dataset scope: Twenty-one papers used publicly available datasets containing code changes from before 2016, which may not represent current continuous-development and integration practices.The review recommends newer CI-context subjects with broader features, higher failure rates, and larger execution times.
  • Reporting gaps: Providing datasets alone did not ensure repeatability because some papers omitted technical details such as language or library versions.The review argues that reproducible studies need detailed, consistently applied reporting guidelines.

3.6 ML-based TSP Summary

The paper summarizes ML-based TSP studies across learning techniques, feature sets, evaluation metrics, and reproducibility criteria. This high-level synthesis is intended as a taxonomy for classifying future TSP studies.

  • Taxonomy scope: The summary characterizes ML techniques, model-training feature sets, evaluation metrics, and criteria for repeatability or reproducibility.It consolidates the review’s research-question findings into a single overview.
  • Taxonomy scope: Figure 8 presents the high-level summary of ML-based TSP.The figure is positioned as a taxonomy for future classification of TSP studies.

4 Discussion

The discussion links ML-based TSP performance to subjects, features, models, and CI conditions. It highlights execution history and incremental learning as relevant to CI, while emphasizing the need for standardized evaluations and current benchmarks.

  • Subjects, features, and models: TSP performance varies with project characteristics, ML models, and feature sets, so evaluations should cover diverse subjects and compute many relevant test and code features.The discussion also identifies advanced NLP similarity methods such as BERT and CodeBERT as future options.
  • CI: Execution history is significantly associated with higher TSP performance, and CI supplies historical execution data under varied runtime environments.The discussion connects CI’s frequent builds and rich history with the potential use of TSP models.
  • Standards and benchmarks: No standard practices govern TSP experiment design, feature selection, or evaluation metrics, making cross-study comparisons difficult.Metrics may be computed differently, and few studies build directly on earlier experimental designs.
  • Application scenarios: Reinforcement-learning TSP is described as more suitable for CI than supervised or unsupervised approaches because it can integrate new data without retraining from scratch.The discussion associates supervised batch learning with outdated models and unsupervised clustering with tuning and scalability challenges.

5 Related Work

Earlier reviews examined broader machine-learning applications, test selection, or test prioritization, whereas this review focuses exclusively on machine-learning-based test case selection and prioritization. It provides a more detailed analysis of technique usage, feature sets, performance, advantages, limitations, and research directions.

  • Review scope: The review papers table compares each study’s focus, publication year, covered years, and number of included primary studies.The table summarizes two systematic reviews, two systematic mapping studies, and one review related to machine-learning-based test selection or prioritization.
  • Prior reviews: Existing reviews also studied test selection or prioritization through categories such as cost, coverage, fault effectiveness, goals, techniques, evaluation measures, and continuous-integration challenges.These reviews addressed 47 empirical studies on test selection and 35 primary studies on test prioritization in continuous-integration environments.
  • Prior reviews: Earlier reviews covered machine-learning applications across software testing, including test-case generation, refinement, evaluation, and testing-cost prediction.One mapping study classified 48 primary studies by study type, testing activity, and machine-learning technique.
  • Review scope: This review focuses exclusively on machine-learning techniques for test case selection and prioritization.It analyzes usage and performance across different feature sets and outlines research directions.

6 Threats to Validity

The review’s validity is constrained by possible omissions, subjectivity, extraction inconsistencies, repository coverage, and heterogeneous primary-study evidence. The authors describe mitigation steps, including broader search terms, collaborative screening, validation, detailed procedures, and statistical analysis.

  • Search and selection: The search may have missed relevant studies or information because the review covered many techniques and application contexts and included manually extracted data.Some search and screening steps were automated, while data synthesis and extraction were manual.
  • Search and selection: Search terms might not have been comprehensive enough despite adding specific terms such as regression, neural network, and reinforcement learning.The initial queries used general machine-learning and test case selection and prioritization terms.
  • Data handling: Study inclusion and exclusion can be subjective, although two co-authors collaboratively applied well-defined criteria with Covidence support.The tool was also used to screen collected papers and remove duplicates.
  • Data handling: Manual data extraction may omit or misinterpret study information, despite second-author validation of uncertain cases and discussion-based resolution.One co-author extracted data from all papers, and a second validated difficult or uncertain entries.
  • Scope: The search results are limited to the online repositories used as data sources, although these repositories are widely used in software-engineering surveys.The selected repositories include well-known software-engineering venues.
  • Evidence synthesis: Different subjects and evaluation metrics made clear patterns and definitive conclusions difficult, motivating statistical analysis of associations with TSP performance.The analysis assessed features, subjects, and machine-learning models in relation to performance.
  • Replication: Replication is possible by following the documented search process, but differences in data-extraction judgments may produce inconsistent results.The authors provide detailed search steps and a high-level taxonomy to support replication and future classification.

7 Conclusion

This systematic literature review analyzes machine-learning applications to test case selection and prioritization in continuous-integration contexts using 29 primary studies from 2006–2020. It finds diverse techniques and feature sets, while heterogeneous subjects and metrics prevent reliable performance conclusions and motivate common evaluation procedures.

  • Contribution: The review presents a high-level synthesis that can support classification of future TSP studies.Its stated purpose is to summarize how machine learning has been used, what it has achieved, and its limitations.
  • Review scope: The review analyzes 29 primary studies published from 2006 to 2020 and reports growing interest in machine-learning-based test case selection and prioritization.The search also returned systematic reviews and mapping studies.
  • Review scope: The review investigates machine-learning techniques, hyper-parameters, experiment designs, evaluation metrics, experimental results, and limitations through five research questions.The authors extracted and synthesized data from the selected studies.
  • Findings: The four main machine-learning categories used for TSP are supervised learning, unsupervised learning, reinforcement learning, and NLP-based techniques.NLP techniques were often combined with supervised or unsupervised learning for feature preprocessing and test case prioritization.
  • Findings: Studies used execution history, coverage information, user inputs, textual data, code complexity, and domain-specific features, but most techniques relied mainly on historical data.Coverage and complexity features were used less extensively, supporting further analysis of feature impacts on accuracy.
  • Findings: Reinforcement-learning research has grown recently, focusing on reward functions and policy learning, while existing work mainly uses execution-history features.The review identifies broader feature sets as a potentially beneficial direction for reinforcement-learning-based TSP.
  • Limitations and future work: Different subjects and evaluation metrics, together with a lack of standard procedures and suitable public subjects, make reliable performance conclusions challenging.The authors call for convergence toward common evaluation procedures and metrics.

Authors’ Biographies

The authors are researchers in software engineering, machine learning, software testing, and empirical software engineering, with affiliations spanning the University of Ottawa and the University of Luxembourg.

  • Authors: Rongqi Pan is a University of Ottawa PhD student whose research focuses on machine learning and software regression testing in continuous integration.She is a member of Nanda Lab and holds a master’s degree in statistics.
  • Authors: Mojtaba Bagherzadeh is a University of Ottawa postdoctoral fellow researching machine-learning-based systems, model-driven engineering, software testing, and empirical software engineering.He has software-development experience at IBM and a startup company.
  • Authors: Taher A. Ghaleb is a University of Ottawa postdoctoral fellow with doctoral training in computing and experience as a research and teaching assistant.His biography identifies education and academic appointments but the supplied passage ends before listing his research interests.
  • Authors: Lionel C. Briand is a software-engineering professor with appointments at the University of Ottawa and the University of Luxembourg.He leads the SVV department at the SnT Centre and holds a Canada Research Chair in Intelligent Software Dependability and Compliance.
Loading 2106.13891v2…