Source-linked AI summary
Reinforcement Learning for Automatic Test Case Prioritization and Selection in Continuous Integration
Helge Spieker, Arnaud Gotlieb, Dusica Marijan, Morten Mossige
TL;DR
CI must prioritize and select promising tests despite uncertain code-change impacts, unavailable traceability, time limits, and changing test suites. Retecs applies online, model-free reinforcement learning using historical test metadata and feedback, and evaluation across three industrial case studies reports fast adaptation and performance comparable to basic deterministic methods after about 60 CI cycles. The authors also report that the method requires no source-code access, while expecting additional metadata to improve performance.
Problem
CI test selection and prioritization are difficult when code–test traceability is unavailable, execution time is limited, and test suites and priorities change.
Method
Retecs is an online reinforcement-learning method that uses test results, durations, last execution times, actions, and reward feedback to adapt schedules without source-code access.
Results
Evaluation across three industrial case studies reports fast learning and adaptation, with performance comparable to basic deterministic prioritization after approximately 60 CI cycles without previous training.
Takeaways & Limitations
Retecs provides a lightweight, model-free, language-agnostic approach requiring only test metadata and historical results for adaptive CI test prioritization and selection.
Takeaways & Limitations
The implementation evaluation is exposed to random-decision effects and possible implementation faults, which the authors mitigate through repeated experiments, established components, and public availability.
Abstract
from arXiv · showhide
Testing in Continuous Integration (CI) involves test case prioritization, selection, and execution at each cycle. Selecting the most promising test cases to detect bugs is hard if there are uncertainties on the impact of committed code changes or, if traceability links between code and tests are not available. This paper introduces Retecs, a new method for automatically learning test case selection and prioritization in CI with the goal to minimize the round-trip time between code commits and developer feedback on failed test cases. The Retecs method uses reinforcement learning to select and prioritize test cases according to their duration, previous last execution and failure history. In a constantly changing environment, where new test cases are created and obsolete test cases are deleted, the Retecs method learns to prioritize error-prone test cases higher under guidance of a reward function and by observing previous CI cycles. By applying Retecs on data extracted from three industrial case studies, we show for the first time that reinforcement learning enables fruitful automatic adaptive test case selection and prioritization in CI and regression testing.
1 INTRODUCTION
Continuous Integration relies on automatic regression testing to detect defects early, but selecting and prioritizing promising tests is difficult when code–test traceability is unavailable and test environments change. Retecs frames this challenge as online reinforcement learning and reports adaptive learning on industrial CI data.
- Continuous Integration combines frequent integration with automatic builds and regression testing to detect defects early while controlling verification costs.
- CI test selection must identify tests likely to detect failures early, especially when traceability links between system tests and changed code are unavailable.
- Non-adaptive prioritization is constrained by limited selection time, changing test repositories, shifting feature priorities, and evolving execution environments.
- Retecs adapts to added or deleted tests and changing priorities by updating knowledge from failures and actions across recent CI cycles.
- The paper models history-based test selection and prioritization as reinforcement learning using states, actions, policies, and reward functions.
- An online RL method learns test prioritization without a previous training phase and is reported as the first RL application compared with other approaches.
- Experiments use one year of CI data from industrial settings and report that the method is deployable in those settings.
2 FORMAL DEFINITIONS
The paper formalizes CI test prioritization as maximizing early failure-detection performance under execution-time limits and changing test suites, using historical execution information. It defines the test-suite objects, performance measures, and reinforcement-learning representation used for the problem.
- 2.1 Notations and Definitions: In 2.1 Notations and Definitions, each CI cycle has a potentially changing unordered test set, from which an ordered selected schedule is formed.
- 2.1 Notations and Definitions: In 2.1 Notations and Definitions, rank(t) denotes a test case’s position within the ordered selected schedule.
- 2.1 Notations and Definitions: In 2.1 Notations and Definitions, verdicts and durations become available after execution; failed cases include executed failures and unexecuted cases coded as verdict 0.
- 2.1 Notations and Definitions: In 2.1 Notations and Definitions, test-case performance estimates efficiency for detecting failures, while suite performance may aggregate individual estimates using functions such as sum, maximum, or average.
- 2.2 Problem Formulation: In 2.2 Problem Formulation, test case prioritization seeks an ordering that maximizes failure-detection performance and reveals failures as early as possible.
- 2.2 Problem Formulation: In 2.2 Problem Formulation, time-limited prioritization additionally requires the selected suite’s total duration to remain below the available execution time M.
- 2.2 Problem Formulation: In 2.2 Problem Formulation, the CI problem repeats time-constrained prioritization each cycle while incorporating historical test-execution information when code–test links are unavailable.
3 THE RETECS METHOD
Retecs applies model-free, online reinforcement learning to prioritize and select test cases in CI. It learns from test metadata and execution feedback, adapting priorities and schedules as testing conditions change.
- RL formulation: Retecs formulates test case prioritization and selection as a reinforcement learning problem.The approach uses states, actions, agents, policies, and reward functions to guide prioritization and selection.
- RL formulation: Retecs learns online without an initial environment model, allowing priorities to adapt as failure indicators and test suites change.Online learning addresses changing development focus, execution platforms, and test-case availability.
- RL formulation: Each test case is represented by metadata including approximated duration, last execution time, and previous execution results.The agent returns a priority for each test case before the prioritized suite is scheduled and executed.
- Reward functions: The reward functions use available execution outcomes to reinforce schedules that detect failures, but CI feedback cannot automatically penalize unexecuted tests.Consequently, Retecs rewards are zero or positive rather than directly measuring missed failures.
- Reward functions: Retecs evaluates failure-count, test-case-failure, and time-ranked rewards to guide test-case prioritization.The time-ranked reward additionally accounts for test order and penalizes passed tests placed before failures.
- Memory representation: The tableau representation stores state-action counts and average rewards, selecting the action with the highest expected reward for the current state.Its use is limited because states and actions must be discrete and convergence slows as their number grows.
- CI integration: Retecs integrates with CI by providing prioritization and selection/scheduling while receiving test results through an added feedback channel after each cycle.The feedback is the same as, or part of, information supplied as developer feedback.
4 EXPERIMENTAL EVALUATION
The evaluation examines whether Retecs can prioritize and select tests effectively and whether its lightweight, model-free approach is comparable to deterministic methods. Experiments use three industrial datasets and compare multiple Retecs configurations and baselines.
- Evaluation design: The evaluation first studies how parameter choices affect Retecs performance before addressing its research questions.The parameter exploration precedes the main evaluation of the method.
- Evaluation design: RQ1 evaluates Retecs effectiveness across combinations of memory representations and reward functions on three industrial datasets.The study examines whether the method can successfully prioritize and select test cases.
- Evaluation design: RQ2 compares the lightweight, model-free Retecs method with deterministic, domain-specific methods and random prioritization.The comparison includes three comparison methods alongside basic deterministic approaches.
4.1 Evaluation Metric
The evaluation uses Normalized Average Percentage of Faults Detected as its metric for test prioritization and selection. NAPFD extends APFD by accounting for the proportion of detected failures, making it suitable when only a subset of tests is selected.
- Metric definition: The evaluation introduces Normalized Average Percentage of Faults Detected as a common performance metric.The metric is used to compare different test prioritization and selection methods.
- Metric definition: APFD measures prioritization quality from the ranks of failure-detecting test cases in execution order.It assumes that all detectable faults are detected and therefore targets prioritization without subset selection.
- Metric definition: NAPFD extends APFD with the ratio of detected to detectable failures, supporting evaluation when not all test cases are executed.This extension makes the metric suitable for test case selection tasks.
4.2 Experimental Setup
The experimental setup evaluates two reinforcement-learning agents against three basic prioritization methods on industrial and shared test-suite data. Experiments use historical CI outcomes, repeated runs, a relative time limit, and parameter studies including history length.
- Agents: Two RL agents are evaluated: a Tableau-based agent for discrete states and a Network-based agent using an artificial neural network for continuous states.Their reward functions vary across experiments.
- Execution: Test cases are scheduled in descending priority until the CI cycle’s time limit is reached.The setup therefore evaluates prioritization together with time-constrained selection.
- Baselines: Retecs is compared with Random, Sorting by recent verdicts, and Weighting based on summed test-case features.These comparison methods use the same information as Retecs and represent approaches likely to occur in industrial environments.
- Evaluation: Evaluation compares NAPFD across subsequent CI cycles, with each experiment repeated 30 times and results reported as means unless stated otherwise.The longitudinal evaluation reflects Retecs’s online-learning setting.
- Data Sets: The study uses ABB Robotics, Paint Control, IOF/ROL, and GSDTSR data containing historical executions and verdicts across more than 300 CI cycles.ABB data are divided into daily intervals, while GSDTSR uses hourly intervals and has larger average suites but fewer failed executions.
- Parameter Selection: A longer execution history does not necessarily improve performance because it enlarges the state space, increasing complexity and the data required for adaptation.The history-length effect is evaluated on the ABB Paint Control data set.
4.3 Results
Retecs’ performance depends strongly on the agent’s memory representation and reward function, with the Network-based agent using Test Case Failure reward performing best overall. Compared with deterministic methods, Retecs initially explores but learns competitive prioritization over subsequent CI cycles, while shorter schedules slow learning and industrial-data noise causes fluctuations.
- RQ1: Learning Process & Effectiveness: Memory representation and reward function strongly influenced performance, with some combinations failing to improve beyond their initial level.The reward design affected whether agents learned meaningful test schedules.
- RQ1: Learning Process & Effectiveness: The Network-based agent with Test Case Failure reward delivered the best performance across all three data sets.Its test-case-specific feedback supports learning which tests are likely to fail.
- RQ1: Learning Process & Effectiveness: Performance fluctuated with noisy industrial data, including a Paint Control drop when many manually added test cases repeatedly failed.These manually adjusted cycles raise questions about fully automated prioritization during such periods.
- RQ2: Comparison to Other Methods: Retecs initially underperformed deterministic comparison methods during exploration, then reached similar or better performance after approximately 60 CI cycles on Paint Control.Other data sets showed longer adaptation or early comparable performance.
- RQ2: Comparison to Other Methods: On GSDTSR, Retecs was overall comparable to the comparison methods, with an NAPFD difference of up to 0.2.Few failures reduced the impact of exploration on early-cycle performance.
- Internal Evaluation: Schedule Time Influence: Lower scheduling-time limits reduced learning information for RL agents, but performance did not decrease proportionally to the time reduction.Retecs still learned to prioritize test cases despite limited data from previous cycles.
4.4 Threats to Validity
The evaluation is constrained by randomness, implementation faults, parameter sensitivity, limited metadata, and a small set of industrial data and comparison methods.
- Experiments were repeated 30 times and averaged to mitigate the influence of random decisions.
- The implementation threat was addressed through established components and public availability for inspection and experiment reproduction.
- The evaluation used fixed parameters across problems, although feasible settings may differ between environments.The authors state that real-world deployments can tune parameters for specific environments.
- Three industrial data sets limit conclusions across the wider variety of CI environments and failure distributions.The authors note that further experiments require additional accessible data.
- The method uses few test-case metadata features and was compared with baseline approaches without considering additional techniques.The authors suggest using more information about test cases or the system under test and note that other methods lacked comparable data or required adaptation.
4.5 Extensions
The paper proposes extensions through pre-training the reinforcement-learning agent and supporting more complex industrial scheduling environments.
- Pre-training could improve initial performance by imitating desirable deterministic methods or using historical data before CI deployment.
- Future work could integrate multiple scheduling methods and constraints into the learning process.The current work uses a linear scheduling model, whereas industrial environments may involve multiple systems and constraints beyond time limits.
5 RELATED WORK
Prior regression-testing research mainly optimizes cost, coverage, fault detection, or combinations of these criteria, while related CI work uses historical results and adaptive learning.
- Test case prioritization and selection for regression testing: Regression-test prioritization and selection commonly optimize cost, coverage, fault detection, or combinations of these aspects.
- Test case prioritization and selection for regression testing: Existing approaches include integer linear programming, greedy, coverage-based, search-based, swarm, ant-colony, and genetic algorithms.
- Machine learning for software testing: Retecs uses historical results and experience from previous CI cycles and is designed for evolving code and test suites.
6 CONCLUSION
Retecs is a lightweight, adaptive reinforcement-learning method for CI test prioritization and selection. Across three industrial case studies, it learned an effective strategy without prior training and reached performance comparable to basic deterministic methods after an initial learning period.
- Retecs combines reinforcement learning with historical test information to adaptively prioritize and select CI test cases.It learns indicators of failing test cases by observing tests, results, actions, and their effects.
- After approximately 60 CI cycles without prior training, Retecs achieved performance comparable to basic deterministic prioritization methods.
- Retecs is model-free, language-agnostic, and requires only historical results, durations, and last execution times rather than source-code access.The authors expect additional metadata to enhance performance.
- Neural-network RL agents performed best, especially with test-case-individual reward functions.The study used small networks and identifies larger networks and deep learning as a future research direction.