Source-linked AI summary
Evaluating Change Point Detection Methods for Software Performance Regression Analysis
Diego Elias Costa, Michele Tucci, Luca Traini, Daniele Di Pompeo, Thomas Bach, Francois Farquet, David Daly, Simon Eismann, Petr Tůma, Vittorio Cortellessa, André van Hoorn
TL;DR
Evidence on the effectiveness of change point detection for software performance measurements remains limited, despite the importance of identifying performance changes. The paper characterizes time series from three industrial systems, gathers expert annotations, and evaluates 12 methods; effectiveness varies across datasets, while tuning improves scores and binary segmentation performs consistently well.
Problem
Method
The study characterizes three industrial datasets, collects structured expert annotations of potential performance changes, and evaluates 12 CPD algorithms against them.
Source: §11 · p. 27Results
Average default F1-scores ranged from 0.65 to 0.84 across datasets and rose to 0.70 to 0.90 with tuning; BINSEG was most stable and consistently high-performing.
Source: §11 · p. 27Takeaways & Limitations
CPD performance differs substantially across datasets and configurations, and tuning can markedly improve effectiveness, primarily through improved precision.
Source: §11 · p. 27The three industrial datasets do not encompass all performance-critical domains, and the study covers time-based and throughput metrics rather than resource- or event-based metrics.
Source: §10 · p. 26
Abstract
from arXiv · showhide
Performance issues in software systems are a critical quality issue that can erode user trust, violate service-level agreements, and ultimately affect business efficiency. Consequently, software performance engineering has shifted its focus to developing robust techniques to detect performance regressions as early as possible in the development cycle. Performance regression analysis often relies on time series of performance measurements to detect significant changes in performance behavior. Change Point Detection (CPD) methods have been widely used to automate the identification of such changes in various domains, including finance, healthcare, and performance monitoring. However, the effectiveness of these methods for software performance measurements has not been thoroughly evaluated. In this paper, we present a comprehensive study to evaluate the effectiveness of various CPD methods on real-world software performance measurement datasets. We start by collecting performance measurement data from three large software systems and characterizing the unique properties of performance time series data. Then, we undertake a large-scale effort to annotate and evaluate the consistency of human annotators' identification of potential performance changes. Thereafter, we evaluate the accuracy of twelve distinct CPD methods in detecting potential performance changes, providing insights into their applicability and effectiveness in software performance regression analysis.
1 Introduction
The paper addresses limited evidence on how well different change point detection methods identify software performance changes, despite the difficulty of interpreting variable measurements and the risks of threshold-based detection. It characterizes performance time series, builds an expert-annotated dataset, and evaluates 12 methods; results show that effectiveness varies by infrastructure and improves with optimal tuning.
1 Introduction
- Evidence on CPD for performance change detection was limited to one algorithm and a few industrial reports, with no rigorous, extensive empirical evaluation.Performance measurements also vary substantially and are sensitive to changes in the testing environment, while simple thresholds can produce false positives.Source: §1 · pp. 2–3
- F1-score improved by 28% on average when CPD methods were optimally tuned, while effectiveness varied considerably across performance testing infrastructures.The study also found distinctive statistical properties in performance time series and high, though somewhat subjective, annotator agreement.Source: §1 · p. 3
- The authors characterize software performance time series against time series from other relevant domains and conduct extensive manual labeling to establish a dataset of suspected performance changes.The dataset and analysis scripts are publicly available.Source: §1 · p. 3
- The authors adapted an existing time-series labeling tool with difficulty evaluation, a performance-specific tutorial, and new annotation types.The original tool was proposed by van den Burg and Williams [14].Source: §1 · p. 3
- The study compares 12 CPD algorithms on performance time series from three real-world software systems.The evaluation uses expert-identified suspected changes and examines method effectiveness across the resulting datasets.Source: §1 · p. 3
2 Motivation
Performance measurements can be represented as time series to track system behavior across versions, but identifying performance flaws in them requires specialized techniques. Change point detection partitions a series at shifts in behavior by optimizing segment costs and a penalty on the number of change points.
2 Motivation
- Performance measurements are aggregated over time to track system performance across history, but identifying performance flaws in these series is non-trivial and has received little comprehensive research.The paper frames its analysis around performance time series and change point detection methods.Source: §2 · pp. 3–4
2.1 Change Point Detection methods
- CPD methods partition a time series into segments that represent shifts in behavior, with change points marking the first observation of a new segment.The objective combines segment costs with a penalty on the number of change points; the penalty is controlled by a nonnegative hyperparameter.Source: §2.1 · p. 4
3 Related Work
Prior work describes both the value and limits of human review in performance regression analysis, alongside varied statistical and model-based approaches to detecting changes. This paper motivates expert annotations and a broad comparison because CPD effectiveness may depend on performance time-series characteristics.
3.1 How consistent are experts in flagging suspected performance changes?
- Prior reports describe inconsistent expert judgments and the difficulty of manual analysis, while emphasizing experts’ role in interpreting complex data and validating automated detections.The authors therefore use expert labels to create a ground-truth dataset for evaluating CPD methods.Source: §3.1 · pp. 4–5
3.2 How efficient are CPD methods in identifying suspected performance changes?
- Prior work spans statistical tests, CUSUM, supervised and unsupervised methods, Bayesian detection, PELT, binary segmentation, and E-Divisive variants.Reported findings include high precision and recall for the T-test, reliable CUSUM detection of small changes with low false alarms, and a modified E-Divisive method outperforming PELT on a synthetic dataset.Source: §3.2 · pp. 5–6
- Because CPD effectiveness can vary with time-series characteristics, the authors evaluate a comprehensive set of methods on real-world software performance data to examine their strengths and limitations.The review covers literature on expert consistency and CPD efficiency.Source: §3 · p. 4, §3.2 · p. 6
4 Study Design
The study compares established change point detection methods for software performance regression analysis using data from three large-scale industrial systems. It characterizes the series, gathers expert annotations, and evaluates 12 methods against the annotations and consensus.
4 Study Design
- The study collects performance time series from MongoDB, GraalVM, and SAP HANA to compare established CPD methods for identifying software performance regressions.Its design investigates time-series characteristics, expert annotation consistency, and CPD effectiveness.Source: §4 · p. 6
- RQ1 compares software performance time series with series from domains such as finance and weather.The comparison examines how their characteristics differ.Source: §4 · pp. 6–7
- RQ2 uses 14 software performance experts to annotate 1,500 performance time series and assesses annotation convergence and quality.The annotation effort builds on a platform proposed by van den Burg and Williams [14].Source: §4 · p. 7
- RQ3 compares 12 CPD methods for detecting potential performance changes using the annotated data and consensus from RQ2.Source: §4 · p. 7
5 Datasets
The study assembled performance time series from GraalVM, MongoDB, and SAP HANA, then curated them to support comparisons across datasets. Collection and curation differed by source, while the common process focused on runtime-related measurements and removed short or potentially invalid series.
5.1 GraalVM performance measurements dataset
- GraalVM measurements were collected through a public API over two years, with the first half removed as warm-up and the remainder averaged per system version.The collection covered October 2019 to October 2021; each resulting time-series point represents an average benchmark run.Source: §5.1 · p. 7
5.2 MongoDB performance measurements dataset
- MongoDB data came from a public Zenodo dataset, narrowed to four systems selected with advice from MongoDB engineers.The selected systems were sys-perf, performance, sys-perf-4.4, and performance-4.4.Source: §5.2 · p. 8
5.3 SAP performance measurements dataset
- The SAP HANA dataset contained 7,522 time series spanning about half a year and 84 measurement types, which were averaged by system version.Measurements ranged from elapsed time and throughput to indicators such as I/O operations and commit rate.Source: §5.3 · p. 8
5.4 Curation process
- The common curation retained 253 GraalVM, 14,507 MongoDB, and 6,554 SAP time series.It focused on execution-time or throughput metrics, converted throughput to elapsed time, and excluded series with fewer than 50 measurements or zero values.Source: §5.4 · p. 8
6 Performance Time Series Characteristics
Because performance time-series properties were underexplored relative to domains where CPD is widely used, the study compared three software datasets with four domains from the Libra benchmark. It characterized time-series properties and found that performance data generally combine lower variability with more extreme values, greater stationarity, and lower autocorrelation than series from other domains.
6.1 Selecting Time Series from Other Domains
- The comparison used three performance datasets and four Libra domains, forming a seven-group comparison across time-series characteristics.Libra supplied 400 heterogeneous series from 50 sources.Source: §6.1 · p. 9
6.2 Characterization Methods
- Characterization covered trend, stationarity, autocorrelation, coefficient of variance, skewness, and kurtosis using spline slopes, ADF, and Ljung-Box tests.Trend was extracted after min-max normalization using a third-degree penalized spline with ten knots.Source: §6.2 · pp. 9–10
- Statistical differences across the seven characteristic distributions were tested with Mann-Whitney U, while Cliff's delta quantified effect size.The study used a 5% significance level and customary thresholds to classify effect sizes.Source: §6.2 · p. 10
6.3 RQ1. How performance time series differ from time series from other domains?
- Performance series had lower coefficient of variance and weaker trends but more extreme values; skewness varied, with GraalVM least skewed and SAP HANA/MongoDB resembling several other domains.SAP HANA and MongoDB showed more outliers than other datasets, while GraalVM's kurtosis resembled Nature and Human series.Source: §6.3 · pp. 10–11, Figure 3 · p. 11
- Performance measurements were generally more stationary but less autocorrelated than other-domain series, with GraalVM mostly stationary and non-autocorrelated while MongoDB and SAP HANA had more non-stationary and autocorrelated series.The authors note that forecasting models relying on strong temporal dependencies may struggle with performance data.Source: §6.3 · pp. 11–12, Figure 4 · p. 12
7 Annotating suspected Performance Changes
Because real system alerts were unavailable, the study used expert annotations of performance changes considered worthy of practitioner investigation. It sampled performance series, collected annotations, and assessed agreement and annotation difficulty.
7.1 Annotation Tool
- AnnotateChange presented connected time-series points for users to click as suspected change points, with a tailored tutorial on mean, variance, and outlier distinctions.The study added difficulty ratings, task timing, and the ability to revisit annotations.Source: §7.1 · pp. 12–13
7.2 Pilot Study
- A pilot used ten manually selected series from each of three datasets; feedback led to interface changes, including requiring a difficulty rating before proceeding.Pilot participants reported 493 change points across 30 series and spent about one minute per series on average.Source: §7.2 · p. 13
7.3 Sampling the Performance Time Series
- The researchers sampled 500 series per dataset, stratifying by the number of unique change points produced by CPD methods.This design gave stable series from below the 20th percentile equal representation with higher-change groups.Source: §7.3 · p. 14
7.4 Annotating Performance Changes
- Fourteen performance-engineering annotators were recruited, and each was assigned 720 series, with up to five annotators per series.Participants had four weeks to complete an estimated six to seven hours of annotation work.Source: §7.4 · p. 14
7.5 Evaluating Annotation agreement
- The study compared each annotator against the other four on each series using OvR F1-score, where values near 1 indicate agreement and values near 0 indicate disagreement.Series with fewer than five annotators were excluded from this analysis.Source: §7.5 · p. 15
7.6 RQ2: How consistent are humans in identifying potential performance changes?
- 74% of series had OvR F1-scores above 0.8, with a mean of 0.87 and median of 0.93 across annotators.About 40% of tasks achieved a perfect score of 1.0, indicating broad agreement across annotators.Source: §7.6 · p. 15
- Shared change points fell from 5,815 identified by at least one annotator to 3,286 identified by at least two, indicating subjectivity despite high overall agreement.The authors explain that OvR scores average multiple annotators’ viewpoints; relying on one annotator may not be reliable. OvR agreement was negatively correlated with perceived difficulty (r = −0.65, p-value < 0.001).Source: §7.6 · pp. 16–18
- Annotation took a mean of 101 seconds per task and a median of 11 seconds, adding up to more than three days for MongoDB’s thousands of tests at the mean rate.About 67% of tasks were considered trivial, easy, or doable, while 4.2% were deemed impossible.Source: §7.6 · pp. 17–18
8 Evaluating Change Detection Methods
The study benchmarked twelve CPD methods on annotated software-performance series, comparing default settings with an oracle configuration that selects the best parameters for each series. BINSEG had the best overall accuracy across datasets, while results varied by dataset and tuning substantially affected performance.
8.1 Change Point Detection Methods
- The benchmark included twelve open-source, actively maintained methods, mostly offline methods, plus BOCPD as an online reference point.The authors focused on offline methods to compare algorithms on complete series; MONGO was also included as an adaptation of E-Divisive Means used by MongoDB.Source: §8.1 · pp. 19–20
8.2 Evaluation Metrics
- F1 combines precision and recall as a single effectiveness measure, with change points counted as matches when within a margin of error of human annotations.Precision is the proportion of detected points that are correct; recall is the proportion of true points detected.Source: §8.2 · p. 20
8.3 RQ3. How efficient are CPD methods in identifying potential performance changes?
- BINSEG achieved the best overall F1-score under both default and Oracle settings, with 94% overall precision while capturing 83% of suspected change points.It ranked first or second across evaluations and was the best method with default parameters.Source: Table 5 · p. 21, §8.3 · p. 22
- Default-setting average F1 was 0.84 on GraalVM, versus 0.67 on MongoDB and 0.65 on SAP HANA.The GraalVM series were primarily stationary with low coefficient of variation and often had a single change point; MongoDB and SAP HANA had lower precision.Source: §8.3 · p. 21
- Oracle tuning raised global average F1-score by 28%, from 0.70 to 0.90, and average precision from 72% to 92%.Oracle results use each series’ best tested configuration and assume access to its ground truth, so they represent an upper bound rather than a practical tuning strategy.Source: §8.1 · p. 20, §8.3 · p. 21
- MONGO achieved an overall F1-score of 0.86 by default and 0.91 in Oracle mode, showing strong default performance but less tuning gain than several alternatives.Its original workflow intentionally tolerates false positives for engineer review, whereas this benchmark measured precision and recall symmetrically with F1.Source: §8.3 · p. 22
- BOCPD and SEGNEIGH reached accuracy comparable to BINSEG after tuning, despite low default precision and sensitivity to performance-data noise.BOCPD precision rose from 62% to 97% when tuned; SEGNEIGH reached an overall Oracle F1-score of 0.97.Source: §8.3 · p. 22
- AMOC performed best on GraalVM with and without tuning, but it detects at most one mean change and can miss additional changes.The authors associate its strong GraalVM results with the few change points in those series and suggest multi-change methods for more general cases.Source: §8.3 · p. 22
9 Discussion and Implications
The discussion relates CPD accuracy to properties of performance time series and detection behavior. It draws implications for method choice, parameter tuning, expert review, and interpretation of human annotations.
9.1 Characteristics of performance time series and CPD methods’ accuracy.
- Time-series size was the strongest predictor of CPD F1-score (permutation importance 0.499 ± 0.099), followed by autocorrelation (0.142 ± 0.040).Skewness and kurtosis had smaller importance; trend, coefficient of variation, and stationarity added little predictive information once other characteristics were considered.Source: §9.1 · p. 23, Figure 12 · p. 23
- Many methods reported change points early, often with lower F1, while later detections generally agreed more with human annotations; several methods also over-segmented series with no changes.The authors suggest sufficient post-change observations may be needed to accumulate evidence before triggering a detection.Source: §9.1 · p. 24
9.2 Implications
- The authors caution that CPD methods evaluated on other domains may not transfer reliably to software performance data, whose statistical properties include frequent extreme values and limited autocorrelation.They recommend research attention to robustness to heavy-tailed noise and assumptions about trend or temporal dependence.Source: §9.2 · p. 24
- Because expert annotations show both high aggregate agreement and substantial subjectivity, the discussion cautions against treating them as simple binary ground truth or relying on one reviewer.The authors position CPD as a prioritization aid for ambiguous or high-risk cases, with expert effort reserved for those cases.Source: §9.2 · pp. 24–25
- The authors caution that aggressive outlier removal may filter meaningful regression signals, and recommend care when preprocessing performance data.They also describe periodic rerunning of detection as new measurements arrive, while noting the need to balance timeliness with sufficient evidence.Source: §9.2 · pp. 24–25
- BINSEG is presented as a stable default when tuning resources or historical data are limited, while tuning can substantially improve accuracy.Practitioners may use time-series characteristics to guide method selection; the study used exhaustive grid search but notes more efficient search strategies as alternatives.Source: §9.2 · p. 25
10 Threats to Validity
The study’s conclusions are limited by its expert-judgment ground truth, measurement choices, datasets, and tuning assumptions. It also treats analyses of time-series characteristics and CPD performance as hypothesis-generating rather than causal.
Construct validity
- Expert annotations reflect judgments about potential changes, not developer-confirmed regressions, so high agreement does not establish that labels represent meaningful or user-impacting regressions.The study therefore limits claims about actual regression-detection effectiveness.Source: §10 · pp. 25–26
- CoV, skewness, kurtosis, trend, stationarity, and autocorrelation may not capture performance-specific nuances such as environment variability or CI-specific noise.These measures were chosen because they are widely used in time-series and CPD literature and support cross-domain comparisons.Source: §10 · pp. 25–26
Internal validity
- Manual labeling remains subjective despite experienced annotators, up to five independent reviews per series, and generally high consensus measured with OvR F1-scores.The tool also supported review, required difficulty selection, and provided clearer prompts.Source: §10 · p. 26
- Sampling the 1,500 labeled series according to CPD-detected change-point counts may bias selection toward series more likely to contain detectable changes.Stratification across percentiles of unique change points was used to represent both stable and highly variable series.Source: §10 · p. 26
- Oracle tuning may overfit sampled series, and its results are expensive-optimization upper bounds rather than expected deployment performance.The study separates Default and Oracle configurations when reporting results.Source: §10 · pp. 26–27
External validity
- The three industrial datasets may not generalize to embedded, mobile, or highly variable-workload systems, and curation excludes resource- or event-based metrics such as memory or I/O.The included datasets are GraalVM, MongoDB, and SAP HANA; curation focused on time-based and throughput metrics.Source: §10 · p. 26
- Excluding series with fewer than 50 observations may limit applicability where organizations have sparse historical performance data.The restriction follows recommendations because many CPD algorithms underperform on very short series.Source: §10 · p. 26
Conclusion validity
- The correlation analysis uses a small set of structurally similar datasets and cannot establish general or causal rules about which characteristics determine CPD effectiveness.Dataset-specific environments, measurement frequency, and instrumentation may confound the relationship, so the analysis is used to generate hypotheses.Source: §10 · pp. 26–27
11 Conclusion
Across three industrial datasets, CPD performance varied by dataset and configuration, while BINSEG was the most stable and consistently high-performing method. The study concludes that CPD can approximate expert-identified changes under specified assumptions, and calls for benchmark reassessment and evaluation metrics reflecting operational priorities.
Study design
- The study curated three industrial time-series datasets, established expert annotations as a reference ground truth, and evaluated 12 CPD algorithms under default and exhaustively tuned settings.The datasets were GraalVM, MongoDB, and SAP HANA; evaluation used precision, recall, and F1-score.Source: §11 · p. 27
Results
- Average default F1-scores ranged from 0.65 to 0.84 and tuned averages from 0.70 to 0.90; BINSEG was most stable and consistently high across datasets and configurations.Tuning gains came primarily through improved precision, while other methods showed strong hyperparameter sensitivity.Source: §11 · p. 27
- Performance time series had comparatively low variance and weak trends, but frequent heavy-tailed behavior and limited autocorrelation relative to traditional CPD benchmark domains.Annotators showed mean OvR F1 agreement of approximately 0.87, though stricter consensus thresholds revealed residual subjectivity.Source: §11 · p. 27
Future work
- Future work should reassess benchmark quality and representativeness and consider cost-sensitive metrics because F1-score weights false positives and false negatives equally.Operational priorities may assign different costs to missing a regression and investigating a false alarm.Source: §11 · p. 27