Source-linked AI summary
A Systematic Review of Unsupervised Learning Techniques for Software Defect Prediction
Ning Li, Martin Shepperd, Yuchen Guo
TL;DR
Software defect prediction often lacks the labeled data required by supervised models. This systematic review synthesizes 49 studies and 2456 experimental results using reconstructed confusion matrices and MCC, finding comparable unsupervised and supervised performance but substantial reporting and benchmarking concerns.
Problem
Labeled defect classifications are difficult to collect for training supervised defect prediction models, motivating investigation of unsupervised alternatives.
Method
The review analyzes 49 studies and 2456 experimental results, reconstructing confusion matrices and using MCC as the main performance measure.
Results
Unsupervised models are comparable with supervised models for within-project and cross-project prediction, with Un Fuzzy and Un SC showing the most potential.
Takeaways & Limitations
Unsupervised models may be considered when labelled training data are scarce.
Takeaways & Limitations
33% (823/2456) of experimental results could not be checked because of incomplete reporting, and the relationship between non-reporting and errors may make extrapolation unsafe.
Abstract
from arXiv · showhide
Background: Unsupervised machine learners have been increasingly applied to software defect prediction. It is an approach that may be valuable for software practitioners because it reduces the need for labeled training data. Objective: Investigate the use and performance of unsupervised learning techniques in software defect prediction. Method: We conducted a systematic literature review that identified 49 studies containing 2456 individual experimental results, which satisfied our inclusion criteria published between January 2000 and March 2018. In order to compare prediction performance across these studies in a consistent way, we (re-)computed the confusion matrices and employed the Matthews Correlation Coefficient (MCC) as our main performance measure. Results: Our meta-analysis shows that unsupervised models are comparable with supervised models for both within-project and cross-project prediction. Among the 14 families of unsupervised model, Fuzzy CMeans (FCM) and Fuzzy SOMs (FSOMs) perform best. In addition, where we were able to check, we found that almost 11% (262/2456) of published results (contained in 16 papers) were internally inconsistent and a further 33% (823/2456) provided insufficient details for us to check. Conclusion: Although many factors impact the performance of a classifier, e.g., dataset characteristics, broadly speaking, unsupervised classifiers do not seem to perform worse than the supervised classifiers in our review. However, we note a worrying prevalence of (i) demonstrably erroneous experimental results, (ii) undemanding benchmarks and (iii) incomplete reporting. We therefore encourage researchers to be comprehensive in their reporting.
1. Introduction
Unsupervised defect prediction is attracting attention because labeled defect data are difficult to collect. This review evaluates unsupervised algorithms and compares their predictive performance with supervised models.
- Labeled defect classifications are often difficult to collect for training supervised defect prediction models.
- The review analyzes 49 unsupervised software defect prediction primary studies.
- The review investigates deployed unsupervised algorithms and their predictive performance relative to supervised models.
- The review identifies 14 unsupervised prediction technique families, including six cluster-labelling techniques.
- The review regularizes data from 49 studies, obtaining 2456 individual experimental results for meta-analysis.
- A bibliometric analysis examines publication trends and the quality of reported experimental results.
2. Review Methodology
The review follows a systematic literature-review process to investigate publication trends, reporting quality, experimental practices, predictive comparisons, model families, and dataset effects. Its questions address both research concerns and practitioner-relevant prediction performance.
- The review clarifies research questions, identifies relevant primary studies, synthesizes their results, and answers the questions through meta-analysis.
- RQ1–RQ3 address publication trends, experimental reporting quality, and the kinds of unsupervised learning experiments conducted.
- RQ4 examines the predictive-performance difference between unsupervised and supervised defect prediction.
- RQ5 asks which unsupervised prediction models or model families perform better.
- RQ6 examines how dataset characteristics affect predictive performance.
- The first three questions primarily interest researchers, while the remaining three interest both practitioners and researchers.
2.2. Search Process
The search process covered five academic search engines and targeted software defect prediction studies using unsupervised, unlabelled, or clustering-related terminology. Selection reduced 1360 initially identified papers to 49 primary papers.
- The search covered papers published between January 2000 and March 2018 using five academic search engines.
- The key search string combined software fault, defect, bug, or error prediction terms with unsupervised, unlabelled, or clustering terms.
- 49 papers were selected from an initial 1360 papers identified by the search and selection process.
2.3. Inclusion Criteria
The inclusion criteria restricted the review to accessible English-language studies using real data and new unsupervised defect-prone-module prediction experiments published between January 2000 and March 2018.
- Included studies had to be written in English, have full content available, and be published between January 2000 and March 2018.
- Studies had to use real data rather than simulations and apply at least one unsupervised method to software defect-prone module prediction.
- Included papers had to contain new software defect prediction experiments rather than reanalyses of previously published experiments.
- The review included the best prediction-performance results per dataset and learner when studies reported experiments with another primary focus.
2.4. Data Extraction and Synthesis
The review synthesizes experimental data from 49 studies by standardizing names, reconstructing confusion matrices where possible, and comparing results across 2456 experiments. It uses MCC alongside extracted performance indicators to summarize prediction quality, while documenting substantial reporting gaps.
- Data extraction: 2456 experimental results span 128 software-project defect datasets and 25 prediction-model families, including 14 unsupervised and 11 supervised learner families.Each paper contributed between 1 and 751 results, with a median of 12.
- Data extraction: The review extracts study metadata, dataset and prediction settings, learning type, validation use, fault rate, and reported prediction results for each experiment.The extracted attributes include project pairing, prediction type, input-metric count, dataset family, clustering method, and inconsistency status.
- Data synthesis: Confusion matrices are reconstructed or recomputed where possible because they provide the fundamental descriptor for deriving most performance indicators.The authors also standardize names and perform preprocessing and data-quality checks using R scripts.
- Prediction performance measures: MCC is emphasized because it incorporates all four confusion-matrix quadrants and ranges from -1 to 1, where 1 denotes perfect classification and 0 denotes no correlation.The review notes that F1 and AUC can be problematic for defect prediction because they do not fully capture negative-class performance or deployment thresholds.
- Data summary: Within-project classification dominates the dataset, and more than half of the results involve supervised learners used generally as comparators to unsupervised methods.The categorical summary covers individual experimental results rather than only study-level counts.
- Data summary: After removing 262 inconsistent results, the analysis summarizes 2194 internally consistent results, although 823 results remain unchecked because reporting was incomplete.The valid results include the unchecked cases, and the reported fault rate spans 0.4% to over 93%, while AUC ranges from 0.31 to 0.948.
2.5. Quality Factors
The review assesses quality through publication characteristics, validation practices, and internal consistency checks. It recomputes confusion matrices and removes results that violate predefined plausibility or consistency rules.
- Quality factors: Quality assessment separates external factors, such as publication venue and type, from internal factors involving experimental design and result consistency.Cross-validation use is examined as an indicator of experimental quality.
- External factors: The 49 selected papers span venues with widely varying experimental-result counts, from 1 to 751, which may give some papers greater propensity for error.The review also classifies publishers using predatory-publisher lists, while acknowledging that those lists have been questioned.
- Internal experiment quality: The authors recompute or construct confusion matrices from reported measures and use them to calculate MCC when the available information permits.They define d as defective-module proportion and normalize TP, TN, FP, and FN so their sum equals 1 when preprocessing raw data.
- Consistency checks: 262 of 2456 experimental results are removed as problematic data: 171 under Rule 1, 7 under Rule 2, 60 under Rule 3, 3 under Rule 4, 19 under Rule 5, and 2 under Rule 6.The removed results are excluded from subsequent data analysis.
3. Bibliometric analysis
The bibliometric analysis finds substantial growth in unsupervised software defect prediction publications, alongside widespread reporting problems and concerns about comparator benchmarks. It also characterizes the diverse unsupervised techniques and their clustering-based workflows.
- Publication trends: 49 studies were published between 2000 and March 2018, with conference papers dominating and pronounced growth in recent years.2018 is incomplete.
- Experimental results: 2456 experimental results were reported across the 49 studies, ranging from 1 to 751 per paper, with a median of 12.The review separately summarizes 2194 non-erroneous results.
- Publication venues: 18 of 49 papers, approximately 37%, came from venues classified as predatory publishers.The proportion possibly declined after 2014.
- Reporting quality: 823 of 2456 experimental results, over 30%, could not be checked because of incomplete reporting, while 262 checked results were problematic.Consistency checking covered approximately 66% of all results, or 1633 of 2456.
- Errors and venues: The likelihood of an error was 1.21 with 95% confidence limits [0.31, 4.73] at the paper level, whereas the individual-result odds ratio was 0.02 with 95% confidence limits [0.02, 0.04].The individual-result estimate was skewed by a small number of papers containing many results and inconsistencies.
- Comparator benchmarks: Among 28 papers using supervised learners as UnSDP comparators, 71% cited related articles, but only four used previous prediction results.Eight of 28 papers used supervised methods without citation.
- Unsupervised techniques: Clustering-based approaches dominate and generally cluster instances before labeling each group defective or non-defective.The review categorizes techniques into 14 families and 21 sub-families; generic thresholds are the most popular labeling approach.
4. Analysis of Unsupervised Defect Prediction Performance
The review finds that unsupervised defect-prediction models are broadly comparable with supervised models across within-project and cross-project settings, although performance varies by model family and reporting quality limits comparisons.
- Comparison of UnSDP and SDP: Vote-counting across 26 studies and 110 projects found UnSDP comparable with SDP for both within-project and cross-project prediction.The 95% confidence intervals straddled zero across the reported comparisons.
- MCC comparison: Un Fuzzy had the highest median MCC among the compared within-project model families, but its wide confidence interval overlapped other models.Bayes appeared strongest among supervised approaches in this comparison.
- MCC comparison: KPart had a median MCC of 0.33 versus 0.41 for all SDP models, cautioning against using KMeans as an undemanding benchmark.Traditional partition-based clustering includes KMeans and its variants.
- Reporting constraints: Incomplete reporting left 823 experimental results unavailable for MCC comparison, while one spectrum-clustering study reported only AUC and could not enter the MCC meta-analysis.More complete confusion-matrix reporting would enable integration into meta-analyses.
- Best unsupervised families: FCM and FSOM learners performed best among the evaluated unsupervised model families.EXP was competitive but was not recommended because it requires manual expert classification of software modules.
- JiT defect prediction: MR-CCUM performed best among the reported unsupervised JiT prediction methods for both within-project and cross-project prediction.The comparison used 197 high-quality Popt values, including 97 cross-project experiments.
- Dataset characteristics: Dataset characteristics appeared to affect predictive performance, but fault rate was a weak explanatory factor for maximum MCC (F = 0.66, p-value = 0.6261).The review concludes that the important characteristics remain unclear.
5. Threats to Validity
The review’s validity is constrained by unchecked or inconsistent results, limited metric coverage, uncertain dataset representativeness, and potentially weak supervised baselines.
- Internal validity: 823 of 2456 results could not be checked because incomplete reporting prevented confusion-matrix reconstruction, while 262 checked results were problematic.The authors warn that non-reporting and errors may interact, making simple extrapolation unsafe.
- Metric coverage: The fine-grained comparison used MCC, with an additional analysis based on 458 high-quality AUC results because metric coverage was limited.The authors note that fewer experiments were available for the additional analysis.
- Error detection: Only consistency errors could be checked, so the actual experimental-analysis error rate may exceed the detected rate.The authors report an overall knowable error rate of 11%.
- External validity: The selected datasets span many sources, but their representativeness of all software defect-prediction scenarios cannot be established.This limits confidence in generalizing the review’s findings.
- Comparator quality: Many supervised comparators were basic benchmarks with unclear state-of-the-art status or hyper-parameter tuning; only 3 of 28 papers explicitly reported tuning them.This creates a potential source of bias in UnSDP–SDP comparisons.
- Researcher bias: Researcher bias was identified as an additional threat, motivating public release of the review materials and research processes.The authors connect openness with reducing tendencies to confirm prior beliefs.
6. Summary and actionable findings
The review finds that unsupervised defect-prediction models can perform comparably to supervised models, while reporting quality remains a serious concern. Dataset characteristics influence performance, and no single learner is consistently optimal.
- Scope and evidence: 49 studies yielded 2456 independent experimental results for the review’s analyses.The results covered 128 software projects and 25 prediction model families.
- Reporting quality: 11% of assessable experimental results were demonstrably inconsistent, while many studies incompletely reported validation and data-quality information.Twenty-five of 49 studies did not explicitly report whether cross-validation was used, and 14 could not be checked for problematic data because of incomplete reporting.
- Predictive performance: Unsupervised models were comparable with supervised models for both within-project and cross-project prediction.The review used recomputed confusion matrices and Matthews Correlation Coefficient as its main performance measure for consistent comparison.
- Actionable findings: UN MR was considered a practical choice because it had comparable prediction performance to GA while requiring less time.For effort-aware just-in-time defect prediction, GA performed better than UN MR; the comparison was less clear when effort was ignored.
- Actionable findings: Pareto-based node distributions provide a simple and effective way to label clusters as defective or not in clustering-based approaches.The Pareto principle is sometimes referred to as the 80:20 rule.
- Future directions: Dataset characteristics clearly affect predictive performance, and interactions between learner and dataset warrant further investigation.The review considers it unlikely that one supervised or unsupervised algorithm will always be optimal.
Primary Studies
The literature search was completed on 7 March 2018.
- Primary Studies: 7 March 2018 marked completion of the literature search.
Appendix A. Supervised defect prediction procedure
The review directs readers to an appendix table for the supervised defect-prediction models used in the reviewed studies.
- Appendix A. Supervised defect prediction procedure: Table A.17 presents the supervised defect-prediction models used in the reviewed studies for performance comparison.