Source-linked AI summary
Is "Better Data" Better than "Better Data Miners"? (On the Benefits of Tuning SMOTE for Defect Prediction)
Amritanshu Agrawal, Tim Menzies
TL;DR
Prior classifier-ranking studies did not evaluate multiple criteria or account for variation in training data. This paper uses SMOTUNED, a self-tuning SMOTE preprocessing method, and reports that better training data can matter more than classifier choice for defect prediction.
Problem
Prior software-analytics ranking studies overlooked multi-criteria evaluation and the effects of data variation, while imbalanced datasets commonly used default SMOTE parameters.
Method
The paper applies SMOTUNED, which uses search-based tuning to select SMOTE parameters for each dataset before classification and evaluates multiple performance criteria.
Results
SMOTUNED produced dramatically large defect-prediction improvements, including up to 60% higher AUC against SMOTE, across classifiers in the reported study.
Takeaways & Limitations
For defect prediction, preprocessing may be more important than classifier choice, and classifier-ranking studies are incomplete without preprocessing.
Takeaways & Limitations
The experiments support conclusions about defect prediction, while whether the results hold for proprietary projects or other open-source sources remains open.
Abstract
from arXiv · showhide
We report and fix an important systematic error in prior studies that ranked classifiers for software analytics. Those studies did not (a) assess classifiers on multiple criteria and they did not (b) study how variations in the data affect the results. Hence, this paper applies (a) multi-criteria tests while (b) fixing the weaker regions of the training data (using SMOTUNED, which is a self-tuning version of SMOTE). This approach leads to dramatically large increases in software defect predictions. When applied in a 5*5 cross-validation study for 3,681 JAVA classes (containing over a million lines of code) from open source systems, SMOTUNED increased AUC and recall by 60% and 20% respectively. These improvements are independent of the classifier used to predict for quality. Same kind of pattern (improvement) was observed when a comparative analysis of SMOTE and SMOTUNED was done against the most recent class imbalance technique. In conclusion, for software analytic tasks like defect prediction, (1) data pre-processing can be more important than classifier choice, (2) ranking studies are incomplete without such pre-processing, and (3) SMOTUNED is a promising candidate for pre-processing.
1 INTRODUCTION
The paper argues that classifier-ranking studies overlook data preprocessing and presents SMOTUNED, which tunes SMOTE separately for each dataset to improve defect prediction.
- Motivation: Software-quality budgets leave blind spots where defects may be missed, motivating automated defect prediction.Quality assurance resources are concentrated in safety- or mission-critical areas, while static code predictors can sample less scrutinized regions.
- Research gap: Prior ranking studies ask which classifiers produce the best predictors, but the paper asks whether preprocessing changes those rankings.The paper focuses on imbalanced software-engineering datasets, where SMOTE parameters are often set by expertise or left at defaults.
- Approach: SMOTUNED automatically sets SMOTE parameters for each dataset, and its learned settings differ substantially from default SMOTE.The method uses search-based software engineering to tune candidate SMOTE parameter settings before classification.
- Results: AUC improvements reach 60% against SMOTE, while SMOTUNED also outperforms a recent class-imbalance technique.The paper reports these improvements on defect data from 3,681 Java classes containing over a million lines of code.
- Results: No learner is consistently best across datasets and criteria, while SMOTUNED appears most often among the best results.The reported conclusion is that better training data may matter more than the subsequent classifier choice.
- Practical implications: SMOTUNED terminates in under two minutes, supporting its recommended widespread use.The paper also presents a methodology for assessing preprocessing and a reproduction package for its results.
2 BACKGROUND AND MOTIVATION
Software defect prediction uses classifiers on code and process data, but classifier rankings are complicated by multiple evaluation criteria and imbalanced data. The paper motivates SMOTUNED as a differential-evolution-based way to tune SMOTE for each dataset while evaluating classifiers more broadly.
- Defect Prediction: Defect prediction applies binary classification to software components using code and process metrics to identify likely defects.
- Performance Criteria: Recall, precision, false alarm, and AUC evaluate different aspects of defect-prediction performance.Recall and precision are defined from the confusion matrix; false alarm is FP/(FP + TN), and AUC is the area under the ROC curve.
- Performance Criteria: Increasing recall generally raises false alarms because more sensitive detectors trigger more often.
- Class Imbalance: Class imbalance occurs when the minority class is under-represented relative to the majority class, a common condition in software-engineering data.A 5:95 minority-to-majority ratio is described as worse than 20:80.
- Ranking Studies: Prior ranking studies commonly compared classifiers, but most did not meet the paper’s definition of a good study using multiple learners, datasets, criteria, and imbalance handling.The survey covered 22 highly cited studies, and the overwhelming majority failed this definition.
- SMOTUNED: SMOTUNED auto-tunes SMOTE parameters for each dataset using differential evolution to search parameter settings for class rebalancing.Differential evolution compares candidate parameter settings while SMOTE generates synthetic minority examples.
3 EXPERIMENTAL DESIGN
The study compares No-SMOTE, SMOTE, and SMOTUNED across imbalanced open-source Java defect datasets using repeated cross-validation, multiple performance criteria, and statistical tests. It tunes SMOTE while leaving classifier parameters unchanged, then checks whether gains on one measure damage others.
- Experimental setup: The experiment compares MAHAKIL, SMOTE, and SMOTUNED on imbalanced open-source Java systems using 5*5 cross-validation.The evaluation uses datasets described with CK metrics and reports median performance over 25 repeats.
- Assessment design: The study evaluates AUC, precision, recall, and false alarm, using within-measure and cross-measure assessment rigs.The cross-measure rig checks whether optimizing AUC changes precision, recall, or false alarm.
- Experimental setup: Each treatment pre-filters only the training data before a classifier learns and is evaluated on unchanged test data.SMOTUNED further splits training data for differential-evolution validation and parameter search.
- Experimental setup: SMOTUNED tunes preprocessing parameters while classifier control parameters remain untuned.This isolates the contribution of tuning SMOTE rather than jointly optimizing data preprocessing and learners.
- Statistical analysis: Scott-Knott combines bootstrap sampling with the A12 effect size test, dividing results only at statistically significant, non-small differences.The procedure uses 99% confidence and A12 ≥0.6 as the effect-size threshold.
4 RESULTS
Across the evaluated defect datasets, tuning SMOTE produced large AUC and recall gains, while precision changed little and false-alarm increases were generally smaller. SMOTUNED frequently appeared in the best results across learners and outperformed the compared class-imbalance treatments, though broader claims remain limited.
- Parameter tuning: No: learned SMOTUNED parameters differed substantially from SMOTE defaults and varied dramatically across datasets.Median k was never below 11, r was usually 3 rather than 2, and IQRs were large.
- Within-measure results: About 60% AUC and 20% recall improvements were observed with SMOTUNED, with little precision damage and minimal false-alarm change.The AUC improvement remained similar across datasets with 34% and 2% imbalance.
- Treatment comparisons: SMOTUNED was always used by the best combination for AUC and recall across the reported learner comparisons.Precision differences were usually pragmatically negligible, while recall gains generally cost smaller false-alarm increases.
- Learner rankings: Random Forest was best in only 11/36 results, whereas SMOTUNED consistently accompanied whichever learner was best for recall and AUC.This supports examining preprocessing alongside classifier selection in defect-prediction rankings.
- Comparison with MAHAKIL: SMOTUNED won 8/9 AUC and 8/9 recall comparisons against MAHAKIL, while MAHAKIL won 9/9 false-alarm comparisons.SMOTUNED produced slightly higher false alarms but larger AUC and recall results.
- Scope and limitations: The paper does not report definitive conclusions for jointly tuning data preprocessors and data miners, and broader software-engineering applications require further study.The authors identify the joint search space as very large and limit current conclusions to defect prediction.
5 THREATS TO VALIDITY
The study identifies several threats to validity, especially sampling bias from predominantly Apache open-source data and evaluation bias in how SMOTUNED is assessed. It also compares within- and cross-measure assessment and addresses order bias through repeated randomization.
- General validity: The paper acknowledges that empirical-study biases can affect its conclusions.This general validity warning frames the specific sampling, evaluation, and order-bias concerns.
- Order bias: 25 random reorderings of the data samples were used to mitigate order bias in training and testing assignments.Random allocation could otherwise place unusually favorable or unfavorable samples in the two sets.
- Sampling bias: Sampling bias limits whether the results generalize to proprietary projects or open-source projects from other sources.The nine datasets came from the SEACRAFT repository and were mostly Apache projects.
- Evaluation bias: Cross-measure assessment is recommended only when CPU is critically restricted because within-measure assessment can produce larger improvements.The cross-measure results were similar, but some improvements were larger under within-measure assessment.
6 CONCLUSION
The conclusion argues that improving training data with tuned preprocessing can matter more than selecting a better classifier for defect prediction. It recommends reevaluating ranking studies without preprocessing analysis and tuning SMOTE parameters for each dataset.
- Conclusion: SMOTUNED was often associated with the best AUC and recall results, while no learner was consistently best across datasets and criteria.The conclusion reports little evidence against SMOTUNED for precision and false alarm.
- Conclusion: For defect prediction, better training data may be more important than the subsequent choice of classifier.The paper summarizes this as “better data” being better than “better data miners.”
- Recommendations: Prior ranking studies that omitted data-preprocessing effects should be analyzed again.Future ranking studies should include a SMOTE-like preprocessing step.
- Recommendations: SMOTE should be automatically tuned for each new dataset rather than used with default parameters.The paper recommends tuning with the final evaluation criteria, or with AUC when CPU is insufficient.