PaperlayerView exact revision on arXiv

Source-linked AI summary

The Impact of Class Rebalancing Techniques on the Performance and Interpretation of Defect Prediction Models

Chakkrit Tantithamthavorn, Ahmed E. Hassan, Kenichi Matsumoto

arXiv:1801.10269v1cs.SE

TL;DR

Class imbalance can make defect prediction models inaccurate, while prior studies provide contradictory evidence about when rebalancing helps. This paper evaluates four techniques across ten measures and model interpretation using 101 datasets, finding benefits for Recall but risks for interpretation and no effect on AUC.

  • Problem

    Imbalanced datasets can cause inaccurate defect prediction models, while differing datasets, classifiers, and performance measures have produced contradictory rebalancing findings.

  • Method

    The study evaluates four class rebalancing techniques across ten performance measures and model interpretation, using 101 datasets and statistical models of experimental factors.

  • Results

    Class rebalancing improves Recall, decreases Precision, does not impact AUC, and shifts learned concepts across classifiers.

  • Takeaways & Limitations

    Rebalancing is beneficial for increasing defect-identification completeness, should be avoided when deriving knowledge from models, and AUC should standardize model comparisons.

  • Takeaways & Limitations

    Experimental design settings may impact the study's results.

Abstract

from arXiv · show

Defect prediction models that are trained on class imbalanced datasets (i.e., the proportion of defective and clean modules is not equally represented) are highly susceptible to produce inaccurate prediction models. Prior research compares the impact of class rebalancing techniques on the performance of defect prediction models. Prior research efforts arrive at contradictory conclusions due to the use of different choice of datasets, classification techniques, and performance measures. Such contradictory conclusions make it hard to derive practical guidelines for whether class rebalancing techniques should be applied in the context of defect prediction models. In this paper, we investigate the impact of 4 popularly-used class rebalancing techniques on 10 commonly-used performance measures and the interpretation of defect prediction models. We also construct statistical models to better understand in which experimental design settings that class rebalancing techniques are beneficial for defect prediction models. Through a case study of 101 datasets that span across proprietary and open-source systems, we recommend that class rebalancing techniques are necessary when quality assurance teams wish to increase the completeness of identifying software defects (i.e., Recall). However, class rebalancing techniques should be avoided when interpreting defect prediction models. We also find that class rebalancing techniques do not impact the AUC measure. Hence, AUC should be used as a standard measure when comparing defect prediction models.

1 INTRODUCTION

Defect prediction models support both defect-prone module prioritization and interpretation of factors associated with defects, but class imbalance can undermine both uses. This study examines four rebalancing techniques across performance, interpretation, and experimental settings to derive practical guidance.

  • Defect prediction models help quality assurance teams prioritize likely-defective modules and understand software metrics associated with defect-proneness.
  • Imbalanced training datasets can produce inaccurate models because traditional classifiers often fail to identify defective modules, the minority class.
  • Prior studies report contradictory rebalancing effects because they use different datasets, classification techniques, and performance measures.
  • The study evaluates four rebalancing techniques with seven classifiers across 101 datasets and constructs statistical models linking experimental factors to performance and interpretation.
  • Rebalancing improves Recall most, decreases Precision most, and does not impact AUC, while its largest performance gains occur for under-sampling with logistic regression on highly imbalanced, high-EPV datasets.
  • Rebalancing shifts learned concepts, with only 23%-34%, 55%-62%, and 68%-71% of top variables retained for neural network, logistic regression, and random forest models, respectively.

2 THE NATURE OF IMBALANCED DEFECT DATASETS

The study investigates how common class imbalance is in defect prediction datasets. Across 101 datasets, most have low defective ratios and only a small fraction are approximately balanced.

  • Class imbalance means defective and clean modules are not represented equally in a dataset.
  • The analysis covers 101 publicly available defect datasets collected from five sources, including Tera-PROMISE, NASA, and other research corpora.
  • 64% of defect datasets have a defective ratio below 30%.
  • 38% of defect datasets have a defective ratio between 10%-20%.
  • As little as 8% of defect datasets have a defective ratio between 45%-55%, indicating that nearly balanced datasets are uncommon.

3 CLASS REBALANCING TECHNIQUES FOR DEFECT PREDICTION MODELS

The study selects four commonly used techniques to rebalance defective and clean modules before model construction. They differ in whether they replicate, remove, or synthetically generate training examples.

  • The four studied techniques are over-sampling, under-sampling, SMOTE, and ROSE.
  • Over-sampling replicates minority modules until they match the majority class, avoiding information loss but potentially introducing redundant modules and overfitting.
  • Under-sampling reduces majority-class modules until their number matches the minority class, shrinking training data but potentially losing important information.
  • SMOTE creates synthetic minority examples from feature-space similarities and combines them with undersampling of majority modules.
  • ROSE combines bootstrap-based undersampling and oversampling with synthetic neighborhood generation to produce an approximately 50% defective ratio.

4 RELATED WORK & RESEARCH QUESTIONS

Prior defect-prediction studies disagree about whether class rebalancing improves performance, and the literature had not examined its effect on model interpretation. This study frames those unresolved issues as research questions.

  • Earlier studies examined rebalancing using different datasets and classification techniques, including proprietary, open-source, and NASA systems.
  • Reported findings conflict: one study found F-measure improvements of 7.8%-22.4%, while others reported differing effects on Accuracy and AUC.
  • A meta-analysis of 42 defect prediction studies found class imbalance was not harmful when the minority class exceeded 20%.
  • These inconsistent findings make practical guidance about applying rebalancing difficult.
  • Prior work suggested concept drift from rebalancing, but no research had investigated its impact on defect-model interpretation.
  • RQ3 asks how class rebalancing techniques impact the interpretation of defect prediction models.

5 CASE STUDY DESIGN

The case study uses 101 publicly available defect datasets selected across different corpora and domains, with out-of-sample bootstrap validation to estimate performance robustly.

  • Case study design: The case study design is applied to each studied dataset.Figure 3 provides an overview of the design and its constituent steps.
  • Dataset selection: 101 publicly available defect datasets were selected to span different corpora and domains.The selection also supports replication by using datasets hosted in public repositories.
  • Validation: Out-of-sample bootstrap validation trains models on bootstrap samples and tests them on rows omitted from those samples.The process is repeated 100 times, and average out-of-sample performance is reported.

5.3 Apply Class Rebalancing Techniques

The study applies four class rebalancing techniques to training data, evaluates seven selected classifiers, and preprocesses variables to reduce correlation and redundancy before modeling.

  • Class rebalancing: Four techniques—oversampling, undersampling, SMOTE, and ROSE—are applied only to training datasets.Testing data remains untreated, allowing performance measurement on the original class distribution.
  • Classification techniques: The study selects seven commonly used classification techniques, including random forest, logistic regression, naive bayes, neural network, and boosting methods.The classifiers are chosen as a manageable set of top-ranked techniques for defect prediction.
  • Variable preprocessing: Correlation analysis reduces collinearity because correlated metrics can distort the interpretation of defect prediction models.Prior work reported that 10%-67% of metrics in publicly available defect datasets are redundant.
  • Variable preprocessing: Redundancy analysis removes variables that lack a unique signal before constructing defect prediction models.Variables are iteratively dropped when preliminary models explain them above the default R2 threshold of 0.9.
  • Parameter settings: Caret parameter optimization is applied before constructing models because the studied classifiers have configurable settings.

5.5 Calculate Performance

Model performance is measured on untreated testing data using threshold-independent and threshold-dependent metrics, including Brier score, calibration slope, and AUC.

  • Performance evaluation: Models trained on training data are evaluated on the untreated testing corpus using threshold-independent and threshold-dependent measures.
  • Threshold-independent measures: The Brier score measures the distance between predicted probabilities and observed outcomes.It ranges from 0 for best performance to 1 for worst performance, with 0.25 representing random guessing.
  • Threshold-independent measures: Calibration slope measures the direction and spread of predicted probabilities through a logistic regression model.A calibration slope of 1 indicates that predicted probabilities are consistent with module labels.
  • Threshold-independent measures: AUC measures discriminatory power by assessing whether defective modules tend to receive higher predicted probabilities than clean modules.AUC is threshold-independent and is computed from the area under the receiver operator characteristic curve.
  • Threshold-dependent measures: Nine threshold-dependent measures are computed after converting probabilities to binary predictions with a default threshold of 0.5.Modules above 0.5 are classified as defective; otherwise, they are classified as clean.

5.6 Rank the Importance of Variables

Variable importance is estimated by permutation-based changes in misclassification rates, then statistically grouped and ranked for baseline and rebalanced models.

  • Variable importance: A generic variable importance score is calculated for every classifier to rank variables in baseline and rebalanced models.Figure 4 summarizes the calculation used to generate these rankings.
  • Variable importance: Each variable’s values are randomly permuted in the testing data, and the resulting change in misclassification rate determines its importance.A larger difference between original and permuted datasets indicates greater variable importance.
  • Statistical analysis: The improved Scott-Knott ESD test clusters variables using statistically significant differences in their mean importance scores.The test uses α = 0.05 and accounts for effect-size magnitude during clustering.
  • Statistical analysis: Scott-Knott ESD v2.0 merges statistically distinct groups when their Cohen’s d difference is negligible across treatments.This addresses overlapping groups that can arise with other post-hoc tests.

5.7 Statistical Analysis of the Experimental Settings

The study uses regression models and ANOVA-based sensitivity analysis to relate experimental settings to defect-prediction performance and interpretation. Model stability is assessed with optimism-corrected adjusted R2 estimates.

  • Statistical modeling: Regression models quantify how experimental settings relate to performance differences in defect prediction models.The models use Ordinary Least Squares (OLS) regression.
  • Model stability: Adjusted R2 evaluates model fit while penalizing additional degrees of freedom.Because adjusted R2 is optimistic on training data, the study estimates and removes average bootstrap optimism.
  • Model stability: 1,000 bootstrap repetitions estimate average optimism for correcting the adjusted R2.Each bootstrap sample is compared with the original dataset and the bootstrap sample.
  • Sensitivity analysis: ANOVA with Wald χ2 chunk tests estimates each experimental setting’s relative explanatory contribution.Larger Wald χ2 values indicate larger impacts on the response, with bootstrap 95% confidence intervals also reported.
  • Sensitivity analysis: Figure 5 summarizes the statistical-analysis workflow for studying experimental settings.The workflow covers regression construction, model assessment, and explanatory-variable power estimation.

6 CASE STUDY RESULTS

Class rebalancing produces strongly measure- and setting-dependent performance changes. It most improves Recall, leaves AUC and MCC relatively insensitive, and can substantially reduce Precision and Accuracy.

  • Measures with limited impact: AUC is largely insensitive to class rebalancing, with observed differences from -4 to 7 percentage points for over-sampling, under-sampling, and SMOTE.For C5.0, SMOTE produces a maximum AUC improvement of up to 5 percentage points.
  • Measures with limited impact: MCC is also insensitive to class rebalancing because it incorporates true and false positives and negatives.The AUC and MCC performance-difference distributions are centered near zero.
  • Probability-based measures: Brier and Slope are sensitive because they rely heavily on predicted probabilities.Unlike AUC, these measures depend on probability values and their relationship to outcomes or score direction and spread.
  • Performance measures: Up to 69 percentage points of Recall improvement, 60 percentage points of G-measure improvement, and 27 percentage points of F-measure improvement are reported.The largest Recall increase occurs with over-sampling on the mylyn dataset using logistic regression.
  • Performance measures: Up to 57 percentage points of Precision loss and 73 percentage points of Accuracy loss occur after class rebalancing.The Precision decline is associated with more false-positive modules.
  • Experimental settings: The largest Recall benefits occur below a 20% defective ratio, especially below 10%, and with EPV above 40.Defective ratio is identified as the most influential experimental factor affecting performance.
  • Experimental settings: Logistic regression is the most sensitive classifier to imbalanced defect datasets among the studied techniques.The impact of rebalancing varies across classification techniques, with logistic regression tending to yield the largest benefits.
  • Recall: Recall benefits most from under-sampling, averaging 18 percentage points versus 10 for SMOTE, 7 for over-sampling, and 3 for ROSE.The study identifies under-sampling as the strongest technique when correctly identifying defective modules is the main objective.

7 PRACTICAL GUIDELINES

The paper recommends using class rebalancing to improve defect identification, but avoiding it when interpreting models. It also recommends AUC as a stable comparison measure.

  • Performance guidance: Class rebalancing is beneficial when quality-assurance teams seek greater completeness in identifying defects, measured by Recall.The largest Recall improvement is associated with under-sampling logistic-regression models on highly imbalanced datasets with EPV above 40.
  • Interpretation guidance: Figure 11 compares classification techniques through the percentage of top variables retaining the same importance rank after rebalancing.Neural networks are shown as most sensitive and naive Bayes as least sensitive to rebalancing.
  • Interpretation guidance: Class rebalancing should be avoided when deriving knowledge from defect-prediction models for quality-improvement planning.The recommendation follows evidence that logistic regression, random forest, and neural networks are sensitive in their interpretations.
  • Evaluation guidance: AUC should be the standard measure for comparing defect-prediction models because it is insensitive to rebalancing and imbalanced datasets.The paper contrasts AUC with threshold-dependent measures that are primarily affected through changes in predicted probability scores.

8 THREATS TO VALIDITY

The study identifies threats involving experimental design, system coverage, prediction scenarios, performance measures, classification alternatives, and SMOTE parameter settings. These constraints limit how broadly its conclusions can be generalized.

  • Experimental design settings may affect the study’s results.
  • 8.1 External Validity: Results may not generalize to all software systems because only a limited number of proprietary and open-source systems were studied.
  • 8.1 External Validity: The conclusions rely on within-project defect prediction, so practical guidelines may differ for cross-project, just-in-time, and heterogeneous scenarios.
  • 8.1 External Validity: The results may not generalize to measures such as AUCEC because developer effort and other performance measures were not evaluated.
  • Random forest without rebalancing tends to be the top-performing classifier for AUC, but other rebalancing and advanced classification techniques remain for future exploration.
  • The study’s conclusions may depend on the SMOTE default setting k = 5, while different parameter settings may produce different results.

9 CONCLUSIONS & FUTURE WORK

The study evaluates four rebalancing techniques across seven classifiers, ten performance measures, and 101 defect datasets. It finds that rebalancing most benefits Recall, leaves AUC largely unchanged, and can substantially alter model interpretation.

  • The study investigates four rebalancing techniques using seven classifiers, ten measures, and statistical models across 101 datasets.
  • Class rebalancing does not impact AUC, improves Recall most positively, and impacts Precision most negatively.
  • The largest performance improvement occurs with under-sampling, logistic regression, and highly imbalanced, low-dimensionality defect datasets.
  • Only 23%-34%, 55%-62%, and 68%-71% of top-ranked variables overlap between rebalanced and baseline models for neural network, logistic regression, and random forest, respectively.
  • The interpretation effects of rebalancing depend heavily on the classification technique.
  • The recommended guidelines are to use rebalancing to increase Recall, avoid it when deriving knowledge from models, and use AUC for comparisons.
Loading 1801.10269v1…