PaperlayerView exact revision on arXiv

Source-linked AI summary

The Impact of Automated Parameter Optimization on Defect Prediction Models

Chakkrit Tantithamthavorn, Shane McIntosh, Ahmed E. Hassan, Kenichi Matsumoto

arXiv:1801.10270v1cs.SE

TL;DR

Defect prediction models often rely on configurable parameters whose default settings may be suboptimal, while exhaustive parameter exploration is impractical. This paper evaluates automated optimization across 18 datasets and multiple classifier techniques, finding large benefits for some techniques but little impact for others. The results support exploring parameter settings selectively rather than treating optimization as uniformly necessary.

  • Problem

    Default settings may underperform, but the impact of automated parameter optimization across defect prediction models is insufficiently known and exhaustive parameter exploration is impractical.

  • Method

    The paper studies automated parameter optimization across 18 datasets, evaluating performance, stability, interpretation, transferability, computational cost, and classifier rankings.

  • Results

    Optimization improves AUC by up to 40 percentage points, while optimized classifiers are at least as stable as default-settings classifiers and effects vary substantially by technique.

  • Takeaways & Limitations

    Parameter optimization should be considered for sensitive techniques such as neural networks, whereas it is not essential for relatively insensitive techniques such as random forest; optimized C5.0 can outperform random forest.

  • Takeaways & Limitations

    The authors do not claim that their results generalize beyond the studied setting.

Abstract

from arXiv · show

Defect prediction models---classifiers that identify defect-prone software modules---have configurable parameters that control their characteristics (e.g., the number of trees in a random forest). Recent studies show that these classifiers underperform when default settings are used. In this paper, we study the impact of automated parameter optimization on defect prediction models. Through a case study of 18 datasets, we find that automated parameter optimization: (1) improves AUC performance by up to 40 percentage points; (2) yields classifiers that are at least as stable as those trained using default settings; (3) substantially shifts the importance ranking of variables, with as few as 28% of the top-ranked variables in optimized classifiers also being top-ranked in non-optimized classifiers; (4) yields optimized settings for 17 of the 20 most sensitive parameters that transfer among datasets without a statistically significant drop in performance; and (5) adds less than 30 minutes of additional computation to 12 of the 26 studied classification techniques. While widely-used classification techniques like random forest and support vector machines are not optimization-sensitive, traditionally overlooked techniques like C5.0 and neural networks can actually outperform widely-used techniques after optimization is applied. This highlights the importance of exploring the parameter space when using parameter-sensitive classification techniques.

1 INTRODUCTION

The paper examines how automated parameter optimization affects defect prediction models across performance, stability, interpretation, transferability, cost, and technique ranking. Results show substantial benefits for parameter-sensitive classifiers, while random forest is comparatively insensitive.

  • Findings: Up to 40 percentage points, optimization improves AUC performance for defect prediction models, with non-negligible gains for 16 of 26 techniques.C5.0, neural networks, and CART are examples of techniques benefiting from optimization; random forest generally changes little.
  • Findings: Optimized classifiers are at least as stable as default-settings classifiers, and 9 of 26 techniques are significantly more stable after optimization.For random forest and naïve bayes, default and optimized models are equally stable.
  • Findings: As few as 28% of top-ranked variables retain the same ranks after optimization, whereas 95% retain their ranks for techniques with negligible performance improvement.Interpretation shifts most for C5.0, neural networks, and CART, and least for random forest.
  • Findings: Seventeen of the 20 most sensitive parameters transfer across datasets with similar metrics without a statistically significant performance drop.Parameters for LogitBoost, FDA, and xGBTree cannot be transferred across any studied datasets.
  • Findings: Less than 30 minutes of additional grid-search computation is required for 12 of 26 techniques, while the three slowest techniques cost less than $1 on Amazon EC2.The computational burden varies substantially across techniques.
  • Implications: C5.0 can substantially outperform widely used techniques such as random forest after optimization, while random forest is not consistently top-performing across datasets.The results motivate exploring parameter settings rather than relying only on commonly used classifiers.

2 THE RELEVANCE OF PARAMETER SETTINGS FOR DEFECT PREDICTION MODELS

Parameter settings are central to defect prediction because most commonly used classification techniques require configurable parameters. Selecting those settings is therefore an important experimental design choice.

  • Parameter prevalence: The study examines 30 classification techniques spanning 11 classifier families, including techniques added from recent defect prediction literature.Some techniques, such as logistic regression, do not require parameter settings.
  • Parameter prevalence: 26 of the 30 most commonly used classification techniques require at least one parameter setting.The study identifies 25 unique parameters across the techniques examined.
  • Parameter prevalence: Selecting optimal parameter settings is an important experimental design choice for defect prediction models.This follows from the prevalence of configurable parameters among commonly used techniques.

3 RELATED WORK & RESEARCH QUESTIONS

Prior work identifies suboptimal defaults, instability, interpretation concerns, transferability questions, and computational costs as unresolved issues in defect prediction parameter optimization. The paper frames these issues through research questions about performance and practical use.

  • Motivation: Default parameter settings are often used despite evidence that they can be suboptimal and despite toolkit-dependent differences in defaults.Prior studies and the authors’ own work report reliance on default settings.
  • Research questions: The central performance question is how automated parameter optimization affects defect prediction model performance.This question addresses limited prior knowledge about optimization’s impact in this application area.
  • Research questions: Automated optimization may increase overfitting risk and produce unstable performance estimates, motivating a question about performance instability.Instability can make replication more difficult and may introduce misleading conclusions.
  • Research questions: The paper asks how much model interpretation changes when automated parameter optimization is applied.The motivation is that defect models are also used to understand characteristics of defect-prone modules.
  • Research questions: The paper asks how well optimal parameter settings transfer from one dataset to another.Prior work indicates that model performance can depend on dataset characteristics.
  • Research questions: The paper asks what computational cost automated parameter optimization introduces.Existing work suggests that some techniques are slow and that cost depends on training-data size.

4 STUDIED DATASETS

The study uses datasets selected by three criteria addressing dataset diversity, overfitting risk, and class imbalance. The resulting collection includes 18 datasets from proprietary and open-source systems with varying size and domain.

  • The dataset selection process was designed to reduce bias from repeatedly reusing experimental components such as datasets, metrics, and classifiers.Dataset choice can affect defect-prediction performance and conclusions.
  • Datasets with EPV values below 10 were excluded to reduce the risk of unstable results from overfitting.EPV is the ratio of defective modules to the number of independent variables.
  • Five datasets with defective rates above 50% were excluded because imbalanced data can favor the majority class and produce overly optimistic performance values.A ZeroR classifier may suffice when the defective ratio is high.
  • 18 datasets satisfy the study’s analysis criteria.The datasets include proprietary and open-source systems of varying size and domain.

5 CASE STUDY APPROACH

The case study compares defect prediction models trained with default and optimized parameters across repeated out-of-sample bootstrap evaluations. Grid search evaluates controlled candidate settings, selects the best-performing configuration, and measures models with threshold-dependent and threshold-independent metrics.

  • 5.1 Generate Bootstrap Sample: 100 out-of-sample bootstrap repetitions estimate average model performance on rows excluded from each bootstrap sample.Each bootstrap sample has the original dataset size, and testing uses non-overlapping observations.
  • 5.2 Identify Optimal Setting: Grid search evaluates controlled candidate parameter settings that remain consistent across datasets and bootstrap samples.The study uses Caret’s train function and a tuneLength budget threshold of 5.
  • 5.2 Identify Optimal Setting: All combinations of candidate parameter values are evaluated before selecting the setting with the highest estimated performance.For two parameters with five candidate values each, grid search evaluates 25 combinations.
  • 5.3 Construct Defect Prediction Models: Models are compared using optimized and default settings without re-balancing or re-sampling the training data.This keeps the training and testing corpora similar in their characteristics.
  • 5.4.2 Threshold-Independent Performance Measures: Threshold-independent measures are included because threshold-dependent metrics depend on an arbitrary threshold and are sensitive to imbalanced data.The threshold-independent measures are AUC, Brier, and LogLoss; AUCEC could not be computed with Caret’s implementation.
  • 5.4 Measure Performance: The study uses nine threshold-dependent and three threshold-independent performance measures.Threshold-dependent measures use a default probability threshold of 0.5, while AUC measures discrimination without a threshold.

5.5 Rank Variables by Importance Score

The paper ranks variables by a generic importance score based on how permuting each variable changes misclassification rates, then uses Scott-Knott ESD to group statistically distinct importance levels.

  • Importance-score calculation: Variable importance is computed for each classifier using a score derived from Breiman’s Variable Importance Score.The approach applies across classification techniques and generates importance ranks for optimized- and default-setting models.
  • Importance-score calculation: Each variable is randomly permuted in testing data while all other variables remain unchanged.This produces a dataset in which the selected variable’s values are shuffled.
  • Importance-score calculation: The larger the increase in misclassification rate after permutation, the greater the variable’s importance.Scores are computed repeatedly across variables and 100 experiment repetitions.
  • Scott-Knott ESD test: Scott-Knott ESD partitions treatment means into statistically distinct groups using hierarchical clustering and non-negligible effect-size differences.The test ranks variables while reducing overlapping-group ambiguity in post-hoc comparisons.
  • Scott-Knott ESD test: The Scott-Knott ESD procedure first finds a partition maximizing between-group mean differences, then splits or merges groups using effect-size magnitudes.Cohen’s d is used to assess the magnitude of differences between treatment means.

6 CASE STUDY RESULTS

Across the case study, parameter optimization often improves defect-prediction performance without reducing stability, while changing variable interpretations and imposing technique-dependent computational costs.

  • Technique sensitivity: Random forest gains little on AUC from optimization but improves by up to 12% in Precision, 18% in Recall, and 15% in F-measure.The reported AUC improvement is as little as 5%, compared with as little as 2% for Brier performance.
  • Performance: C5.0 achieves the largest median improvement, at 27 percentage points of AUC, with gains ranging from 6-40 percentage points.The #boosting iterations parameter is identified as C5.0’s most influential parameter; its optimized value is 40 versus the default value of 1.
  • Performance: Up to 40 percentage points of AUC improvement results from optimization, with non-negligible gains for 16 of 26 classification techniques.C5.0, neural networks, and CART are examples of techniques with substantial gains.
  • Stability: Optimized classifiers are at least as stable as default-settings classifiers across all studied techniques.Nine of 26 techniques are significantly more stable after optimization, including C5.0, neural networks, and CART.
  • Interpretation: Optimization shifts variable-importance rankings, with as few as 28% of optimized top-ranked variables appearing at the same ranks as non-optimized variables.For techniques with negligible performance improvement, 95% of variables appear at the same rank.
  • Transferability: The optimal settings of 17 of the 20 most sensitive parameters transfer across similar-metric datasets without a statistically significant performance drop.LogitBoost, FDA, and xGBTree parameters cannot be transferred across any studied datasets.

7 REVISITING THE RANKING OF CLASSIFICATION TECHNIQUES FOR DEFECT PREDICTION MODELS

The paper revisits classifier rankings across 18 datasets using double Scott-Knott ESD analysis and finds that optimized classifiers often rank above their default-settings counterparts.

  • Ranking methodology: Double Scott-Knott ESD compares optimized and default-settings classifiers within datasets and then summarizes their ranks across all 18 datasets.The method uses dataset-specific AUC distributions and bootstrap iterations before the across-dataset comparison.
  • Random forest: Random forest appears at the second Scott-Knott ESD rank rather than being the most frequent top performer.Its average rank is 3.277, and optimization does not change its ranking.
  • Ranking results: Optimized xGBTree, C5.0, and GBM are the only classifiers identified as top-performing across all studied datasets.Their average ranks are reported as 2.39, 2.56, and an additional value in the cited passage.
  • Ranking results: Optimization changes C5.0 from the ninth rank to the first rank.This ranking change is consistent with C5.0 having the largest performance benefit from optimization.
  • Ranking results: Optimization improves the ranking of 18 of 26 classification techniques, including C5.0, GBM, xGBTree, AVNNet, NNet, and CART.Optimized classifiers generally appear more often at the top Scott-Knott ESD ranks than default-settings classifiers.
  • Implications: C5.0 can substantially outperform widely used techniques such as random forest after parameter exploration.The result highlights the importance of examining classifier parameter settings.

FECT PREDICTION MODELS

The paper compares grid search, random search, genetic algorithms, and differential evolution for optimizing defect-prediction parameters. These techniques yield similar performance benefits, although computational cost can change the practical choice.

  • Research question: The study asks which automated optimization techniques yield the largest performance improvement for defect-prediction models.The motivation is that different optimization techniques may influence study conclusions.
  • Optimization techniques: The study selects grid search, random search, genetic algorithms, and differential evolution as a manageable representative set of search-based techniques.The selection is informed by a literature analysis of commonly used SBSE methods.
  • Optimization techniques: Grid search systematically evaluates candidate settings generated within a specified budget and selects the optimal setting.With three parameters and a budget of 5, it generates 5 ∗5 ∗5 = 125 combinations.
  • Optimization techniques: Random search generates a fixed number of random candidate combinations according to an iteration threshold, independent of parameter count.An iteration threshold of 5 generates five combinations for a classifier.
  • Optimization techniques: Genetic algorithms evolve parameter-setting populations through parent selection, crossover, mutation, and performance evaluation.Mutation introduces diversity and expands exploration of unexplored search regions.
  • Optimization techniques: Differential evolution uses mutation rather than crossover as its search mechanism and is described as potentially more stable than genetic algorithms.The cited discussion also reports better global solutions for differential evolution than for genetic algorithms.
  • Results: Grid search, random search, genetic algorithms, and differential evolution yield similar AUC performance-improvement benefits.The performance-difference ratios are centered at one across the studied performance measures.
  • Results: Computational cost can change the optimization recommendation despite similar performance improvement.The paper explicitly notes that other dimensions, such as computation time, may alter the choice.

9 DISCUSSION

The discussion finds that parameter optimization can materially affect defect-prediction results, while its effect is negligible for random forest. It also identifies implications for model interpretation, parameter reuse, computational adoption, and classifier comparisons.

  • Conclusions of Prior Defect Prediction Studies: Automated optimization has little impact on random forest AUC, stability, or model interpretation, reducing concern about prior studies using random forest.The paper therefore views automated optimization as posing little threat to conclusions based on random forest classifiers.
  • Implications: Parameter settings can substantially affect defect-prediction model performance, stability, and interpretation, so researchers should experiment with classifier parameters.The discussion contrasts these effects with cases where optimization has negligible or small impact.
  • Related Work: The study adds analyses of model stability, parameter transferability across datasets, and empirical comparisons of optimization techniques to prior work.These contributions complement earlier evidence on performance, model interpretation, and rankings of top-performing classifiers.
  • Parameter Optimization for Defect Prediction Models: Four off-the-shelf optimization techniques can efficiently tune defect-prediction classifiers, extending search-based software engineering to this task.The techniques discussed are grid search, random search, genetic algorithm, and differential evolution.
  • Implications: Optimization can produce large performance benefits, whereas widely used techniques such as random forest may gain little from it.The discussion supports considering optimization particularly for parameter-sensitive classification techniques.

10 THREATS TO VALIDITY

The validity discussion identifies boundaries involving parameter-search budgets, data transformations and noise, dataset coverage, evaluation measures, and optimization-technique selection. Several checks produce consistent conclusions, but some choices may still affect generalization or practical selection.

  • Construct Validity: The datasets vary in metrics and module granularity, although analyses on datasets sharing metrics found that predictor number and type did not influence the findings.This variation may affect results while also supporting generalization beyond one metric set.
  • Construct Validity: The grid-search budget limits parameter-space exploration, and different budgets may yield different results.Experiments with budgets of 3, 5, and 7 produced consistent conclusions, reducing but not eliminating this concern.
  • Construct Validity: Log transformation may not be necessary for all classifiers, and other data-transformation choices may produce different results.The authors specifically note logistic regression as an exception and treat alternative transformations as a remaining concern.
  • Internal Validity: Noisy data may influence conclusions, although removing the problematic NASA datasets did not alter the study’s conclusions.The authors note that even cleaned NASA datasets may remain problematic.
  • External Validity: The study covers a limited number of systems, so its results may not generalize to all open-source and commercial software systems.The authors frame the study as demonstrating that parameter optimization matters for some datasets and call for replication.
  • External Validity: Conclusions about classifier rankings rely on AUC, although repeating the experiment with MCC did not alter those rankings.Optimized C5.0 remained top-performing and outperformed random forest under MCC; 6 of 8 AUC top-three classifiers also appeared in MCC top-three ranks.
  • External Validity: The comparison of optimization techniques relies on performance improvement, while computational cost, metric customization, and optimizer-parameter sensitivity also matter in practice.The study examines only grid search, random search, genetic algorithm, and differential evolution, which may not represent all optimization techniques.

11 CONCLUSIONS

Using a case study of 18 datasets, the paper examines automated parameter optimization across defect prediction models. Optimization can substantially affect performance, stability, model interpretation, parameter transferability, computational cost, and technique rankings, although effects vary by classifier.

  • The study investigates performance improvement, stability, model interpretation, parameter transferability, computational cost, and classification-technique rankings using grid search.
  • Automated parameter optimization improves defect prediction AUC performance by up to 40 percentage points.
  • Optimized classifiers are at least as stable as default-setting classifiers, while random forests show negligible to small AUC impact from optimization.
  • Optimization substantially shifts software-metric importance rankings, with as few as 28% of optimized top-ranked variables also appearing among default-setting top-ranked variables.
  • The optimized settings of 17 of the 20 most sensitive parameters transfer across similar datasets without a statistically significant performance drop.
  • Grid search, random search, genetic algorithm, and differential evolution yield similar performance-improvement benefits across performance measures.
  • Future Work: The authors caution that their results are not intended to generalize, and future work should study technique rankings and ensemble-learning optimization.
  • Future Work: The conclusions about classification-technique rankings rely on grid-search optimization, motivating further investigation of other optimization techniques.

APPENDIX

Figure 15 compares AUC performance for optimized and default-setting models by classification technique.

  • Figure 15 presents optimized-versus-default AUC performance for each classification technique.
Loading 1801.10270v1…