PaperlayerView exact revision on arXiv

Source-linked AI summary

An Empirical Study on Software Defect Prediction with a Simplified Metric Set

Peng He, Bing Li, Xiao Liu, Jun Chen, Yutao Ma

arXiv:1402.3873v4cs.SE

TL;DR

The paper addresses unclear choices between within- and cross-project defect prediction when historical data are insufficient. It evaluates simplified metric sets across classifiers and training scenarios, finding that simplified predictors perform well, simple classifiers are effective, and minimum subsets can support general prediction with acceptable precision loss.

  • Problem

    When historical project data are insufficient, the appropriate choice between within-project and cross-project defect prediction and the usefulness of simplified metric sets remain unclear.

  • Method

    The study evaluates three predictor types across three scenarios and six classifiers, then selects Top-k metrics, removes redundant metrics, and tests minimum-subset stability with one-way ANOVA.

  • Results

    WPDP generally provides higher precision, whereas CPDP achieves higher recall and F-measure; simplified predictors show no significant disadvantage against benchmark predictors, and simple classifiers tend to perform well.

  • Takeaways & Limitations

    Training data should match the required accuracy, while simplified or minimum metric sets and simple classifiers can provide practical defect prediction with limited resources.

  • Takeaways & Limitations

    The study uses static code metrics, and other metric types could produce different results.

Abstract

from arXiv · show

Software defect prediction plays a crucial role in estimating the most defect-prone components of software, and a large number of studies have pursued improving prediction accuracy within a project or across projects. However, the rules for making an appropriate decision between within- and cross-project defect prediction when available historical data are insufficient remain unclear. The objective of this work is to validate the feasibility of the predictor built with a simplified metric set for software defect prediction in different scenarios, and to investigate practical guidelines for the choice of training data, classifier and metric subset of a given project. First, based on six typical classifiers, we constructed three types of predictors using the size of software metric set in three scenarios. Then, we validated the acceptable performance of the predictor based on Top-k metrics in terms of statistical methods. Finally, we attempted to minimize the Top-k metric subset by removing redundant metrics, and we tested the stability of such a minimum metric subset with one-way ANOVA tests. The experimental results indicate that (1) the choice of training data should depend on the specific requirement of prediction accuracy; (2) the predictor built with a simplified metric set works well and is very useful in case limited resources are supplied; (3) simple classifiers (e.g., Naive Bayes) also tend to perform well when using a simplified metric set for defect prediction; and (4) in several cases, the minimum metric subset can be identified to facilitate the procedure of general defect prediction with acceptable loss of prediction precision in practice. The guideline for choosing a suitable simplified metric set in different scenarios is presented in Table 12.

1. Introduction

Defect prediction helps engineers focus limited testing and maintenance resources on defect-prone components, but effective model choice depends on historical data, metrics, classifiers, precision, generality, and cost. This study therefore proposes and evaluates simplified metric-set predictors across different scenarios.

  • Defect prediction estimates defect-prone software components to support quality assessment, quality assurance, testing, and maintenance activities.
  • Within-project defect prediction trains on historical releases from the same project, but new projects may lack sufficient data for accurate models.
  • Cross-project defect prediction uses historical data from other projects, with training-data selection depending on the distributions of available data sets.
  • Complex predictors may improve precision while reducing generality and increasing acquisition and processing costs, whereas simple predictors reduce effort and cost with some precision sacrifice.
  • The study proposes a filter-based approach for selecting representative metrics and evaluates simplified predictors across classifiers and training-data scenarios.
  • It further examines whether minimum metric subsets can provide respectable overall results and remain stable across the classifiers studied.

2. Related Work

Prior defect-prediction research has examined classifiers, metrics, within-project and cross-project settings, and feature selection. However, empirical evidence remains limited on whether simplified or minimum metric sets can jointly balance effort, cost, accuracy, and generality.

  • Defect prediction research has used machine-learning models, with Naive Bayes and Logistic Regression among the simple techniques reported to perform well.
  • Within-project studies often overlook new projects with insufficient historical data, motivating research on cross-project prediction.
  • Cross-project research has studied transferring models, filtering irrelevant data, mixing project data, and selecting suitable training data.
  • Relatively little research has empirically evaluated simplified or minimum metric sets using effort-and-cost, accuracy, and generality as joint perspectives.
  • Prior work has used diverse software metrics, including source-code, process, complexity, code-churn, developer-activity, and network metrics.
  • Feature selection can reduce feature counts and accelerate prediction, but its stability and generality may vary across data sets.

3. Problem and Approach

The study progressively simplifies defect-prediction metrics by filtering features, identifying frequently recurring Top-k metrics, and removing redundant correlations. It evaluates training-data choices, simplified subsets, classifiers, and minimum subsets against practical accuracy requirements.

  • Analysis of Defect Prediction Problem: Directly training on all features increases feature-analysis and learning burdens and can generate redundant information, motivating feature selection.
  • Analysis of Defect Prediction Problem: Filters select features without involving a learning algorithm, while wrappers use feedback from a classification learner.
  • Analysis of Defect Prediction Problem: Metric set simplification progressively reduces the feature set to improve generality and manage strong correlations within Top-k metrics.
  • Research Questions: The study asks whether WPDP or CPDP is more suitable, whether simplified metrics work well, which classifier is preferable, and whether a minimum subset exists.
  • Research Questions: A minimum subset is sought under an explicit trade-off in which removing strong correlations may increase generality while reducing precision, with practical criteria such as recall > 0.7 and precision > 0.5.
  • Top-k Feature Subset: ALL uses no feature selection, FILTER applies CfsSubsetEval with GreedyStepwise in Weka, and TOPK selects frequently occurring metrics from filtering results.
  • Top-k Feature Subset: Coverage measures representativeness between FILTER and TOPK metric groups, increasing with their intersection and decreasing with their union.
  • Minimum Feature Subset: The optimal k depends on metric occurrence counts and selected-subset size; in the study, TOP5 reached a peak Coverage(5) of 0.6.

4. Experimental Setup

The experiments use 34 releases from 10 open-source projects, binary defect labels, three training-data scenarios, and six classifiers to evaluate metric-set simplification.

  • Data Collection: 34 releases from 10 open-source projects provide 34 defect data sets for validation.Each data set contains class-file instances and 20 static code metrics.
  • Variables: Classes are labeled buggy when they contain at least one defect and non-buggy otherwise.The study converts defect counts into a binary classification target.
  • Experiment Design: Three scenarios compare nearest-release WPDP, all-history WPDP, and selected-release CPDP training data.The experiments use 24 corresponding test groups for comparing the scenarios.
  • Experiment Design: The study compares predictors using different metric-set sizes and then searches for a minimum subset from TOP5 metrics.The minimum-subset analysis follows the initial comparison of three metric-set sizes across the three scenarios.
  • Classifiers: Six classifiers—J48, Logistic Regression, Naive Bayes, Decision Table, SVM, and Bayesian Network—are evaluated.The implementations use Weka with default parameters unless otherwise specified.
  • Evaluation Measures: Prediction results are evaluated using Precision, Recall, and F-measure for binary defect classification.The measures distinguish false positives, false negatives, true positives, and true negatives.

5. Experimental Results

This section reports the primary experimental results used to address the study’s four research questions.

  • Primary Results: The primary results are reported to answer the four research questions formulated earlier.
  • Primary Results: The results section presents findings from the experiments rather than introducing a separate evaluation procedure.
  • Primary Results: The section organizes the paper’s empirical findings around the four stated research questions.

5.1. RQ1: Which type of defect prediction models is more suitable for a specific project between WPDP and CPDP?

Within-project and cross-project predictors show different strengths: WPDP favors precision, while CPDP favors recall and F-measure. CPDP may therefore suit projects with insufficient local data, while simplified TOP5 metrics can remain comparable.

  • RQ1 Results: WPDP generally yields higher precision, whereas CPDP achieves higher median recall and F-measure.For J48 with TOP5 metrics, Scenario 1 has precision, recall, and F-measure of 0.504, 0.385, and 0.226; Scenario 3 has 0.496, 0.651, and 0.526.
  • RQ1 Results: 0.504 versus 0.679, 0.385 versus 0.304, and 0.226 versus 0.20 show matched WPDP results for J48 across the two WPDP scenarios.These values correspond respectively to precision, recall, and F-measure for the TOP5 subset.
  • RQ1 Results: Increasing historical training data does not necessarily improve WPDP and can reduce recall through information redundancy.
  • RQ1 Results: CPDP may be feasible when a project lacks sufficient local historical data.The paper attributes WPDP’s precision advantage to project-data authenticity and CPDP’s recall advantage to broader external training data.
  • RQ1 Results: The TOP5 simplified metric subset shows a tendency toward comparable results to the other metric-set cases.The passage presents this as an observed tendency requiring further research-question analyses.

5.2. RQ2: Does the predictor built with a simplified metric set work well?

TOP5, using five frequently used metrics, generally provides acceptable performance versus ALL, FILTER, MaxRel, and mRMR, with limited precision loss and practical resource savings. Results vary by scenario and measure, including some unacceptable cases.

  • Comparison with ALL and FILTER: More than 80 percent of TOP5/ALL median ratios exceed 0.9, indicating little precision loss versus all metrics.Some ratios exceed 1, while several Scenario 1 and Scenario 2 TOP5 cases remain unacceptable, including J48 F-measure values below 0.7.
  • Performance across scenarios: TOP5 maintains relatively high WPDP median Precision of at least 0.5, while Naïve Bayes sharply improves Recall and F-measure.The Recall and F-measure values of other predictors are described as stable across the simplified metric-set evaluation.
  • Comparison with ALL and FILTER: TOP5 generally shows no significant difference from ALL across classifiers and measures, although four exceptions occur in Scenarios 1 and 2.The comparison uses Wilcoxon signed-rank tests with p > 0.01 as the no-significant-difference criterion.
  • Comparison with ALL and FILTER: TOP5 achieves improved Precision for WPDP and improved Recall or F-measure for CPDP compared with ALL, while requiring 25% of the effort for twenty metrics.The TOP5 subset is also generally comparable with FILTER, with only two Scenario 3 Precision cases showing p < 0.01.
  • Comparison with existing approaches: TOP5 compares favorably with MaxRel and mRMR, improving WPDP Precision and CPDP Recall while most comparison ratios exceed 0.9.Against mRMR, all three-measure ratios exceed 0.9 and most exceed 1.

5.3. RQ3: Which classifier is more likely to be the choice of defect prediction with a simplified metric set?

Classifier suitability depends on the scenario and target measure. Naïve Bayes performs strongly for Recall or F-measure, while Logistic Regression, SVM, and Decision Table are preferable for selected Precision or CPDP needs.

  • Classifier choice by scenario: Naïve Bayes provides the best median Recall and F-measure in Scenarios 1 and 2, despite a decreasing Precision trend.The paper therefore regards it as the most suitable WPDP classifier when Recall or F-measure is prioritized.
  • Classifier choice by scenario: Logistic Regression or SVM is more likely to be preferable when prediction models focus on Precision.This recommendation is made specifically in the discussion of WPDP classifier selection.
  • Classifier choice by scenario: Naïve Bayes tends to achieve greater values overall with simplified subsets, while Bayesian Network shows comparable results across scenarios.Decision Table is preferable for CPDP relative to Naïve Bayes and Bayesian Network, and SVM is preferred for WPDP Precision.

5.4. RQ4: Is there a minimum metric subset that facilitates the procedure for general defect prediction?

The study identifies CBO+LOC+LCOM as an empirically validated minimum metric subset for both WPDP and CPDP. It achieves results broadly comparable to TOP5 and remains stable across classifiers.

  • Identifying the minimum subset: Strong-correlation filtering reduces the candidate combinations to thirteen, using r > 0.6 to identify correlated metric pairs.The excluded pairs include CBO–CE, RFC–LCOM, RFC–CE, and RFC–LOC.
  • Identifying the minimum subset: CBO+LOC+LCOM is selected for detailed evaluation because multi-metric combinations show larger Coverage values than single metrics.Other high-Coverage combinations include CBO+LOC and LOC+LCOM+CE.
  • Prediction results: For Scenario 1, CBO+LOC+LCOM generally matches TOP5 across threshold counts, with slight Recall decreases for Logistic Regression and Naïve Bayes.The minimum subset improves or preserves several Precision, F-measure, and combined-condition results, except for noted classifier-specific cases.
  • Prediction results: Scenario 3 shows competitive CPDP results for the minimum subset, including higher Recall, F-measure, and Total counts than WPDP but lower Precision.The findings support the minimum subset for CPDP with little loss of precision.
  • Prediction results: TOP5 and CBO+LOC+LCOM show no statistically significant overall difference, although WPDP Naïve Bayes favors TOP5 for Recall and F-measure and the minimum subset for Precision.The comparison uses Wilcoxon signed-rank tests and Cliff’s effect size.
  • Interpretation: CBO+LOC+LCOM is empirically validated as a basic defect-prediction metric set within and across projects.The paper relates this choice to defect concentration, the LOC–defect relationship, and the roles of coupling and cohesion metrics.
  • Stability of the minimum subset: ANOVA p values greater than 0.1 indicate no significant Consistency difference among the six classifiers, supporting classifier-independent stability.The ANOVA tests the equality of Consistency means for simplified metric subsets at p < 0.05 significance.

5.5. A Summary of the Results

The study finds that simplified metric sets can balance generality, performance, and complexity across WPDP and CPDP. WPDP favors Precision, CPDP favors Recall and F-measure, and classifier or metric choices should follow requirements.

  • Overall findings: Simplified metric-set predictors are empirically feasible and practical for predicting defect-prone classes across different scenarios.The study frames this as a trade-off among generality, performance, and complexity.
  • Overall findings: WPDP models generally achieve higher Precision, whereas CPDP models achieve higher Recall and F-measure, with the distinction most evident for F-measure.The summary reports WPDP median F-measure as no higher than the middle level, while CPDP median F-measure is high or medium.
  • Classifier guidance: Naïve Bayes improves WPDP Recall while maintaining appropriate Precision, and simple classifiers such as Logistic Regression and Naïve Bayes perform well in CPDP overall.Decision Table is stable across WPDP metric sets, while Naïve Bayes and Bayesian Network are relatively stable across CPDP metric sets.
  • Practical guidance: Table 12 offers three metric-subset choices for engineers, including default-feature selections when WPDP requires Precision around 0.5.The guideline is intended to support specific defect-prediction requirements with simple classifiers such as Naïve Bayes.

6. Discussion

The discussion finds that training-data choice should match accuracy requirements, while simplified metric sets can reduce effort with limited performance loss. Simple classifiers and small stable metric subsets often remain practical, though specific performance goals may favor alternatives.

  • Training data: CPDP is feasible with limited project data and can outperform WPDP on Recall and F-measure.The study selected suitable cross-project training combinations exhaustively, using no more than three releases.
  • Guidelines: The study’s scenario-specific recommendations are summarized through performance requirements and metric-set labels in Tables 11 and 12.Table 11 distinguishes ALL, FILTER, and TOP5 sets and categorizes performance as high, medium, or low by median.
  • Training data: Training-data quality matters more than quantity because adding WPDP training sets showed no observable improvement.The authors caution that more data does not necessarily produce more precise predictors.
  • Classifiers: Simple classifiers, especially Naïve Bayes, tend to perform better overall with simplified metrics across all three scenarios.Naïve Bayes is reported as robust in both WPDP and CPDP, but it is not optimal for every performance requirement.
  • Metric subsets: LOC, CBO, and LCOM form a suitable minimum metric subset, while CBO+LOC offers a practical CPDP alternative.CBO+LOC has no statistically significant difference from CBO+LOC+LCOM in the cited Wilcoxon comparison, but performs unevenly across scenarios.
  • Metric sets: Simplified metric sets reduce data-acquisition and processing costs while sacrificing only a little performance.The approach was evaluated with Wilcoxon signed-rank tests and Cliff’s effect size in WPDP and CPDP.

7. Threats to Validity

The authors identify validity threats involving data construction, metric and benchmark choices, measurement settings, statistical interpretation, and limited project coverage. These constraints bound how broadly the findings should be generalized.

  • Construct validity: PROMISE defect datasets may contain identification errors and missing links between bug databases and source repositories.The datasets were nevertheless validated and used in prior studies.
  • Construct validity: Comparing ALL, FILTER, and TOP5 metrics may be biased because other metric combinations could achieve better precision.The selected metrics were retained as feasible comparison oracles because prior studies used them on the same datasets.
  • Internal validity: Using Top 5 metrics makes results sensitive to the chosen k length.The authors state that different k values could change the experimental results.
  • Conclusion validity: Statistical significance was assessed with Wilcoxon signed-rank and one-way ANOVA, while the null hypothesis could not be rejected in most cases.The tests were applied to the relevant research questions and 24 datasets.
  • Conclusion validity: The thresholds for recall and precision are based on prior studies rather than a strict universal criterion.The authors note that evaluation-criterion choice depends substantially on the defect datasets involved.
  • External validity: External validity is limited because the releases come from a very small subset of projects and mainly Java open-source projects.Only 10 Java projects were selected, nine maintained by the Apache Software Foundation.
  • External validity: Results may differ when using software metrics other than the static code metrics examined here.The study focuses on the trade-off among generality, cost, and accuracy rather than comparing metric types.

8. Conclusion

The study finds that simplified metric sets can support practical defect prediction across within- and cross-project scenarios, with training-data choices depending on accuracy requirements. It also identifies future work needed to validate the approach more broadly and incorporate defect counts.

  • WPDP models capture higher precision than CPDP models, whereas CPDP models achieve high recall or F-measure.
  • Training data should be selected according to the specific prediction-accuracy requirement.
  • Simplified metric-set predictors performed well, with no significant differences from more complex benchmark predictors.
  • Simplified or minimum metric subsets can provide satisfactory performance while reducing the effort required to build suitable predictors.
  • Future work will collect more open-source projects to validate generality and consider defect counts in prediction.
Loading 1402.3873v4…