Source-linked AI summary
Tuning for Software Analytics: is it Really Necessary?
Wei Fu, Tim Menzies, Xipeng Shen
TL;DR
Software analytics uses data miners whose tunings are difficult to select, raising the question of whether simple automatic optimization can find effective settings. The paper applies differential evolution to tuning defect predictors and finds that tuning substantially changes performance and analytical conclusions.
Problem
Selecting tunings for software data miners is a difficult research problem, especially when optimizing one performance measure can sacrifice another.
Method
The study uses differential evolution to search tuning spaces for each optimization goal and evaluates the resulting settings on hold-out data.
Results
Tuning usually improves predictor performance, sometimes changing precision from 0 to 60%, while also changing conclusions about better learners and important development factors.
Takeaways & Limitations
For static-code defect prediction, analytics without parameter tuning are considered harmful and misleading, so prior conclusions may need revisiting.
Takeaways & Limitations
The tuning results come from only one software analytics task: defect prediction from static code attributes; implications for other tasks remain unclear.
Abstract
from arXiv · showhide
Context: Data miners have been widely used in software engineering to, say, generate defect predictors from static code measures. Such static code defect predictors perform well compared to manual methods, and they are easy to use and useful to use. But one of the "black art" of data mining is setting the tunings that control the miner. Objective:We seek simple, automatic, and very effective method for finding those tunings. Method: For each experiment with different data sets (from open source JAVA systems), we ran differential evolution as anoptimizer to explore the tuning space (as a first step) then tested the tunings using hold-out data. Results: Contrary to our prior expectations, we found these tunings were remarkably simple: it only required tens, not thousands,of attempts to obtain very good results. For example, when learning software defect predictors, this method can quickly find tuningsthat alter detection precision from 0% to 60%. Conclusion: Since (1) the improvements are so large, and (2) the tuning is so simple, we need to change standard methods insoftware analytics. At least for defect prediction, it is no longer enough to just run a data miner and present the resultwithoutconducting a tuning optimization study. The implication for other kinds of analytics is now an open and pressing issue
1. Introduction
Software analytics increasingly relies on data miners for defect prediction, but tuning their parameters was largely treated as unnecessary or too costly. This study investigates whether simple tuning can improve results and alter common analytics conclusions.
- Data miners generate defect predictors from static code measures that can be extracted automatically, even for very large software systems.
- Researchers rarely tuned defect predictors because defaults were assumed well explored, while tuning was expected to be computationally expensive.
- 5 to 20% improvement was usual, with precision increasing from 0% to 60% in one extreme case.
- Some conclusions about which learners perform better change completely after tuning.
- Conclusions about which software-engineering factors matter most for reducing defects need to be revisited after tuning.
- Differential evolution provides a simple tuning approach that works well for defect predictors.
- Dramatic performance improvements required fewer than 100 evaluations, indicating that tuning was fast rather than impractically slow.
- For static-code defect prediction, data miners should not be used off-the-shelf with default tunings.
2. Preliminaries
Tuning is presented as an under-explored but consequential part of defect prediction: it can change learner behavior, empirical conclusions, and performance, while differential evolution finds useful settings efficiently. The section also emphasizes that optimization goals determine which outcomes matter and that analytics findings require contextual qualification.
- Tuning: Important and Ignored: Tuning changes a learner’s heuristics and returned models, so how we learn changes what we learn.
- Tuning: Important and Ignored: Few highly cited defect-prediction papers studied tuning, while grid search can take days, miss configurations, and waste effort on unimportant parameters.
- Tuning: Important and Ignored: Differential evolution found useful learner settings in less than 100 evaluations, making tuning practical for defect predictors.
- You Can’t Always Get What You Want: Optimization goals cannot always be satisfied simultaneously, so analytics results should be reported with the relevant context and goal.
- Tuning: Important and Ignored: Changing tuning can reverse conclusions about which learners perform better and which software-engineering factors matter most.
• Max
The study tunes learner parameters using differential evolution, a simple search that evolves candidate settings and evaluates them by learner performance. The tuning space includes tree-specific limits and general algorithm parameters.
- Max: The tuning space includes upper bounds on Random Forest leaves and CART tree depth, plus WHERE splitting and pruning controls.WHERE uses depthMin and wriggle to control splitting, while prune removes redundant subtrees sharing their parent’s majority cluster.
- Tuning Algorithms: The optimizer was selected as a simple candidate method, following advice to compare against simple alternatives and use simple learners as preliminary scouts.The paper reports that differential evolution works very well for tuning defect predictors.
- Tuning Algorithms: Differential evolution evolves candidate tunings from a population, replacing solutions when new candidates score better.Candidates pair tuning settings with scores obtained by training and testing a learner.
- Tuning Algorithms: For numeric parameters, differential evolution extrapolates between three randomly selected solutions and trims values to legal ranges.Boolean parameters are mutated by negation, while numeric values use a scale factor and crossover probability.
- Tuning Algorithms: The optimizer uses a population of 10, scale factor f = 0.75, crossover probability cr = 0.3, and life = 5 for early termination.The search terminates when the life counter reaches zero.
- Tuning Algorithms: Each optimization run extracts one performance value from repeated learner evaluations using a selected goal.The paper implements this as a single-objective differential-evolution search for each goal.
3. Experimental Design
The experiments use open-source Java defect datasets and temporally ordered releases to separate training, tuning, and testing. Performance is evaluated with standard binary-classification measures, focusing on precision and F-measure.
- 3.1. Data Sets: The defect data come from the PROMISE repository and cover ten open-source Java systems: ant, camel, ivy, jedit, log4j, lucene, poi, synapse, velocity, and xerces.The datasets are summarized in Table 3.
- 3.1. Data Sets: The experimental design avoids training-test contamination by using consecutive releases in temporal order.The approach addresses the limitations identified for leave-one-out and cross-validation.
- 3.1. Data Sets: The first release trains a candidate model, the second tests it during tuning, and the third supplies the reported performance statistics.This sequence is applied to datasets with at least three consecutive releases.
- 3.1. Data Sets: For untuned learners, the first and second releases are combined for training, and the same third release is used for evaluation.This keeps the tuned and untuned evaluations aligned on the test release.
- 3.1. Data Sets: Datasets with more than three releases support multiple experiments; ant’s five releases yield experiments V0, V1, and V2.The paper states that these datasets are displayed in Table 3.
- 3.2. Optimization Goals: The evaluation considers recall, false-positive rate, precision, and F-measure, with lower values preferred only for false-positive rate.The paper focuses subsequent tuning analyses on precision and F-measure.
- 3.2. Optimization Goals: Figure 1 compares tuned and untuned learners using deltas in precision and F-measure; positive deltas indicate improved performance from tuning.The figure organizes the deltas from Tables 4 and 5.
4. Experimental Results
Tuning usually improved defect-prediction performance and could substantially alter learner rankings, selected factors, and the case for default settings. Differential evolution found useful tunings quickly, often within 50–80 evaluations.
- RQ1: Does Tuning Improve Performance?: Tuning rarely worsened precision or F-measure and often improved them substantially.The authors report that tuning had a positive overall effect on performance scores.
- RQ1: Does Tuning Improve Performance?: 32% to 70%: tuning changed xercesV1 WHERE precision by 38 percentage points.The reported example concerns precision for the WHERE learner.
- RQ1: Does Tuning Improve Performance?: 0 to 35 and 0 to 60: tuned WHERE raised antV0 and antV1 precision from zero.The passage reports tuned WHERE scores of 35 and 60 for antV0 and antV1, respectively.
- RQ2: Does Tuning Change a Learner’s Ranking?: 13 of 17 to 6 of 17: after tuning, Random Forest lost to Logistic Regression in fewer data sets.Untuned Random Forest performed worse than Logistic Regression in 13 of 17 data sets; tuned Random Forest did so in 6 of 17.
- RQ2: Does Tuning Change a Learner’s Ranking?: 12 data sets: tuned CART was better than or equal to tuned Random Forest in precision and F-measure, while the tuned learners were not statistically different by the KS test.Before tuning, the corresponding count was 5 data sets.
- RQ3: Does Tuning Select Different Project Factors?: Tuning changed which features WHERE selected and therefore changed conclusions about important software-engineering factors.Table 6 contrasts features selected by tuned and untuned WHERE; the authors say many earlier conclusions should be revisited.
- RQ4–RQ5: Is Tuning Easy and Fast?: 50 to 80 evaluations generally sufficed to find the reported tuning improvements, contrary to expectations of thousands or millions.The authors also report that tuning time was not extremely long for most data sets.
- RQ6: Should We Use Off-the-Shelf Tunings?: Tuning must be repeated when goals or data sets change, so defect predictors should not be used with default tunings alone.The authors characterize differential evolution as simple and fast enough for routine defect prediction tuning.
5. Reliability and Validity
The study supports reproducibility through public algorithms and data, but its external validity is limited. The experiments cover one software-analytics task and only precision, recall, and F-measure.
- Reliability: The algorithms are defined or drawn from public-domain implementations, and the data and algorithms are available online.The data are available through the PROMISE repository and the algorithms through the cited GitHub repository.
- External Validity: The experiments cover defect prediction from static code attributes and evaluate precision, recall, and F-measure.The authors state that implications for other analytics tasks remain unclear.
- External Validity: Other software-analytics tasks may use similar learners, but the paper does not establish that its conclusions generalize to them.The authors describe broader applicability as possible rather than demonstrated.
6. Conclusions
For static-code defect prediction, untuned analytics can produce misleading conclusions: tuning improves predictor performance and changes comparisons among learners and important factors. The paper recommends tuning studies while limiting broader claims because it examines some learners with one optimizer.
- The authors argue that analytics without parameter tuning are harmful and misleading for learning defect predictors from static code attributes.
- Tuning improves predictor performance, sometimes dramatically, including precision changing from 0% to 60%.
- Tuning changes conclusions about which learners outperform others, requiring numerous prior publications to be revisited.
- Tuning changes conclusions about which factors are most important in software development, so older studies may need revision.
- The scope is limited to some learners and one optimizer, so the paper does not claim differential evolution is best for all learners.