Source-linked AI summary
Evaluating prediction systems in software project estimation
Martin Shepperd, Stephen G. MacDonell
TL;DR
Conflicting evaluations of software-engineering prediction systems make their comparative results difficult to interpret. The paper proposes a framework based on unbiased Standardised Accuracy, random-guessing baselines, chance testing, and effect sizes. Reanalysis finds unsafe conclusions in three published studies, while even the largest observed improvement over guessing has only a medium effect size.
Problem
Empirical evaluations and systematic reviews of competing software-engineering prediction systems produce inconsistent results, making safe practitioner recommendations difficult.
Method
The paper proposes a formal framework using unbiased Standardised Accuracy, random guessing as a baseline, chance-likelihood testing, and effect-size calculation.
Results
All three re-examined published validation studies contain empirical conclusions that are unsafe, and the largest observed effect over guessing is Δ = 0.664, a medium effect.
Takeaways & Limitations
The framework provides a basis for more rigorous appraisal of prediction-system results and their significance, supporting more meaningful interpretation.
Takeaways & Limitations
The analysis is restricted to continuous prediction outputs, and validation data sets are not random samples with their target population often left unspecified.
Abstract
from arXiv · showhide
Context: Software engineering has a problem in that when we empirically evaluate competing prediction systems we obtain conflicting results. Objective: To reduce the inconsistency amongst validation study results and provide a more formal foundation to interpret results with a particular focus on continuous prediction systems. Method: A new framework is proposed for evaluating competing prediction systems based upon (1) an unbiased statistic, Standardised Accuracy, (2) testing the result likelihood relative to the baseline technique of random 'predictions', that is guessing, and (3) calculation of effect sizes. Results: Previously published empirical evaluations of prediction systems are re-examined and the original conclusions shown to be unsafe. Additionally, even the strongest results are shown to have no more than a medium effect size relative to random guessing. Conclusions: Biased accuracy statistics such as MMRE are deprecated. By contrast this new empirical validation framework leads to meaningful results. Such steps will assist in performing future meta-analyses and in providing more robust and usable recommendations to practitioners.
1. INTRODUCTION
Software-engineering prediction research has produced inconsistent empirical and review findings, making it difficult to interpret competing systems or give safe practitioner advice. The paper therefore focuses on a formal way to compare systems rather than endorsing a particular technique.
- Prediction systems have been studied for software-engineering outcomes including cost, schedule, and defect-proneness.
- The inconsistency makes it difficult to determine what advice to offer practitioners and to form safe recommendations.
- More than 300 journal papers had examined cost or effort prediction by 2007, alongside many competing techniques.
- No single prediction technique dominates, and empirical comparisons often produce contradictory results across data sets.
- Systematic reviews likewise report conflicting evidence: seven local-versus-cross-company studies split three to four, while 20 regression-versus-analogy studies split seven, four, and nine.
- The paper proposes a formal framework using randomisation to assess chance likelihood and calculate effect sizes, then illustrates unsafe conclusions in three published studies.
2. A VALIDATION FRAMEWORK
The framework treats validation as estimating accuracy and ordering competing systems, while requiring comparisons with random guessing, chance testing, and practical effect size. It also replaces biased ratio-based accuracy assessment with standardised accuracy relative to guessing and highlights data-population and scope limitations.
- Validation evaluates prediction system Pi on data set D using accuracy statistic S and scheme V, which estimates performance on unseen cases.
- Researchers compare candidate systems through preference and indifference relations, forming an order or partial order over prediction systems.
- The framework asks whether a system beats random guessing, whether the difference could be chance, and whether its effect size justifies practical adoption.
- 2.1. Baselines: Random guessing permutes another case’s observed y value onto the target and requires no parameter estimates.
- 2.2. Significance testing: MMRE is a biased, asymmetric estimator that can differ by an order of magnitude for equal absolute residuals and favors underestimation.
- MAR SA: Standardised Accuracy measures MAR relative to random guessing, providing an unbiased centre measure without ratio-based asymmetry.
MAR SA
The framework estimates prediction accuracy against random guessing and complements this with effect-size analysis, using unbiased residual-based measures rather than ratio-based MMRE.
- MAR: Random guessing samples a prediction from the remaining cases, providing a baseline distribution for evaluating prediction accuracy.Typically, about 1000 runs are used to estimate the distribution and variance of MAR values.
- SA: The random-guessing distribution enables likelihood testing by comparing observed accuracy with its lower-tail quantiles.The 5% quantile acts as a threshold analogous to α: values better than it have less than a one-in-twenty chance under the baseline.
- SA: Standardised Accuracy measures MAR relative to random guessing, making accuracy comparisons possible across data sets.MAR is unbiased because it is not based on ratios, while SA contextualises it against the random-guessing baseline.
- Effect size: Glass’s Δ standardises the difference between a prediction system and random guessing using variation in the control measure.The random-guessing standard deviation is used instead of a pooled deviation because equal variances cannot be assumed.
- Effect size: Effect sizes are interpreted as small at approximately 0.2, medium at 0.5, and large at 0.8, though rigid interpretation has limitations.Glass’s Δ expresses the effect in standard-deviation units of reduction in mean absolute residual.
3. THREE EXAMPLES
Three examples show why prediction-system comparisons require a guessing baseline, unbiased accuracy statistics, chance testing, and effect-size interpretation. Reanalysis overturns some original conclusions and shows that statistically detectable improvements may have little practical significance.
- Q1: Is the prediction system better than guessing?: The first example used a small telecoms-project dataset and compared regression-to-the-mean with a variant of estimation by analogy.Random guessing and its 5% quantile were added as a baseline for interpreting accuracy.
- Q1: Is the prediction system better than guessing?: Neither PR2M nor PEBA’ predicted meaningfully in the first example: both performed worse than random guessing, including its median.Their apparent difference was therefore irrelevant for practical prediction.
- Q1: Is the prediction system better than guessing?: The first-example results should not be generalized to R2M or EBA generally because both implementations used a nonstandard linear productivity assumption.The authors explicitly caution that these results do not establish that either prediction technique is inherently poor.
- Q2: Is the difference due to chance?: On Telecom1, MMRE suggested EBA outperformed SWR, whereas MAR slightly favored SWR and the Mann–Whitney test found no reliable difference.Both techniques nevertheless improved accuracy by about 50% over random guessing and exceeded its 5% quantile.
- Q2: Is the difference due to chance?: MMRE can mislead because residuals are skewed and the measure is asymmetric; inappropriate statistics can therefore reverse researchers’ interpretations.Using a statistic inconsistent with the method’s squared-residual optimization is also inappropriate.
- Q3: Does the effect size have any practical significance?: EBA++ achieved only a medium effect size over guessing, while EBA+ versus EBA produced a significant but practically small effect of Δ = 0.177.High within-sample variance masked some underlying effects, and one comparison treated EBA++ and EBA+ as practically indifferent.
4. DISCUSSION
The paper proposes a formal framework to address inconsistent validation conclusions without endorsing any particular prediction technique. It argues that effect sizes, unbiased validation, and better reporting are needed for interpretable evidence and usable practitioner guidance.
- The framework asks whether a system predicts better than guessing, whether improvement could be due to chance, and whether its effect size is practically meaningful.These questions are applied to three published effort-prediction studies, whose empirical conclusions are judged unsafe.
- Even the largest observed effect, comparing guessing with EBA++, is medium at Δ = 0.664, or about two thirds of one standard deviation of accuracy statistic S.The authors describe this result as sobering and relevant to conclusion instability.
- The analysis focuses on continuous prediction systems, particularly project effort or cost prediction, while suggesting the arguments may also apply to classifiers.The distinction arises because continuous systems use residuals whereas classifiers use confusion matrices.
- The three-study analysis is illustrative rather than exhaustive, and the studies were selected mainly for convenience rather than representativeness.The authors state that a thorough audit of empirical results would support future meta-analysis.
- Preference based only on accuracy omits bias, explanatory value, ease of use, and computational tractability, so small accuracy gains may not be practical.The authors specifically warn that exchanging a very small positive effect for much lower computational tractability is not necessarily practical.
- The paper recommends unbiased validation schemes, clearer dataset justification, detailed reporting and archiving, and avoiding MMRE as an accuracy indicator.These recommendations target estimator bias, data representativeness, sensitivity to preprocessing and parameters, and unsafe asymmetric accuracy measurement.