PaperlayerView exact revision on arXiv

Source-linked AI summary

Evaluation: from precision, recall and F-measure to ROC, informedness, markedness and correlation

David M. W. Powers

arXiv:2010.16061v1cs.LGstat.MEstat.ML

TL;DR

Common evaluation measures can be biased because they ignore negative cases, prevalence, bias, or chance-level performance. The paper develops Informedness and Markedness as debiased counterparts to Recall and Precision, connects them with correlation and significance, and extends the framework toward multiclass settings.

  • Problem

    Recall, Precision, F-Measure, and related measures omit negative-case performance or reflect prevalence, bias, and chance-level effects.

  • Method

    The paper constructs Informedness and Markedness from contingency matrices as debiased counterparts to Recall and Precision, including dichotomous and multiclass formulations.

  • Results

    The resulting statistics form connections among Informedness, Markedness, Correlation, Significance, Recall, and Precision.

  • Takeaways & Limitations

    Recall and Informedness suit effectiveness relative to conditions, whereas Precision and Markedness suit effectiveness relative to predictions.

  • Takeaways & Limitations

    The proposed correlation and significance statistics have not yet been applied to multiclass experiments or compared extensively with conventional approaches.

Abstract

from arXiv · show

Commonly used evaluation measures including Recall, Precision, F-Measure and Rand Accuracy are biased and should not be used without clear understanding of the biases, and corresponding identification of chance or base case levels of the statistic. Using these measures a system that performs worse in the objective sense of Informedness, can appear to perform better under any of these commonly used measures. We discuss several concepts and measures that reflect the probability that prediction is informed versus chance. Informedness and introduce Markedness as a dual measure for the probability that prediction is marked versus chance. Finally we demonstrate elegant connections between the concepts of Informedness, Markedness, Correlation and Significance as well as their intuitive relationships with Recall and Precision, and outline the extension from the dichotomous case to the general multi-class case.

INTRODUCTION · THE BINARY CASE · Recall & Precision, Sensitivity & Specificity

The paper argues that Recall, Precision, F-Measure and related measures have biases because they ignore negative-case performance, propagate prevalence and prediction biases, and omit chance-level performance. It revisits binary contingency-table measures and develops probabilistic and information-theoretic alternatives to address these limitations.

  • INTRODUCTION: Recall, Precision and F-Measure ignore correctly handled negative examples, propagate marginal prevalences and biases, and fail to account for chance-level performance.The paper identifies these measures as poorly motivated for evaluating machine-learning experiments.
  • INTRODUCTION: The paper re-examines relationships among evaluation measures and introduces probabilistic and information-theoretic measures intended to overcome problems with Recall, Precision and their derivatives.
  • THE BINARY CASE: Binary classification is represented with positive and negative labels, one predicting model, one gold-standard labeling, and a four-cell contingency table.The paper generally assumes a non-trivial contingency in which both predicted and real positive and negative states occur.
  • Recall & Precision, Sensitivity & Specificity: Recall or Sensitivity measures the proportion of Real Positive cases correctly Predicted Positive and is also called True Positive Rate.Recall is defined as Recall = Sensitivity = tpr = tp/rp = TP / RP = A /(A+C).
  • Recall & Precision, Sensitivity & Specificity: Precision or Confidence measures the proportion of Predicted Positive cases that are correctly Real Positives and can be called True Positive Accuracy.Machine Learning, Data Mining and Information Retrieval focus on Precision, whereas ROC analysis ignores it.
  • Recall & Precision, Sensitivity & Specificity: Recall and Precision focus on positive examples and predictions, while neither captures how well the model handles negative cases or incorporates True Negatives.Their arithmetic, geometric and harmonic means inherit this limitation.
  • Recall & Precision, Sensitivity & Specificity: Inverse Recall is the True Negative Rate, and Inverse Precision is True Negative Accuracy, providing complementary measures for negative cases.They are defined as Inverse Recall = tnr = tn/rn = TN/RN = D/(B+D) and Inverse Precision = tna = tn/pn = TN/PN = D/(C+D).
  • Recall & Precision, Sensitivity & Specificity: F1 completely ignores TN, whereas Rand Accuracy includes negatives but remains sensitive to prevalence and bias because both are independent or controllable variables.Rand Accuracy is a prevalence-weighted average of Recall and Inverse Recall and a bias-weighted average of Precision and Inverse Precision.

Prevalence, Bias, Cost & Skew · ROC and PN Analyses

The section frames prevalence, bias, skew, and cost as sources of variation in surface measures, then uses ROC analysis to distinguish chance, better-than-chance, and worse-than-chance performance. It identifies WRAcc and tpr−fpr as unbiased or skew-insensitive criteria while relating Accuracy, Precision, and F-Measure to Recall, Bias, and Prevalence.

  • Prevalence, Bias, Cost & Skew: Prevalence rp is the population’s positive-case proportion, whereas bias pp concerns predicted positive labels; evaluation therefore seeks measures independent of prevalence.Prevalence may vary across subpopulations but is treated as outside the experimenter’s control.
  • Prevalence, Bias, Cost & Skew: Under high skew, poor Inverse Recall produces more errors than equally poor Recall because negative cases are more numerous, even when False Positive and False Negative costs are individually equal.The larger negative class makes False Positives dominate the total cost at significant chance-performance levels.
  • Prevalence, Bias, Cost & Skew: Recall, Inverse Recall, and Prevalence—or equivalently tpr, fpr, and cs—determine all ratios and measures derivable from the normalized binary contingency table, while N recovers count information.The normalized table has three degrees of freedom, with N additionally required for the original counts.
  • Prevalence, Bias, Cost & Skew: Differential positive and negative values enter through the Cost Ratio cv=cn/cp, and value and skew effects can combine into the skew-like cost factor c=cvcs.Costs may represent losses for errors or gains for correct cases.
  • ROC and PN Analyses: ROC plots tpr against fpr, with the top-left corner perfect, the bottom-right corner worst, and the positive diagonal tpr=fpr representing random classification.The negative diagonal tpr+c·fpr=1 represents Bias matching Prevalence for skew c.
  • ROC and PN Analyses: ROC model selection can target proximity to (0,1), distance from chance, AUC, or normalized-error heuristics, but these criteria are not identical.Euclidean distance minimizes fpr^2+fnr^2, while the ROC curve can also support convex-hull interpolation, repair, and adaptation to changing prevalence and costs.
  • ROC and PN Analyses: For c=1, maximizing AUC is equivalent to maximizing tpr−fpr, minimizing fpr+fnr, and selecting the highest parallel isocost line.The chance line has tpr−fpr=0, and skew-insensitive WRAcc is precisely tpr−fpr.
  • ROC and PN Analyses: Accuracy, Precision, and F-Measure vary with Prevalence and Bias, whereas WRAcc subtracts chance-attributable True Positive performance and is unbiased; maximizing WRAcc is equivalent to maximizing AUC when c is constant.Their formulations are Accuracy = [tpr+c·(1-fpr)]/[1+c], Precision = tpr/[tpr+c·fpr], F1 = 2·tpr/[tpr+c·fpr+1], and WRAcc = 4c·[tpr−fpr]/[1+c]^2.

DeltaP, Informedness and Markedness · Causality, Correlation and Regression

The paper defines Informedness and Markedness as chance-relative measures of prediction and condition, respectively, and relates them to DeltaP, regression, correlation, causality, and ROC geometry. These coefficients depend on contingency-table deviations from expectation, normalized by prevalence, bias, or their combination.

  • Definition 1: Informedness measures how informed a predictor is for a specified condition, expressing the probability that its prediction is informed rather than chance.It is also described as an unbiased accuracy measure addressing prevalence and label bias.
  • Definition 2: Markedness measures how marked a condition is for a specified predictor, expressing the probability that the condition is marked rather than chance.In psychology, Markedness is known as DeltaP and predicts human associative judgements when DeltaP is high.
  • Definition 2: For binary outcomes, Informedness = Recall + Inverse Recall − 1, while Markedness = Precision + Inverse Precision − 1.The equivalent forms are tpr-fpr and tpa-fna, respectively.
  • Causality, Correlation and Regression: Informedness and Markedness are dual regression coefficients: predicting outcomes from predictors yields Markedness, while predicting predictors from outcomes yields Informedness.The equations identify rP with DeltaP = Markedness and rR with DeltaP' = Informedness.
  • Causality, Correlation and Regression: Strong prediction does not establish causal direction: A → B does not generally imply B → A, and regression or DeltaP direction can differ from causality.The paper notes that multiple causes or necessary contributing factors can reverse the relative strength of the two predictive directions.
  • Causality, Correlation and Regression: The Matthews/Pearson correlation is the Geometric Mean of Bookmaker Informedness and Markedness, while BookMark acts as a probability-like coefficient of determination.Perfect correlation requires perfect regression because both Informedness and Markedness are bounded above by 1.
  • Causality, Correlation and Regression: These coefficients represent Delta True Positives, dtp=tp-etp, renormalized by Prevalence, Bias, or their combination.Prevalence · Bias is the expected proportion of True Positives relative to N.
  • Causality, Correlation and Regression: Informedness is twice the triangular ROC area between a positively informed system and the chance line.The corresponding geometry uses a trapezoid bounded by the system, its perversions, and the chance-line endpoints.

Effect of Bias and Prev on Recall and Precision

Recall and Precision combine unbiased performance with Bias and Prevalence, so their values—and F-measure—can be skewed relative to chance. Bookmaker Informedness and Markedness remove these chance-level effects and clarify the duality between the measures.

  • Equations: Recall = Bookmaker (1−Prevalence) + Bias, while Precision = Markedness (1-Bias) + Prevalence.The corresponding renormalizations are Bookmaker = (Recall-Bias)/(1−Prevalence) and Markedness = (Precision−Prevalence)/(1-Bias).
  • Bias effects: At chance, Bookmaker and Markedness are zero, whereas Recall, Precision, and F-measure are skewed by Bias and Prevalence.Bookmaker and Markedness are described as unbiased estimators of above-chance performance.
  • Bias effects: Increasing Bias or decreasing Prevalence increases Recall, while high Bias drives Recall up and Precision down according to Informedness.High Prevalence drives Precision up and Recall down according to Markedness.
  • Interpretation: Recall reflects Bias plus discounted Informedness, while Precision reflects Prevalence plus discounted Markedness; their complements weight unbiased performance upward, including in F1.The passage specifies the usual conditions Prevalence << ½ and Bias << ½, with Inverse Prevalence >> ½ and Inverse Bias >> ½.
  • Duality: The dual problem interchanges Precision and Recall, Prevalence and Bias, and Markedness and Informedness.This interchange reverses which condition is the predictor and which is the predicted condition.

110 Monte Carlo simulations with 11 stepped

The simulations compare Informedness, Markedness and Correlation with traditional measures, showing that the former measures characterize prediction–condition information more precisely. Kappa fits these measures relatively well but remains biased and inferior to Correlation as an independent agreement measure.

  • Informedness precisely characterizes how often a model informs the condition, while Markedness characterizes how often the condition informs the predictor.
  • Correlation characterizes how often the condition and predictor inform or mark each other in the dichotomous case.
  • Kappa is more meaningful than Recall, Precision and Accuracy after debiasing, but remains nonlinear, handles error poorly and retains bias.
  • Kappa fits Correlation and Informedness relatively well, whereas Rank Weighted Average and the Geometric and Harmonic means fit poorly.
  • The Evenness weighted determinant has a perfect fit, while its determinant and geometric-mean components are also displayed separately.

Significance and Information Gain

Significance testing asks whether contingency-table effects depart from chance, while comparing tests and measures such as Informedness, Markedness, Correlation, χ2, G2, and Fisher’s exact test. The section shows that significance depends on sample size, effect strength, prevalence, bias, and test assumptions.

  • Significance and Information Gain: Significance requires comparing observed contingency-table values with expectations under chance, including expectations implied by marginal counts, prevalence, and label bias.Model training may also determine whether prevalence, bias, or cost levels are appropriate for evaluation.
  • Significance and Information Gain: Fisher’s exact test evaluates the entire contingency table without distributional constraints, whereas Pearson’s χ2 and likelihood-based G2 tests rely on approximations and may focus on predicted positives.For a binary table with known margins, the relevant degree of freedom is r=1.
  • Significance and Information Gain: G2 measures total information gain, equals N times average information gain in nats, and is twice as sensitive as χ2 for small effects.The paper notes that G2 is preferred when N and cell values are small because χ2 is unreliable in that setting.
  • Significance and Information Gain: χ2 and G2 significance for Informedness increase with N, Bookmaker^2, EvennessR, and the number of Predicted Negatives, while positive bias reduces significance.Greater Informedness requires less data for significance, whereas stronger positive bias requires more.
  • Significance and Information Gain: Full-table significance links correlation to explained deviance discounted by Global Evenness, so uneven prevalence or bias requires more data, while maximum EvennessG is 0.25.The full approximate χ2 also normalizes to Matthews/Pearson (BMG=Phi) Correlation, and full-table estimates can be more liberal than individual estimates.

Confidence Intervals and Deviations

The section develops confidence estimation for Bookmaker Informedness using normalized deviations from chance and full hypotheses, while rejecting fixed-margin chance assumptions and noting limits from small samples or cells.

  • Confidence estimation: Confidence estimation for Bookmaker and ROC statistics should use normalized isocost deviations rather than fixed-margin chance cases.The approach normalizes deviations to [0,1] and treats all points on an isocost line as equivalent chance cases.
  • Confidence estimation: The standard confidence interval uses SE =√[SSE/(N∙(N-1))] =√[sse/(N-1)] with multiplier 1.96 for 95% confidence under approximate normality.A multiplier around X=2 is usual when the central limit theorem applies.
  • Deviations: For testing the null B=0, √sseB0=|1-B|=1, whereas deviation from the full hypothesis uses √sseB2=|B|=0 without measurement error or 1 conservatively with full-range measurement error.These definitions produce error bars around the null and full hypotheses under different measurement-error assumptions.
  • Deviations: The recommended self-weighted arithmetic correction is √sseB1=1-2|B|+2B2; alternative geometric and harmonic means are symmetric but differ at B=0 and B=±1.The weighted arithmetic mean has a minimum of 0.5 at B=±0.5, while the unweighted geometric and harmonic means reach 0 at both endpoints.
  • Limitations: Traditional SSE error bars and chi-squared significance are unreliable when normality or central-limit conditions fail, especially for N<12 or cell-count<5.They are also deemed inappropriate for probabilistic measures of association or error.

SIMPLE EXAMPLES

The examples show that Accuracy, Kappa, and other measures can behave misleadingly because of Prevalence and Bias, while Informedness, Markedness, Correlation, and significance provide alternative perspectives. At N=100, conventional measures can rise even as Bookmaker Informedness falls, and Fisher’s Exact Test can disagree with χ2-based significance assessments.

  • Informedness and related measures: Bookmaker Informedness represents the probability of an informed decision and is identical to DeltaP' and WRAcc, with a close relationship to ROC AUC.A system informed with probability B has Bookmaker Informedness (DeltaP') of B and Recall of B·(1−Prev) + Bias.
  • Dependence on distribution: Accuracy depends on Prevalence and Bias, while Kappa also suffers from non-linearity caused by its complex denominator.These dependencies complicate interpretation of the measures across different distributions.
  • Contrasting evaluation measures: At N=100, all other measures rise in the Table 2 examples, some considerably, while Bookmaker Informedness falls.The examples illustrate that apparent improvement under common measures can oppose improvement captured by Informedness.
  • Significance testing: At N=100, neither contingency is significant at the 0.05 level under the distributional and approximative assumptions for χ2.The passage attributes this to low Informedness, Markedness, and Correlation, while doubling system performance would suffice for significance under the specified Evenness.
  • Significance testing: Fisher Exact Test finds significance at the 0.05 level for both Table 2 examples, unlike the χ2 assessment.This difference reflects Fisher’s hypergeometric assumption and its requirement that Bias and Prevalence be fixed in advance.

PRACTICAL CONSIDERATIONS · THE GENERAL CASE · Generalization of Association

The section specifies when Correlation, Informedness, Markedness, Recall, and Precision are appropriate, then generalizes Informedness, Markedness, and Correlation from binary to multi-class settings. It emphasizes normalization, fully characterized test sets, marginal assumptions, and the independence of the dual measures.

  • PRACTICAL CONSIDERATIONS: Correlation is preferred to Kappa when comparing systems without a priori preference because it correctly normalizes for their respective Biases.
  • PRACTICAL CONSIDERATIONS: With an unimpeachable Gold Standard, normalizing for Prevalence or Evenness gives Informedness, and optimizing Informedness is equivalent to optimizing dt.
  • PRACTICAL CONSIDERATIONS: Recall and Informedness assess effectiveness relative to conditions, whereas Precision and Markedness assess effectiveness relative to predictions.
  • PRACTICAL CONSIDERATIONS: Incompletely characterized test sets lack a fully specified contingency matrix, so evaluation requires appropriately sized, fully labelled sets for applying Informedness and Markedness.
  • THE GENERAL CASE: The binary analysis is extended by assuming K non-empty real and predicted classes, since Precision and Recall are ill-defined when predicted labels or class members are absent.
  • Generalization of Association: For multi-class labels, Bookmaker Informedness is derived as a pointwise contingency-cell average, while each label is reduced to a binary Positive-versus-Negative dichotomy.
  • Generalization of Association: Multi-class Correlation is proposed as the Geometric Mean of Informedness and Markedness, with its square providing a Coefficient of Determination.
  • Generalization of Association: Informedness can approach 0 while Markedness approaches 1 because, with a fixed determinant numerator, their marginal denominators vary independently.

Generalization of Significance

The section generalizes Significance statistics to K-class contingency tables using prevalence- and bias-weighted summation, while adjusting degrees of freedom for mirrored variables. It argues that conventional full-table χ2, G2, and Cramer’s V can overestimate association and lack the probabilistic interpretation of generalized Informedness and Markedness.

  • Weighted multi-class generalization: Prevalence- and bias-weighted multi-class Significance statistics extend the binary Bookmaker Informedness and Markedness forms to K>2 by probability-weighted summation.The summation is a weighted arithmetic mean targeted to r=K−1 degrees of freedom.
  • Degrees-of-freedom correction: For mirrored condition and prediction variables, r=K−1 degrees of freedom is suggested for beta, instead of the conventional r=(K−1)2 used for alpha.The correction reflects the near-complete association between the mirrored variables.
  • Degrees-of-freedom correction: Equations 47–49 are proposed under r=K−1 degrees of freedom to estimate the probability of the specified alternate hypothesis, HA.The equations are hypothesized to be more accurate for beta under the relevant conditions.
  • Degrees-of-freedom correction: Equations 50–52 apply under r=(K−1)2 degrees of freedom to estimate the null-hypothesis probability, alpha, with only a slight practical difference expected.They sum over the K−1 complements of each class and label before applying prevalence- or bias-weighted sums.
  • Limits of conventional correlation: Cramer’s V from standard full-contingency χ2 and G2 estimates tends to vastly overestimate association relative to Bookmaker Informedness and Markedness.These full-matrix estimates are independent of permutations of predicted labels or real classes, whereas generalized Informedness and Markedness have a strict probabilistic interpretation not shared by traditional correlations outside the 2D case.

Sketch Proof of General Chi-squared Test

The generalized χ2 statistic is constructed by aggregating label-specific dichotomous statistics into an Informedness-based, label-independent form, with analogous Markedness and Correlation forms. The proof defines EvennessR as an arithmetic mean, but assumes constant B across labels; Cramer’s V underestimates association at high Informedness.

  • Generalized χ2 construction: Label-specific χ2+lP statistics are summed across non-target labels to form a (K-1)2 degree of freedom estimate, χ2-lXP.The target label is treated as the positive dichotomous case before aggregation.
  • Generalized χ2 construction: A Bias(l)-weighted sum yields the label-independent statistic χ2XB = K(K-1) · N·B2·EvennessR.The derivation also gives χ2XB = (K-1) · χ2KB.
  • Generalized χ2 construction: The proof establishes the Informedness form, defines EvennessR as the arithmetic mean of dichotomous EvennessR(l) terms, and derives Markedness by a dual argument.The Correlation form is the geometric mean of the Informedness and Markedness forms.
  • Assumptions: The derivation assumes B is constant across all labels, limiting the stated proof’s generality.The passage explicitly notes this assumption before discussing the determinant.
  • Approximation accuracy: Cramer’s V underestimates association at high levels of Informedness but is reasonably accurate at lower levels.This comparison concerns estimates against true Correlation, Markedness, and Informedness.

Generalization of Evenness

The section generalizes Evenness for multiclass settings through the Determinant of Correlation and related Evenness forms, while distinguishing this from its dichotomous relation to Significance. It also cautions that significance-based approximations are unsuitable for estimating association strength.

  • Generalization of Evenness: The product of dichotomous Evenness factors generalizes multiclass Bookmaker Informedness and Markedness, but does not necessarily generalize dichotomous Evenness for Significance.An arithmetic rather than geometric mean emerged in the corresponding sketch proof.
  • Generalization of Evenness: The Determinant of Correlation generalizes dp as the volume of possible deviations from chance and yields an EvennessP+ statistic equal to Informedness when K=2.The normalization is the product of the Prevalences.
  • Generalization of Evenness: The + Evenness form is a squared Geometric Mean and the product of the other two forms, while uneven bias or Prevalence reduces all corresponding Evenness forms.These reductions compensate against lowered measures of association and significance due to reduced determinants.
  • Generalization of Evenness: Cramer’s V is flawed for association strength because it relies on rough assumptions and substantial errors from significance tests.The section recommends starting with a good association measure and using analogous formulas to estimate significance or confidence.

Generalization of Confidence

The confidence framework generalizes to polychotomous settings through Bookmaker Informedness, Markedness, and their geometric-mean correlation measure. Its confidence intervals address null and full hypotheses, while simulations reveal substantial differences among significance estimates and limited discrimination of intermediate Informedness values.

  • Generalization of Confidence: Confidence generalizes directly to the general case using a polychotomous Bookmaker analogy and algorithm.The generalized formulation applies the confidence discussion through Bookmaker Informedness and an analogous polychotomous construction.
  • Generalization of Confidence: Markedness applies directly, while Informedness becomes a prevalence-weighted sum involving squared and absolute errors.The passage contrasts the direct Markedness form with the corresponding generalized Informedness expression.
  • Generalization of Confidence: Equations 63–65 define confidence intervals for Bookmaker Informedness, Markedness, and their geometric-mean correlation measure.The intervals are associated respectively with B, M, and C, and support significance testing of B=0 as well as bounds around theoretical expectations.
  • Generalization of Confidence: Traditional confidence and significance measures omit discretization error and margin distributions, motivating sse estimates that remain non-zero at B=0 and B=1.The weighted arithmetic mean is given as an example of a non-trivial sse function satisfying this condition.
  • Generalization of Confidence: >~95% of Bookmaker estimates fall within the confidence bands, while the intervals nearly meet and distinguish no intermediate B values except marginally at B=0.5.The simulation reports Fisher as marginally significant or better almost everywhere, G2 for B>~0.6, and χ2 for B>~0.8.

EXPLORATION AND FUTURE WORK

Informedness and Markedness are mature performance measures, but significance and confidence methods remain less developed. Future work includes validating new correlation statistics and extending relationships among measures to multiclass settings.

  • Established applications: Informedness has proven useful across disciplines, including studies spanning multiple classes and varied noise and artefact conditions.These studies used the multi-class form of Bookmaker Informedness, where traditional Recall, Precision and Accuracy measures were limited.
  • Validation and extension: The new χ2KB, χ2KM, χ2KBM, χ2XB, χ2XM and χ2XBM statistics remain heuristic and have been tested only in toy situations and Monte Carlo simulations.They show an advantage over traditional χ2 approaches in dichotomous settings, but have not yet been applied to the paper’s multiclass experiments.
  • Significance and confidence: Confidence-interval overlap provides indications of significance and statistical power when comparing systems, but non-overlap should not be overinterpreted because hypothesis probabilities also depend on unknown priors.The paper identifies improved understanding of significance and confidence as an area for further work.
  • Multiclass theory: Further work should examine the relationship between Informedness, Markedness, Evenness and the Determinant of Contingency in the general multiclass case.The determinant’s multidimensional generalization represents parameterizations that are more random than a contingency matrix and its subchance permutations.

MONTE CARLO SIMULATION · CONCLUSIONS · FUTURE WORK

Monte Carlo simulations compare distributional and constraint-handling variants, while the conclusions establish elegant links among chance-correct measures, ROC analysis, correlation, significance, and multiclass evaluation. Future work focuses on optimizing chance-correct measures, with evidence that appropriate optimization can substantially improve accuracy.

  • MONTE CARLO SIMULATION: An alternate simulation approach samples prediction-bias and prevalence margins uniformly, then applies conventional distributions around each cell’s expected value.Normal distributions are appropriate when normality or the central-limit assumption is believed to apply.
  • MONTE CARLO SIMULATION: Monte Carlo simulations use Matlab variants, enforcing strictly positive margins because violating this assumption produces NaNs for many statistics.Uniform margin-free distributions can avoid these NaN problems by relaxing integral or discreteness assumptions.
  • CONCLUSIONS: From contingency matrices, dichotomous and mutually exclusive multiclass statistics provide debiased versions of Recall and Precision.These measures are related to Recall-based ROC analysis and its dual Precision-based method for fully specified classifiers.
  • CONCLUSIONS: Informedness, Markedness, and Matthews Correlation connect directly with significance tests, unbiased significance generalizations, and confidence-interval formulations.The relationships also involve contingency-matrix determinants and geometric areas defined by ROC points and chance lines.
  • CONCLUSIONS: The measures generalize readily to multiclass contingency tables and have been used as primary goodness measures across Artificial Intelligence, Cognitive Science, and Psychology.In Psychology, related dichotomous measures are known as DeltaP and DeltaP’.
  • CONCLUSIONS: The dichotomous measures are conceptually and programmatically simple: Informedness equals Recall plus Inverse Recall minus 1, while significance testing and conservative intervals reduce to simple calculations.The paper notes these techniques are routinely taught and used by students in the authors’ laboratory.
  • CONCLUSIONS: A remaining limitation is that the proposed intuitive relationships among unbiased measures, significance, and confidence lack formal proof of optimal bounds.The paper presents a rough rationale but does not establish definitive formulae for close significance and confidence bounds.
  • FUTURE WORK: Future work develops learning algorithms that optimize chance-correct measures, especially Bookmaker-odds costs, with weighted averaging of dichotomous Informedness by prediction bias.Current applications include neural-network, machine-learning, and boosting optimization, while appropriate optimization has produced major accuracy differences, particularly for boosting.
Loading 2010.16061v1…