Source-linked AI summary
A unifying view for performance measures in multi-class prediction
Giuseppe Jurman, Cesare Furlanello
TL;DR
Multiclass performance measures remain an open research topic despite many proposed metrics. This paper analyzes Confusion Entropy through confusion matrices and shows a strong monotone relation with the multiclass Matthews Correlation Coefficient, while identifying interpretability limitations.
Problem
Defining and comparing meaningful performance measures for multiclass classification remains an open research topic, beyond the well-understood binary setting.
Method
The paper formulates true and predicted class assignments through the confusion matrix and analyzes Confusion Entropy and multiclass Matthews Correlation Coefficient relations.
Results
A strong linear relation exists between Confusion Entropy and a logarithmic function of Matthews Correlation Coefficient regardless of problem dimension, with consistent Matthews Correlation Coefficient behavior in binary classification.
Takeaways & Limitations
Matthews Correlation Coefficient is suggested as a general-purpose evaluating tool, while Confusion Entropy is reserved for applications requiring more refined discrimination.
Takeaways & Limitations
Confusion Entropy does not assign a unique value to the broad range of random-classification cases represented by Matthews Correlation Coefficient = 0 and may behave erratically for binary problems.
Abstract
from arXiv · showhide
In the last few years, many different performance measures have been introduced to overcome the weakness of the most natural metric, the Accuracy. Among them, Matthews Correlation Coefficient has recently gained popularity among researchers not only in machine learning but also in several application fields such as bioinformatics. Nonetheless, further novel functions are being proposed in literature. We show that Confusion Entropy, a recently introduced classifier performance measure for multi-class problems, has a strong (monotone) relation with the multi-class generalization of a classical metric, the Matthews Correlation Coefficient. Computational evidence in support of the claim is provided, together with an outline of the theoretical explanation.
1. Introduction
Comparing classifier performance is central in machine learning, but multi-class performance measures remain an open research topic. This work investigates Confusion Entropy and Matthews Correlation Coefficient, finding a strong, globally monotone relationship supported by experiments and mathematical links.
- Multi-class performance-measure design remains an open research topic despite several proposed functions.
- Binary metrics such as AUC do not have an automatic extension to multi-class classification.
- MCC has become a prominent single-value summary of binary confusion matrices and a preferred metric in the MAQC-II initiative.
- The paper experimentally shows that CEN and MCC are strongly correlated, globally monotone, and locally almost linear.It also outlines mathematical links between the two measures.
2. Confusion Entropy and Matthews Correlation Coefficient
The paper defines MCC and CEN from multiclass confusion matrices and examines their relationship, discrimination, and behavior across structured examples. It also contrasts their binary-case behavior and evaluates the relationship computationally.
- MCC and CEN are defined from the confusion matrix, whose entries count samples by true and predicted class.
- CEN ranges from 0 for perfect classification to 1 for extreme multiclass misclassification, although this upper endpoint fails in the binary case.
- CEN distinguishes confusion matrices that share MCC values, especially when MCC = 0 or sample counts are small.
- For a structured family with increasing misclassification, accuracy decreases toward zero while CEN decreases toward its minimal-entropy case, opposite to MCC and accuracy.
- The authors report a strong, globally monotone and locally almost linear relationship between CEN and a transformed MCC across randomly generated multiclass confusion matrices.
- In the binary case, CEN can exceed 1 when false classifications outnumber correct classifications, making it unusable there, whereas MCC remains interpretable across its stated range.
3. Conclusions
Accuracy, MCC, and CEN each suit different evaluation situations. The paper presents MCC as an interpretable compromise for general-purpose evaluation, while CEN is reserved for cases requiring finer discrimination.
- Accuracy provides a rough classifier-quality estimate but cannot handle unbalanced classes or distinguish misclassification types.
- Confusion Entropy offers extremely high discrimination between similar confusion matrices, but its values are harder to interpret.
- MCC combines reasonable discrimination with an easily interpretable value describing the type of misclassification.
- CEN has a strong linear relation with a logarithmic function of MCC across problem dimensions, while MCC remains consistent in binary classification.
- MCC is suggested as the best off-the-shelf tool for general-purpose tasks, whereas CEN suits topics where refined discrimination is crucial.