Source-linked AI summary
Metrics for Multi-Class Classification: an Overview
Margherita Grandini, Enrico Bagli, Giorgio Visani
TL;DR
Multi-class classification requires metrics for evaluating and comparing classifiers across multiple classes, including during parameter analysis. The paper reviews multi-class metrics and their trade-offs, concluding that metric choice determines how class size and chance agreement affect interpretation.
Problem
Classification development needs performance indicators for comparing models and analysing parameter choices across multi-class datasets.
Method
The paper conducts an in-depth analysis of different multi-class classification performance metrics, highlighting their advantages, disadvantages, and uses.
Results
The reviewed metrics differ in how they weight class sizes and account for chance agreement: Accuracy suits balanced data, Balanced Accuracy weights classes equally, weighted Balanced Accuracy follows frequency, Macro-Average methods equalize class influence, and Cohen’s Kappa removes chance agreement for cross-sample comparison.
Takeaways & Limitations
Metric selection should reflect whether evaluation prioritizes overall instance correctness, under-represented classes, class-frequency importance, or comparisons across different samples.
Abstract
from arXiv · showhide
Classification tasks in machine learning involving more than two classes are known by the name of "multi-class classification". Performance indicators are very useful when the aim is to evaluate and compare different classification models or machine learning techniques. Many metrics come in handy to test the ability of a multi-class classifier. Those metrics turn out to be useful at different stage of the development process, e.g. comparing the performance of two different models or analysing the behaviour of the same model by tuning different parameters. In this white paper we review a list of the most promising multi-class metrics, we highlight their advantages and disadvantages and show their possible usages during the development of a classification model.
A WHITE PAPER
The paper lists CRIF S.p.A. as an affiliated organization.
- CRIF S.p.A. is identified in the paper’s affiliation information.
1 Introduction
Multi-class classification predicts among more than two classes, and performance metrics support model evaluation, comparison, and parameter analysis. The paper introduces confusion matrices and binary concepts as foundations for reviewing multi-class metrics.
- Multi-class classification concerns prediction tasks with more than two outcome classes.
- Performance metrics help compare different classification models and analyse one model under different parameter settings.
- Confusion Matrix: A confusion matrix records true and predicted classifications, with predictions in columns and true classes in rows.
- Precision & Recall: Binary Precision measures the proportion of predicted positives that are truly positive, whereas Recall measures the proportion of actual positives correctly identified.
- The paper reviews multi-class metrics, outlining their advantages, disadvantages, and appropriate uses.
2 Accuracy
Accuracy measures overall correct classification and is intuitive, but it gives greater influence to populous classes. It is therefore suited to instance-level priorities but can hide errors in minority classes.
- Accuracy is computed directly from the confusion matrix and measures the proportion of correctly classified instances.
- Because each instance contributes equally, Accuracy gives greater weight to classes containing more units.
- Accuracy is most suitable when maximizing correctly classified individuals matters more than class distribution.
- On imbalanced datasets, Accuracy can hide substantial errors affecting classes with few units.
- Accuracy is intuitive, ranges from 0 to 1, and its complement to 1 is the MisclassificationRate.
3 Balanced Accuracy
Balanced Accuracy averages per-class recalls, giving every class equal weight and exposing performance on under-represented classes. Its weighted variant restores frequency-based class importance while retaining class-specific performance information.
- Balanced Accuracy averages the recalls computed separately for each class.
- When class sizes are similar, Accuracy and Balanced Accuracy tend to converge; their difference is greatest with imbalanced class distributions.
- For the imbalanced Figure 3 setting, Accuracy is 0.689 whereas Balanced Accuracy is 0.615.
- Balanced Accuracy gives equal weight to classes, making poor predictions in under-represented classes more visible than with Accuracy.
- In the example, class a has recall 0.0806 after 57 of 62 elements are misclassified.
- Balanced Accuracy Weighted: Balanced Accuracy Weighted multiplies each class recall by its class-frequency weight and normalizes by the total weight.
- Balanced Accuracy Weighted: Balanced Accuracy Weighted preserves class-specific tracking while giving large and small classes effects proportional to their frequencies.
4 F1-Score
F1-Score assesses classification performance by combining Precision and Recall through their harmonic mean.
- F1-Score aggregates Precision and Recall under the concept of a harmonic mean.
precision · recall
The paper develops Precision-, Recall-, and F1-based metrics for multi-class classification, contrasting class-balanced Macro averaging with unit-weighted Micro averaging and describing Cross-Entropy's probability-based limitation.
- F1-Score combines Precision and Recall, giving equal relative contribution to both through a harmonic mean.The harmonic mean helps identify a trade-off between the two quantities.
- In multi-class classification, Precision and Recall are computed for all classes one by one, incorporating all confusion-matrix entries.
- 80% Precision and 80% Recall yield an 80% F1-Score, whereas 60% Precision and 100% Recall yield 75%.The harmonic mean rewards more similar Precision and Recall values than the arithmetic mean.
- When either Precision or Recall approaches 0, the F1-Score drops sharply because the harmonic mean weights lower values more heavily.
- 4.2.1 Macro F1-Score: Macro-Precision and Macro-Recall average per-class precision and recall, assigning equal weight to every class regardless of population.
- 4.2.1 Macro F1-Score: Macro F1-Score is the harmonic mean of Macro-Precision and Macro-Recall, so small and large classes are equally important.
- 4.3 Micro F1-Score: Micro-averaging pools all units across classes, making Micro-Precision and Micro-Recall equal and causing Micro F1-Score to equal Accuracy.
- 4.4 Cross Entropy: Cross-Entropy uses only the predicted probability assigned to the true class and therefore does not evaluate the classification rule translating probabilities into predicted classes.Two algorithms can receive the same Cross-Entropy despite differing predicted classes when their true-class probabilities match.
5 Independence between two Random Discrete Variables
The paper frames MCC and Cohen’s Kappa through dependence between predicted and true classifications, using confusion-matrix information to assess their relationship.
- MCC uses the Phi-Coefficient, while Cohen’s Kappa relates predicted and true classifications through dependence between random variables.Both metrics measure how much predictions depend on the ground-truth classification.
- Both scores measure how dependent model predictions are on the ground-truth classification.
- Both metrics account for True Negative values in binary classification and may therefore be preferable to F1-Score when TN information matters.
5.1 Mattheus Correlation Coefficient
MCC measures association between predicted and actual classifications from the confusion matrix, extending the Phi-Coefficient idea to multi-class settings. It is balanced and informative for unbalanced predictions, but its lower bound varies with the actual class distribution.
- MCC ranges from −1 to 1: 1 indicates strong positive agreement, 0 no correlation, and negative values inverse association between predictions and true classes.
- MCC applies the Phi-Coefficient concept to predicted and actual classifications treated as discrete random variables.
- In multi-class classification, MCC is defined from a K-class confusion matrix using correct predictions, predicted-class totals, and true-class totals.
- MCC has no fixed minimum value: its lower limit varies between −1 and 0 with the number and actual distribution of classes.
- MCC includes all confusion-matrix entries in its numerator and denominator, making it a generally balanced measure for differently sized classes.
- When all elements are assigned to one class, MCC falls to 0 even though Accuracy is 0.80 and first-class Recall is 1.
- Unbalanced model predictions can make MCC fluctuate widely within [−1, +1] during training.
5.2 Cohen’s Kappa
Cohen’s Kappa measures agreement between predicted and true categorical classifications after removing agreement expected by chance. In multi-class settings it closely coincides with MCC while supporting comparisons across models and datasets.
- Cohen’s Kappa treats predicted and true labels as random categorical variables and measures their concordance through a confusion matrix.
- Marginal columns represent the distribution of predicted values, while marginal rows represent the distribution of true classes.
- K ranges from −1 to +1: 0 denotes chance agreement, 1 perfect agreement, and negative values agreement worse than chance.
- The score subtracts Expected Accuracy from observed agreement, removing the component attributed to chance.
- In multi-class cases, Cohen’s Kappa coincides with MCC apart from a slightly lower denominator, producing slightly higher final scores.
- Kappa can compare models with equal Accuracy but different agreement patterns, including cases where errors are distributed differently across classes.
- Expected Accuracy reflects class count and dataset balance, allowing Kappa to compare model performance across different datasets.
6 Conclusions
The paper reviews multi-class classification metrics, emphasizing how their weighting schemes and underlying information shape model evaluation. It also highlights metric-specific limitations and interpretive differences.
- The paper provides an in-depth analysis of metrics for evaluating classification algorithms on multi-class datasets.
- Accuracy is suitable for balanced datasets because it measures overall correctness without considering class distribution.
- Balanced Accuracy weights every class equally, helping reveal predictive problems in rare and under-represented classes.
- Balanced Accuracy Weighted weights classes according to frequency, so large and small classes affect results proportionally.
- Macro-Average methods give equal weight to classes, shifting the influence of the biggest classes toward the smallest ones.
- Micro F1-Score reduces algebraically to the Accuracy formula, offering a different interpretation of Accuracy.
- Cross Entropy compares predicted and true probability distributions but considers only the correct class probability, which may cause issues when predictions are wrong.
- Matthews Correlation Coefficient and Cohen’s Kappa quantify categorical dependence, with Kappa adjusting for agreement expected by chance.
Funding
The paper acknowledges financial support from CRIF S.p.A. and the Università degli Studi di Bologna.
- Financial support came from CRIF S.p.A. and Università degli Studi di Bologna.