Source-linked AI summary
Towards a Guideline for Evaluation Metrics in Medical Image Segmentation
Dominik Müller, Iñaki Soto-Rey, Frank Kramer
TL;DR
Medical image segmentation studies lack reliable, standardized evaluation because metrics may be improperly implemented or used. This paper interprets common binary and multi-class metrics and proposes a guideline for standardized evaluation to improve quality, reproducibility, and comparability.
Problem
Medical image segmentation evaluation lacks reliable assessment because studies may use improper metrics or omit standards for validation.
Method
The paper reviews and interprets common metrics for binary and multi-class segmentation, including class-wise evaluation and reporting recommendations.
Results
The paper proposes standardized evaluation guidance centered on DSC, complementary metrics, visualizations, and individual class computations for multi-class problems.
Takeaways & Limitations
Applying the guideline can support more comparable and reproducible reporting of medical image segmentation performance.
Takeaways & Limitations
Accuracy is strongly discouraged for medical image segmentation because class imbalance and true negatives can produce illegitimately high scores.
Abstract
from arXiv · showhide
In the last decade, research on artificial intelligence has seen rapid growth with deep learning models, especially in the field of medical image segmentation. Various studies demonstrated that these models have powerful prediction capabilities and achieved similar results as clinicians. However, recent studies revealed that the evaluation in image segmentation studies lacks reliable model performance assessment and showed statistical bias by incorrect metric implementation or usage. Thus, this work provides an overview and interpretation guide on the following metrics for medical image segmentation evaluation in binary as well as multi-class problems: Dice similarity coefficient, Jaccard, Sensitivity, Specificity, Rand index, ROC curves, Cohen's Kappa, and Hausdorff distance. As a summary, we propose a guideline for standardized medical image segmentation evaluation to improve evaluation quality, reproducibility, and comparability in the research field.
INTRODUCTION · MAIN TEXT · Evaluation Metrics
Medical image segmentation evaluation is crucial because these models can influence diagnosis and treatment decisions, yet published studies have reported biased assessments from improper metric selection, implementation, usage, and validation. The evaluation-metrics section frames segmentation assessment as measuring both classification accuracy and localization correctness through established scores and their potential issues.
- INTRODUCTION: Deep-learning medical image segmentation models have demonstrated powerful prediction capabilities and results similar to radiologists.These methods are intended to support clinical diagnosis, treatment, risk assessment, and reduce time-consuming inspection.
- INTRODUCTION: Correct and robust evaluation is crucial because medical image segmentation can directly affect diagnosis and treatment decisions.Clinicians seek to integrate these methods into clinical decision-support systems.
- INTRODUCTION: Published medical image segmentation studies have highlighted or cherry-picked improper metrics to obtain scores close to 100%.Reported statistical bias also stems from incorrect metric implementation or usage and missing hold-out-set sampling for reliable validation.
- Evaluation Metrics: Semantic-segmentation evaluation must measure both classification accuracy and localization correctness by scoring similarity between predictions and ground truth.The MIS literature contains a large variety of evaluation metrics, but only a handful have proven appropriate and are used.
- Evaluation Metrics: Nearly all presented metrics except Hausdorff distance use binary-segmentation confusion matrices containing TP, FP, TN, and FN counts.The metric behaviors are illustrated across multiple use cases, and each metric is defined and discussed in terms of possible issues.
- Evaluation Metrics: F-measure metrics calculate overlap from sensitivity and precision while penalizing false positives, making them widespread in computer vision and medical image segmentation.The false-positive penalty is particularly relevant to highly class-imbalanced medical image segmentation datasets.
Sensitivity and Specificity
Sensitivity measures true-positive detection, whereas specificity measures correct identification of true-negative classes, including background. In medical image segmentation, specificity can be misleading because background pixels dominate, while sensitivity remains valid but is less suited than F-score-based metrics for exact comparison.
- Definitions: Sensitivity, or recall/true positive rate, measures true-positive detection, whereas specificity, or true negative rate, measures correct identification of true-negative classes such as background.Both are established standard metrics for performance evaluation, especially in medicine.
- Specificity limitation: Specificity can be an improper segmentation metric when misunderstood because it primarily reflects the model’s ability to detect the background class.Because background occupies a large fraction of pixels relative to the region of interest, specificity values close to 1 are standard.
- Metric comparison: Sensitivity is a valid and popular metric, but it is less sensitive than F-score-based metrics for exact evaluation and comparison of methods.The passage identifies this limitation specifically in medical image segmentation evaluation.
Accuracy / Rand Index
Accuracy, also known as the Rand index or pixel accuracy, measures correct positive and negative predictions relative to all predictions. Its use in medical image segmentation is strongly discouraged because medical images commonly exhibit severe class imbalance.
- Accuracy (Acc), also known as the Rand index or pixel accuracy, counts correct positive and negative predictions against the total number of predictions.
- Accuracy is strongly discouraged in medical image segmentation because medical images are commonly highly class imbalanced.
Receiver Operating Characteristic
The ROC curve evaluates a classifier across discrimination thresholds by plotting true positive rate against false positive rate. It is widely used to compare classifiers and evaluate diagnostic tests, with AUC providing a single-value performance measure.
- Receiver Operating Characteristic: ROC curves visualize classifier diagnostic ability across different discrimination thresholds using true positive rate against false positive rate.
- Receiver Operating Characteristic: ROC curves are established for comparing multiple classifiers and evaluating diagnostic tests and clinical trials in medicine.
- Receiver Operating Characteristic: AUC summarizes ROC performance as a single-value metric and is commonly used to validate machine learning classifiers.The area under the ROC curve was introduced for diagnostic radiology by Hanley and McNeil in 1982.
- Receiver Operating Characteristic: An AUC value of 0.5 can be interpreted as a random classifier.
Cohen’s Kappa
Cohen’s Kappa is a chance-corrected measure of agreement between annotated and predicted classifications. It ranges from -1 to +1, with 0 indicating a random classifier, and can be applied to imbalanced datasets.
- Cohen’s Kappa: Cohen’s Kappa is a chance-corrected agreement measure between annotated and predicted classifications.It was introduced by Cohen in 1960 in psychology.
- Cohen’s Kappa: Cohen’s Kappa ranges from -1 (worst) to +1 (best), while 0 indicates a random classifier.The metric measures agreement attributable to chance, like the AUC score.
- Cohen’s Kappa: Cohen’s Kappa supports evaluation on imbalanced datasets and has therefore gained popularity in machine learning.Its applicability to imbalanced datasets is identified as a key capability.
Sample Visualization · Other Metrics · Multi-Class Evaluation
The section recommends visualizing annotated and predicted segmentations alongside metric-based evaluation, selecting metrics according to the research question, and avoiding uncritical application of binary metrics to multi-class problems. It emphasizes standardized evaluation to improve reproducibility and warns that class imbalance can bias multi-class results.
- Sample Visualization: Comparing annotated and predicted segmentations visually enables robust performance estimation by eye.Visualization can use binary black-and-white class displays or transparent colors over the original image.
- Sample Visualization: Sample visualization is strongly recommended in addition to exact metric-based performance evaluation.
- Other Metrics: Other metrics may be applied depending on the research question and interpretation focus of the study.
- Other Metrics: The work focuses on metrics suited to standardized medical image segmentation evaluation and increased reproducibility.For broader metric theory and overviews, it refers readers to Taha et al.; Nai et al. demonstrated various metrics on a prostate MRI dataset.
- Multi-Class Evaluation: Applying binary metrics to multi-class problems can produce highly biased results, especially with class imbalance.
- Multi-Class Evaluation: Multi-class evaluation requires computing and analyzing the metrThe supplied passage truncates the final metric-related instruction.
Evaluation Guideline
The guideline recommends DSC as the main validation metric, supplemented by AHD, IoU, Sensitivity, and Specificity for interpretation and comparability. It also emphasizes visual and distributional evaluation, class-wise multi-class reporting, avoidance of biased practices, and accessible evaluation materials for reproducibility.
- Core metrics: Use DSC as the main metric for validation and performance interpretation, with AHD supporting interpretation of point-position sensitivity when needed.The guideline also cautions against interpreting high pixel accuracy scores.
- Core metrics: Report IoU, Sensitivity, and Specificity alongside DSC to improve method comparability.
- Evaluation practice: Provide visualizations comparing annotated and predicted segmentations for visual evaluation and to avoid statistical bias.The guideline further advises against cherry-picking high-scoring samples.
- Evaluation practice: Show scoring distributions across the dataset using histograms or box plots.
- Multi-class evaluation: For multi-class problems, compute metrics for each class individually and avoid macro-averaging classes when background inclusion pushes scores upward.
- Reproducibility: Provide evaluation scripts and results through journal data services or third-party services such as GitHub and Zenodo to support reproducibility.
OUTLOOK
The work proposes standardized medical image segmentation evaluation guidelines to improve evaluation quality, reproducibility, and comparability, while developing a standardized metric-computation package.
- OUTLOOK: The guidelines aim to establish a standardized procedure for evaluating medical image segmentation.
- OUTLOOK: The authors hope the guidelines improve evaluation quality, reproducibility, and comparability in future studies.
- OUTLOOK: Because no universal Python package for metric computations exists, the authors are developing one to compute metric scores in a standardized way.
DECLARATIONS
Ethics approval and consent to participate, as well as consent for publication, were not applicable. The analysis provides reproducibility resources, implementation details, datasets, and a work-in-progress metric library.
- Ethics approval and consent to participate were not applicable.
- Consent for publication was not applicable.
- Training lasted up to 1000 epochs, with early stopping between 77 and 173 epochs, using 75 randomly selected images per epoch and two published datasets.
- Complete analysis code, trained models, evaluation results, and metadata are publicly available through GitHub and Zenodo repositories.
- A work-in-progress public Python package, MISeval, is being developed for medical image segmentation metric computation.
Funding · Authors' contributions
The work was funded through the DIFUTURE project by the German Ministry of Education and Research. The authors divided responsibilities across manuscript review, data analysis, drafting, revision, and final approval, while sharing accountability for the work’s integrity.
- Funding: The study was part of the DIFUTURE project.Funding was provided by the German Ministry of Education and Research (BMBF) under grant FKZ01ZZ1804E.
- Funding: The German Ministry of Education and Research funded the work through BMBF grant FKZ01ZZ1804E.The funding agency is identified as the Bundesministerium für Bildung und Forschung.
- Authors' contributions: FK reviewed and corrected the manuscript.ISR also contributed to manuscript reviewing and correction.
- Authors' contributions: ISR contributed to reviewing and correcting the manuscript.These contributions were shared with FK.
- Authors' contributions: DM performed the data analysis and led manuscript drafting and revision.The passage assigns DM responsibility for both analysis and manuscript preparation.
- Authors' contributions: All authors were accountable for the work’s integrity and approved the final manuscript.The authors also read the final manuscript before approval.