Source-linked AI summary
Common Limitations of Image Processing Metrics: A Picture Story
Annika Reinke, Minu D. Tizabi, Carole H. Sudre, Matthias Eisenmann, Tim Rädsch, Michael Baumgartner, Laura Acion, Michela Antonelli, Tal Arbel, Spyridon Bakas, Peter Bankhead, Arriel Benis, Matthew Blaschko, Florian Buettner, M. Jorge Cardoso, Jianxu Chen, Veronika Cheplygina, Evangelia Christodoulou, Beth Cimini, Gary S. Collins, Sandy Engelhardt, Keyvan Farahani, Luciana Ferrer, Adrian Galdran, Bram van Ginneken, Ben Glocker, Patrick Godau, Robert Haase, Fred Hamprecht, Daniel A. Hashimoto, Doreen Heckmann-Nötzel, Peter Hirsch, Michael M. Hoffman, Merel Huisman, Fabian Isensee, Pierre Jannin, Charles E. Kahn, Dagmar Kainmueller, Bernhard Kainz, Alexandros Karargyris, Alan Karthikesalingam, A. Emre Kavur, Hannes Kenngott, Jens Kleesiek, Andreas Kleppe, Sven Kohler, Florian Kofler, Annette Kopp-Schneider, Thijs Kooi, Michal Kozubek, Anna Kreshuk, Tahsin Kurc, Bennett A. Landman, Geert Litjens, Amin Madani, Klaus Maier-Hein, Anne L. Martel, Peter Mattson, Erik Meijering, Bjoern Menze, David Moher, Karel G. M. Moons, Henning Müller, Brennan Nichyporuk, Felix Nickel, M. Alican Noyan, Jens Petersen, Gorkem Polat, Susanne M. Rafelski, Nasir Rajpoot, Mauricio Reyes, Nicola Rieke, Michael Riegler, Hassan Rivaz, Julio Saez-Rodriguez, Clara I. Sánchez, Julien Schroeter, Anindo Saha, M. Alper Selver, Lalith Sharan, Shravya Shetty, Maarten van Smeden, Bram Stieltjes, Ronald M. Summers, Abdel A. Taha, Aleksei Tiulpin, Sotirios A. Tsaftaris, Ben Van Calster, Gaël Varoquaux, Manuel Wiesenfarth, Ziv R. Yaniv, Paul Jäger, Lena Maier-Hein
TL;DR
Biomedical image-analysis validation often uses performance metrics without adequately accounting for metric properties, dataset properties, or biomedical interests. This living document uses a Delphi process to illustrate metric pitfalls across four image-analysis task categories. It highlights concrete cases in which metric choices can mislead validation, including class imbalance, prevalence differences, class interdependencies, ordinal classes, empty structures, assignment strategies, and aggregated visualization.
Problem
The paper addresses limited practical guidance on choosing validation metrics that reflect metric properties, dataset properties, and biomedical domain interests in biomedical image analysis.
Method
The authors develop a dynamically updated document based on a Delphi process involving an international consortium and organize examples across four biomedical image-analysis problem categories.
Results
The document illustrates that commonly used metrics can produce misleading assessments under conditions including class imbalance, prevalence differences, class interdependencies, ordinal classes, empty target structures, and differing assignment strategies.
Takeaways & Limitations
Metric selection should be reconsidered in relation to the specific image-processing task and its context rather than treated as a routine workflow.
Takeaways & Limitations
Some metrics are undefined or require special handling when reference annotations or predictions contain no target structures.
Abstract
from arXiv · showhide
While the importance of automatic image analysis is continuously increasing, recent meta-research revealed major flaws with respect to algorithm validation. Performance metrics are particularly key for meaningful, objective, and transparent performance assessment and validation of the used automatic algorithms, but relatively little attention has been given to the practical pitfalls when using specific metrics for a given image analysis task. These are typically related to (1) the disregard of inherent metric properties, such as the behaviour in the presence of class imbalance or small target structures, (2) the disregard of inherent data set properties, such as the non-independence of the test cases, and (3) the disregard of the actual biomedical domain interest that the metrics should reflect. This living dynamically document has the purpose to illustrate important limitations of performance metrics commonly applied in the field of image analysis. In this context, it focuses on biomedical image analysis problems that can be phrased as image-level classification, semantic segmentation, instance segmentation, or object detection task. The current version is based on a Delphi process on metrics conducted by an international consortium of image analysis experts from more than 60 institutions worldwide.
1 PURPOSE
The paper addresses pitfalls in validating biomedical image-analysis algorithms by treating image-level classification, semantic segmentation, instance segmentation, and object detection as classification tasks at different scales. It provides a dynamically updated, Delphi-based guide to choosing metrics in a problem- and context-aware manner.
- Purpose: Validation metrics are central to meaningful, objective, and transparent assessment of image-analysis algorithms.Most commonly used metrics compare algorithm predictions with reference annotations that approximate the typically unknown ground truth.
- Contribution: The document illustrates pitfalls and drawbacks of commonly applied metrics to support problem- and context-aware metric selection.Its sections cover foundational metrics, category-metric mismatches, category-specific pitfalls, and cross-topic pitfalls.
- Scope: The paper focuses on image-level classification, semantic segmentation, instance segmentation, and object detection as four biomedical problem categories.These tasks can be interpreted as classification at image, pixel, object, or instance level.
- Illustrative framework: The examples span radiology, cell biology, and surgery while covering classification at image, object, and pixel scales.Figure 1 maps the four task types to their relevant classification scale and predicted class probabilities.
2 FUNDAMENTALS
The paper frames biomedical image-analysis validation as classification at image, object, or pixel level, with metric families differing in inputs and properties. Metric choice requires attention to terminology, class definitions, prevalence, imbalance, and the intended validation goal.
- Task framing: Biomedical image-analysis tasks can be interpreted as classification at image, object, or pixel level, sharing many validation aspects.Image-level classification, object detection, and segmentation correspond respectively to image-, object-, and pixel-level classification perspectives.
- Metric foundations: Most metrics derive directly or indirectly from True Positive, False Negative, False Positive, and True Negative cardinalities in a confusion matrix.Cardinalities may be computed at image, segment, object, or pixel level; thresholds often convert continuous class scores into confusion matrices.
- Metric families: Counting metrics operate on fixed-threshold confusion matrices, while multi-threshold metrics operate on threshold-dependent confusion matrices.The paper also distinguishes distance-based metrics among the major metric families relevant to its problem categories.
- Terminology: Metric names vary across contexts and communities: Sensitivity, TPR, and Recall are synonyms, as are DSC and F1 Score.This terminology variation can complicate interpretation across image-level classification and segmentation literature.
- Class handling: Per-class reporting converts multiclass problems into one-versus-the-rest binary problems, producing different confusion matrices for each positive class.MCC and weighted Cohen’s Kappa have explicit multiclass definitions, whereas several other metrics aggregate rotating binary perspectives.
- Metric limitations: PPV and NPV depend on study-population prevalence, so values cannot be extrapolated between populations without correcting for prevalence.The provided equations express PPV and NPV as functions of Sensitivity, Specificity, and prevalence.
- Metric limitations: Single metrics can mislead on imbalanced data, whereas F1 Score combines PPV and Sensitivity through their harmonic mean and penalizes extreme values.F1 Score is also known as DSC in segmentation and is described as relatively robust against imbalanced data sets.
- Metric limitations: Metric values may change substantially with the choice of positive class, even when metrics are bounded between 0 and 1.The positive-class definition may be straightforward in some applications but arbitrary in others.
3 PITFALLS DUE TO CATEGORY-METRIC MISMATCH
Validation metrics should match both the problem category and the domain-specific goal. The paper illustrates failures when image-level, semantic-segmentation, and object-detection metrics are substituted for one another or when overlap does not represent the biomedical interest.
- General principle: Metrics should reflect the domain-specific validation goal, and the correct problem category should be defined before choosing them.The paper identifies metric use outside the appropriate problem category as a recurring source of validation pitfalls.
- Mismatch semantic segmentation ↔ object detection: Segmentation metrics such as DSC can rank detecting one accurately segmented structure above detecting all three structures in an object-detection task.The paper states that DSC is strongly biased against single objects and is inappropriate for detecting multiple structures.
- Mismatch semantic ↔ instance segmentation: Semantic segmentation can merge touching or overlapping objects, whereas instance segmentation preserves separate object identities and boundaries.If instance segmentation is the intended task, the reference labels must distinguish the individual instances.
- Mismatch image-level classification ↔ object detection: Image-level ROC validation of object detectors does not measure localization performance or account for whether objects are actually hit.The validation is performed per image rather than per object.
- Mismatch image-level classification ↔ object detection: Image-level ROC curves cannot distinguish detecting all objects from detecting one when the largest score is the same.They are invariant to the number of annotated objects under this aggregation scheme.
- Mismatch image-level classification ↔ object detection: Image-level ROC curves also cannot distinguish many false-positive objects from one false positive when their largest score is identical.This follows from aggregating object predictions to an image-level score.
- No matching problem category: When no matching problem category exists, overlap metrics such as DSC may fail to reflect the clinically relevant accuracy of a volume ratio.The paper’s example concerns the percentage of blood volume ejected in each cardiac cycle, where similar DSC scores accompany vastly different volume ratios.
4 PITFALLS RELATED TO IMAGE-LEVEL CLASSIFICATION
Image-level classification metrics can misrepresent performance when class imbalance, prevalence shifts, class interdependencies, ordinal outcomes, or metric-specific assumptions are ignored. Reliable validation therefore requires multiple appropriate metrics, explicit confusion-matrix counts, relevant stratification, and comparison with suitable baselines.
- High class imbalance: Class imbalance can produce high Accuracy or Balanced Accuracy despite poor performance on underrepresented classes.Prediction 2 achieves Accuracy 0.97 while effectively returning the majority vote; AUROC 0.52 and AP 0.04 indicate performance no better than random guessing.
- Metric selection and reporting: Validation should report TP, TN, FP, and FN explicitly and compute multiple metrics against an appropriate random-guessing baseline.Different metrics capture different failure modes, while baseline-aware interpretation helps distinguish informative performance from apparently high scores caused by imbalance.
- Interdependencies between classes: Interdependencies between classes can inflate aggregate performance by making predictions appear accurate only when correlated classes co-occur.A near-perfect Accuracy of 0.94 for one class can conceal poor performance when the associated class is absent.
- Lack of stratification: Available meta-information should be used for stratification because aggregate Accuracy can conceal substantially worse performance for a subgroup.Stratifying by gender in the illustrated example reveals poorer predictions for women than for men.
- Prevalence dependency: Prevalence-dependent metrics are difficult to compare across data sets and can change model rankings when prevalence differs.With the same Sensitivity and Specificity, metrics can vary across prevalences; at prevalences other than 50%, BA and Youden’s Index J may rank predictions differently from MCC and Cohen’s Kappa κ.
- Ordinal classes: Ordinal classification requires metrics that penalize errors according to their distance or clinical severity.Accuracy and MCC treat a severe underestimation and a one-class error alike, whereas weighted metrics such as EC and quadratic-weighted Cohen’s Kappa penalize the more consequential error more heavily.
5 PITFALLS RELATED TO SEGMENTATION
Segmentation metrics can misrepresent performance when structures are small, vary greatly in size, have complex shapes, or contain holes and annotation outliers. Metric choice must reflect whether overlap, boundaries, topology, or the absence of a target structure matters.
- Small size of structures relative to pixel size: Small structures make DSC and IoU highly sensitive to single-pixel differences.The same issue affects clDice for small tubular structures.
- High variability of structure sizes: When structure sizes vary, aggregate DSC can weight large and small structures differently, so separate size-group averages may reveal poorer performance on small structures.ASSD and MASD can also behave differently when a tiny prediction lies near a reference boundary.
- Complex shapes of structures: Overlap metrics can assign identical DSC and IoU values to substantially different shapes, whereas boundary-based metrics detect shape differences.For complex tubular structures, clDice can favor predictions that preserve fine branches and centerlines.
- Reference or prediction without target structures: Empty references or predictions require explicit handling because an undefined metric can be appropriate when both are empty.This issue is relevant when target structures may be absent.
6 PITFALLS RELATED TO OBJECT DETECTION
Object-detection evaluation depends on localization and assignment rules whose mathematical behavior can produce misleading hits, penalties, and rankings. IoU thresholds, bounding-box geometry, matching strategies, and confidence ordering all require task-specific scrutiny.
- Mathematical implications of center-based localization criteria: Center-based criteria can be fooled by oversized detections, equal center distances without overlap, or the greater hit likelihood of large reference objects.For complex tubular shapes, a point-inside-mask criterion may better reflect whether the detection hits the object.
- Mathematical implications of IoU-based localization criteria: IoU-based criteria depend on threshold choice and image dimension, while bounding boxes can grow around highly concave or elongated structures.Very small IoU thresholds can allow large-box behavior, and 3D errors increase with the added dimension.
- Choice of assignment strategies: Different assignment strategies can yield different numbers of true positives, false positives, and false negatives for the same predictions and references.The paper also distinguishes completely missed references from detections that fall outside assignment thresholds.
- Effect of small structures on localization criterion: Boundary IoU produces more consistent scores across small and large structures than Mask IoU, but depends strongly on distance d and can be perfect for an imperfect prediction.This limitation also applies when Boundary IoU is used for segmentation.
- Average Precision: Average Precision can remain unchanged when additional false positives have lower predicted confidence than the last correctly predicted reference.Thus two predictions with different numbers of low-confidence false positives may receive the same AP.
7 PITFALLS RELATED TO ANALYSES AND POST-PROCESSING
Analysis and post-processing choices can conceal distributional, hierarchical, class-specific, aggregation, ranking, and reproducibility problems. Reporting raw values, respecting data structure, handling missing outputs explicitly, and assessing ranking uncertainty are central safeguards.
- Uninformative visualization: Aggregated metric scores can hide multimodal distributions, clusters, or subgroup-specific failures that become visible in violin plots, raw points, and color or shape coding.A single boxplot provides only limited descriptive information.
- Metric aggregation for invalid algorithm output: Ignoring missing values can substantially inflate DSC, while different treatments of missing HD values can produce substantially different results.Fixed-boundary metrics permit worst-case substitution more readily than unbounded distance metrics.
- Hierarchical data aggregation: Because validation data are often hierarchical and non-independent, simple aggregation can overrepresent subjects with more images and hide subject-level differences.Patient-level aggregation can expose disparities concealed by image-level averaging.
- Aggregation in the presence of multiple classes: Overall multi-class averages can conceal very low performance for individual classes, particularly when the background class dominates.Per-class aggregation preserves this information.
- Combination of related metrics: Combining mathematically related metrics such as DSC and IoU adds little ranking information, whereas metrics from different families can change rankings.Metric combinations should therefore be selected for complementary properties.
- Ranking uncertainty: Ranking tables conceal uncertainty because similar algorithm performances can receive the same ordering as clearly separated performances.Rankings are sensitive to aggregation operators, data, and ranking methods.
8 CONCLUSION
Choosing a metric for a specific image-processing task is nontrivial because commonly used reference-based metrics have important practical flaws. The paper raises awareness of these limitations and provides guidance for reconsidering validation workflows.
- Conclusion: The paper illustrates common flaws in frequently used reference-based validation metrics and encourages researchers to reconsider established workflows.Its stated purpose is to support more informed metric use for specific image-processing tasks.
A ACRONYMS
This section defines abbreviations used for image-analysis metrics, evaluation concepts, datasets, and classification outcomes.
- AI denotes artificial intelligence, while COCO denotes Common Objects in Context.
- AP, AUC, and AUROC refer to Average Precision, Area under the Curve, and Area under the Receiver Operating Characteristic curve.
- ASSD and DSC denote Average Symmetric Surface Distance and Dice Similarity Coefficient.
- BA, BM, BS, CE, and ECE denote Balanced Accuracy, Bookmaker Informedness, Brier Score, Calibration Error, and Expected Calibration Error.
- clDice denotes the Centerline Dice Similarity Coefficient, while CI denotes Confidence Interval.
- EC, FN, FP, and FPPI denote Expected Cost, False Negative, False Positive, and False Positives per Image.