PaperlayerView exact revision on arXiv

Source-linked AI summary

Metrics reloaded: Recommendations for image analysis validation

Lena Maier-Hein, Annika Reinke, Patrick Godau, Minu D. Tizabi, Florian Buettner, Evangelia Christodoulou, Ben Glocker, Fabian Isensee, Jens Kleesiek, Michal Kozubek, Mauricio Reyes, Michael A. Riegler, Manuel Wiesenfarth, A. Emre Kavur, Carole H. Sudre, Michael Baumgartner, Matthias Eisenmann, Doreen Heckmann-Nötzel, Tim Rädsch, Laura Acion, Michela Antonelli, Tal Arbel, Spyridon Bakas, Arriel Benis, Matthew Blaschko, M. Jorge Cardoso, Veronika Cheplygina, Beth A. Cimini, Gary S. Collins, Keyvan Farahani, Luciana Ferrer, Adrian Galdran, Bram van Ginneken, Robert Haase, Daniel A. Hashimoto, Michael M. Hoffman, Merel Huisman, Pierre Jannin, Charles E. Kahn, Dagmar Kainmueller, Bernhard Kainz, Alexandros Karargyris, Alan Karthikesalingam, Hannes Kenngott, Florian Kofler, Annette Kopp-Schneider, Anna Kreshuk, Tahsin Kurc, Bennett A. Landman, Geert Litjens, Amin Madani, Klaus Maier-Hein, Anne L. Martel, Peter Mattson, Erik Meijering, Bjoern Menze, Karel G. M. Moons, Henning Müller, Brennan Nichyporuk, Felix Nickel, Jens Petersen, Nasir Rajpoot, Nicola Rieke, Julio Saez-Rodriguez, Clara I. Sánchez, Shravya Shetty, Maarten van Smeden, Ronald M. Summers, Abdel A. Taha, Aleksei Tiulpin, Sotirios A. Tsaftaris, Ben Van Calster, Gaël Varoquaux, Paul F. Jäger

arXiv:2206.01653v8cs.CV

TL;DR

Reliable, objective assessment of biomedical image-analysis algorithms remains largely unexplored, and chosen validation metrics may not reflect domain interests. Metrics Reloaded guides problem-aware metric selection through a problem fingerprint, and its final Delphi recommendations received 93% median agreement across subprocesses.

  • Problem

    Reliable, objective assessment of biomedical image-analysis algorithms remains largely unexplored, although validation metrics are meant to proxy domain interest.

    Source: MAIN · p. 5

  • Method

    Metrics Reloaded combines a multi-stage Delphi process with problem fingerprinting, a structured representation of biomedical problems used to guide metric selection.

    Source: MAIN · p. 7

  • Results

    93% median agreement with the Subprocesses supported the consortium’s consensus recommendation in the final Delphi process.

    Source: DISCUSSION · p. 22

  • Takeaways & Limitations

    Metrics Reloaded provides systematic guidance for choosing validation metrics across biomedical imaging tasks in a problem-aware manner.

    Source: DISCUSSION · p. 24

    The framework focuses on biomedical classification problems with categorical reference data, leaving fuzzy reference data outside its scope.

    Source: DISCUSSION · p. 21

Abstract

from arXiv · show

Increasing evidence shows that flaws in machine learning (ML) algorithm validation are an underestimated global problem. Particularly in automatic biomedical image analysis, chosen performance metrics often do not reflect the domain interest, thus failing to adequately measure scientific progress and hindering translation of ML techniques into practice. To overcome this, our large international expert consortium created Metrics Reloaded, a comprehensive framework guiding researchers in the problem-aware selection of metrics. Following the convergence of ML methodology across application domains, Metrics Reloaded fosters the convergence of validation methodology. The framework was developed in a multi-stage Delphi process and is based on the novel concept of a problem fingerprint - a structured representation of the given problem that captures all aspects that are relevant for metric selection, from the domain interest to the properties of the target structure(s), data set and algorithm output. Based on the problem fingerprint, users are guided through the process of choosing and applying appropriate validation metrics while being made aware of potential pitfalls. Metrics Reloaded targets image analysis problems that can be interpreted as a classification task at image, object or pixel level, namely image-level classification, object detection, semantic segmentation, and instance segmentation tasks. To improve the user experience, we implemented the framework in the Metrics Reloaded online tool, which also provides a point of access to explore weaknesses, strengths and specific recommendations for the most common validation metrics. The broad applicability of our framework across domains is demonstrated by an instantiation for various biological and medical image analysis use cases.

MAIN

The section presents Metrics Reloaded as consensus-built, problem-aware guidance for selecting and applying metrics in biomedical image analysis across different task types. It also addresses application guidance, scope boundaries, and broader validation considerations.

MAIN

  • Metrics that fail to reflect biomedical needs can undermine method validity, clinical translation, and scientific progress [78] [97] [152].Inappropriate metric choice is also identified as a core challenge-design problem [97].Source: MAIN · p. 5

Problem fingerprints encapsulate relevant domain knowledge

  • Problem fingerprints encode the category and domain-, target-, data set-, and algorithm-output properties relevant to metric selection.Items include domain preferences such as boundary versus center importance, target size and shape, class prevalence and annotation uncertainty, hierarchical data structure, and predicted class scores.Source: MAIN · p. 7, Problem fingerprints encapsulate… · p. 12

MAIN

  • The work presents an open online tool to improve users’ experience applying the framework.The framework’s mission is to support reliable tracking of scientific progress and help bridge ML research and biomedical imaging practice.Source: MAIN · p. 7

Metrics Reloaded framework

  • The framework was developed through a multi-stage Delphi process, with five international workshops, nine surveys, expert meetings, and crowdsourced feedback between 2020 and 2022.Identified pitfalls were found to generalize across the four supported problem categories and across domains.Source: MAIN · p. 7, Metrics Reloaded framework · p. 10

Cross-domain approach enables integration of distributed knowledge

  • An international multidisciplinary consortium and broader researcher contributions informed design decisions that treat image classification, detection, and segmentation as related tasks at different scales.The consortium included 73 experts, and 156 researchers contributed overall; the framework addresses all four problem categories and includes category choice in its process.Source: Cross-domain approach enables… · p. 11
  • Metric choice should follow the biomedical domain interest rather than the algorithm’s training loss, and validation should use multiple complementary metrics, preferably from different families.The framework was refined through Delphi processes and community feedback and instantiated across biological and medical use cases; decision guides support choices involving tradeoffs.Source: Cross-domain approach enables… · pp. 11–12

Metrics Reloaded addresses all three types of metric pitfalls

Step 1 - Fingerprinting / Step 2 - Metric Selection

  • The final Delphi consensus (over 75% agreement for all ten components) recommends fingerprint-conditioned selection of reference-based metrics, with custom and non-reference metrics added when needed.The reference-based pool includes metrics that are less familiar in biomedical image analysis, such as Net Benefit and Expected Cost.Source: Step 2 - Metric Selection · pp. 13–14

Image-level Classification (ImLC)

  • For image-level classification, Metrics Reloaded recommends assessing both discrimination and calibration, using multi-class, per-class, and multi-threshold metrics as appropriate.Expected Cost is recommended when class confusions have unequal severity; AUROC is the default multi-threshold metric, while Average Precision may suit high class imbalance.Source: Image-level Classification (ImLC) · pp. 14–15

Semantic segmentation (SemS)

  • For semantic segmentation, compute metrics per image, use overlap metrics by default, and generally complement them with boundary-based metrics.Normalized Surface Distance is the default boundary metric when annotation imprecision should be compensated for; DSC or IoU may be unsuitable for consistently small structures with noisy references.Source: Semantic segmentation (SemS) · pp. 15–16

Object detection (ObD)

  • Object detection requires localization and assignment criteria, and metrics that do not rely on true negatives; when class scores are available, pair counting with a multi-threshold metric.Average Precision is recommended for standardization, while FROC Score favors easy interpretation; aggregation needs special consideration when images contain few or no objects.Source: Object detection (ObD) · pp. 16–17

Instance segmentation (InS)

  • For instance segmentation, assess detection explicitly and optionally evaluate segmentation quality for matched instances on a per-instance basis.Panoptic Quality is an alternative to the Fβ Score because it combines detection performance and segmentation quality for matched instances.Source: Instance segmentation (InS) · p. 17

Step 3 - Metric Application

  • Metric application guidance addresses implementation, aggregation, ranking, interpretation, and reporting pitfalls, with metric-specific cheat sheets and an open-source MONAI implementation.The implementation is available in the MetricsReloaded repository.Source: Step 3 - Metric Application · pp. 18–19

Metrics Reloaded is broadly applicable in biomedical image analysis

  • Across biomedical use cases, shared problem properties produce similar metric recommendations, and image modality can be irrelevant to selection.For semantic segmentation, the examples emphasize properties such as segmenting a single large object relative to the grid size.Source: Metrics Reloaded is broadly applicable… · p. 19, Figure 5 · p. 20

The Metrics Reloaded online tool allows user-friendly metric selection

  • The Metrics Reloaded online tool presents the framework in a user-centric way as a common access point for image-analysis validation.The paper notes that conventional validation standards may reflect historical accretion rather than well-justified choices; the tool was available as a beta with restricted access.Source: The Metrics Reloaded online tool… · p. 19, DISCUSSION · p. 21

DISCUSSION

  • A consortium of more than 70 experts developed Metrics Reloaded as guidelines and tools for problem-aware metric selection, addressing pitfalls identified in a sister publication.The consortium brought together expertise across biomedical image analysis, machine learning, statistics, epidemiology, biology, and medicine.Source: DISCUSSION · p. 21
  • Recommendations target biomedical classification with categorical references; fuzzy references remain out of scope, while application-specific metrics are incorporated when common reference-based measures fail.Calibration metrics were added after community feedback. For instance segmentation where instance matching is infeasible, Rand Index and Variation of Information avoid one-to-one correspondences.Source: DISCUSSION · pp. 21–23
  • Rather than impose black-box rules, Metrics Reloaded uses decision guides to explain tradeoffs, including when EC may suit ordinal data better than WCK and how to handle distinct questions or multilabel outputs.For a single data set, it generates a separate recommendation for each driving question.Source: DISCUSSION · pp. 21–22
  • Framework validation combined consensus-building approaches, social-media feedback, and use-case instantiation; feedback prompted EC recommendations and interpretability assessment in Subprocess S5.The framework was challenged through public feedback and instantiated across biological and medical use cases.Source: DISCUSSION · p. 22
  • 93% median agreement across subprocesses marked the final Delphi consensus, though calibration metrics remained debated and the framework addresses these concerns with a detailed rationale.Some members questioned standalone calibration metrics because calibrated class scores do not necessarily express true posterior probabilities; others favored proper scoring rules such as the Brier Score.Source: DISCUSSION · p. 22
  • Metric choice alone does not establish holistic performance: robustness to distribution shift, reliable uncertainty communication, and nontechnical impacts require assessment beyond predictive accuracy.Robustness and reliability may trade off against predictive performance, while responsible research also considers environmental, ethical, economic, social, and societal aspects.Source: DISCUSSION · p. 23
  • Failure-case analysis [131] and subgroup assessment help detect disparities and bias [52]; clinical translation may also require comparison with conventional care using patient outcomes.Relevant test-set demographics and suitable metric aggregation help reveal subgroup differences that might otherwise be missed.Source: DISCUSSION · pp. 23–24

CODE AVAILABILITY STATEMENT

  • The authors envision a regularly updated resource expanding to regression and reconstruction, with recommendations that may generalize across imaging domains; reference implementations are available in MONAI.The online tool simplifies metric selection, and the framework may be extended with additional domain-specific fingerprint features.Source: DISCUSSION · p. 24, CODE AVAILABILITY STATEMENT · p. 24

ACRONYMS

Together, these materials define terminology and guide problem-aware selection and interpretation of validation metrics across image-level classification, object detection, semantic segmentation, and instance segmentation. They cover the problem fingerprint, metric roles and trade-offs, calibration, decision rules, and aggregation choices in relation to domain interests and data properties.

ACRONYMS

  • The acronym list defines abbreviations for methods, metrics, task categories, and biomedical terms used throughout the framework.Supplementary information includes the metric recommendations and further supporting notes.Source: ACRONYMS · pp. 35–36, SUPPLEMENTARY INFORMATION · p. 47

EXTENDED DATA

  • Extended Data Figs. 1–5 show subprocesses for category mapping, multi-class metrics, per-class counting, multi-threshold metrics, and calibration, with calibration applying to image-level classification.The calibration subprocess also suggests calibration loss [22], calibration slope [143], ECI [160], and O:E ratio [130].Source: Figure 1 · p. 37, Figure 2 · p. 38, Figure 3 · p. 39
  • Extended Data Figs. 6–9 cover overlap- and boundary-based segmentation metrics, localization criteria, and assignment strategies for the specified segmentation and detection tasks.Overlap- and boundary-based metric subprocesses apply to semantic and instance segmentation; localization and assignment subprocesses apply to object detection and instance segmentation.Source: Figure 6 · p. 42, Figure 7 · p. 43, Figure 8 · p. 44

SUPPL. METHODS

  • The supplementary methods organize guidance into problem fingerprinting, metric selection, metric application, use-case recommendations, and terminology, with an overview for four image-analysis task categories.The supplementary contents also list metric recommendations, decision guides, cheat sheets, and use cases.Source: SUPPL. METHODS · p. 47

Delphi process

  • The Delphi process grew from an initial panel of 30 experts to 73 and comprised six stages, five workshops, nine surveys, and a final consensus vote.The panel represented 25 institutions initially and brought expertise across biology, medicine, epidemiology, image analysis, statistics, mathematics, and computer science.Source: Delphi process · p. 48
  • The consortium defined coverage for image-, object-, and pixel-level classification, refined problem fingerprints and metric shortlists, and organized expert work by task category and cross-cutting issues.Cross-topic work addressed aggregation, reporting, implementation, statistical considerations, rankings, and biases; biomedical experts identified scenarios for the recommendations.Source: Delphi process · pp. 48–49
  • Nine framework components reached 0%–7% disagreement in the first consensus round; the revised calibration recommendations received strong support, with only one member disagreeing.The final accelerated Delphi process voted on ten core components; calibration recommendations underwent two revision rounds.Source: Delphi process · p. 50

Reference implementations

  • MONAI provides reference implementations for all Metrics Reloaded metrics, and the web tool guides users through recommendations while displaying information relevant to each step and use case.The tool also provides profiles of metrics in the framework’s pool.Source: Reference implementations · p. 50, Web-based tool · p. 50

1.1 General Instructions

  • For comparative benchmarking, primary metrics determine algorithm rankings, while secondary metrics can report complementary properties or enable comparison across publications.For single-algorithm diagnostic studies, users may instead report as many complementary metrics as desired and ignore the primary–secondary distinction.Source: §1.1 · pp. 52–53
  • Decision-rule choices affect suitable validation metrics; binary tasks use a score cutoff, whereas multi-class tasks require one global rule rather than separate class-wise decisions.Argmax is optimal only with equal severity of class confusions and no requested compensation for class imbalance; it also assumes calibrated scores.Source: §1.1 · p. 53
  • Data-based decision-rule optimization, including recalibration configuration, must use a data split separate from validation data.Multi-threshold metrics can conceal that the optimal global decision rule was not identified and may overestimate practical decision-making performance.Source: §1.1 · p. 53

1.2 Problem Category Mapping

  • When multiple instances of one structure type occur in an image, object detection or instance segmentation is typically preferable to semantic segmentation.Instance-level matching avoids comparing a predicted boundary with the wrong reference instance, which can yield misleading zero distances in semantic segmentation.Source: §1.2 · p. 55

1.3 Generation of the problem fingerprint

  • The problem fingerprint represents metric-relevant properties as user-instantiated binary or categorical items across general, domain-interest, target, data-set, and output families.Fingerprint generation first maps the problem to a category, then instantiates its category-specific items; the tool offers explanations for difficult questions and users may traverse all appropriate branches when uncertain.Source: §1.3 · p. 56
  • Validation with a specific decision rule is optional for all tasks except semantic segmentation, allowing users to encode whether they prioritize actual decisions or continuous predictions.The framework describes target-value, optimization-based, argmax-based, cost-benefit-based, and no-decision-rule strategies; data-driven cutoffs require a dedicated configuration split.Source: §1.3 · pp. 56–57
  • TP means True Positive in the image-level classification fingerprint caption.The caption also defines FP, FN and TN.Source: §1.3 · p. 63
  • FP means False Positive in the image-level classification fingerprint caption.The caption also defines TP, FN and TN.Source: §1.3 · p. 63
  • FN means False Negative in the image-level classification fingerprint caption.The caption also defines TP, FP and TN.Source: §1.3 · p. 63
  • TN means True Negative in the image-level classification fingerprint caption.The caption also defines TP, FP and FN.Source: §1.3 · p. 63
  • Ref means Reference in the image-level classification fingerprint caption.That caption also defines Prediction (Pred) and TP, FP, FN and TN.Source: §1.3 · p. 64
  • Pred means Prediction in the image-level classification fingerprint caption.That caption also defines Reference (Ref) and TP, FP, FN and TN.Source: §1.3 · p. 64
  • For object-detection fingerprints, TP, FP, FN, and TN mean True Positive, False Positive, False Negative, and True Negative, respectively.Binary fingerprint examples mark true properties in blue and counterexamples in red; categorical items appear only in blue.Source: §1.3 · pp. 67–68
  • Instance-segmentation fingerprint captions define TP, FP, FN, and TN; later parts also define Ref as Reference and Pred as Prediction.The captions use the same blue-true, red-counterexample convention for binary items and show categorical items only in blue.Source: §1.3 · pp. 70–72

SUPPL. NOTE 2 STEP 2 - METRIC SELECTION

  • The framework uses problem fingerprints to guide selection of category-specific reference-based metrics and alert users to potential pitfalls, with calibration metrics available as an additional option.Detailed selection explanations are provided for all four problem categories, and formal decision trees and guides are also provided.Source: SUPPL. NOTE 2 STEP 2 - METRIC SELECTION · p. 73

2.1 Metrics Reloaded pool of reference-based metrics

  • The common reference-based metric pool includes counting metrics, multi-threshold metrics, distance-based metrics, and calibration metrics.Counting metrics act on one fixed confusion matrix; multi-threshold metrics use a dynamic confusion matrix; distance-based metrics measure differences between boundaries, volumes, center lines, or shapes.Source: §2.1 · p. 73
  • Most reference-based metrics are directly or indirectly based on confusion-matrix counts: true or false positives or negatives in binary problems.The framework distinguishes counting metrics that operate directly on one fixed confusion matrix from multi-threshold metrics based on a dynamic confusion matrix.Source: §2.1 · p. 73
  • Popular counting metrics are closely related, and the accompanying figure categorizes their relationships, including synonym terms for some metrics.The supplied passage introduces synonyms as one relationship category but does not provide the examples that follow.Source: §2.1 · p. 73
  • Metric relationships include Recall = Sensitivity = TPR = Hit rate, PPV = Precision, and DSC = F1 Score at pixel level.Accuracy = 1 - Error Rate, DSC = (2 * IoU)/(1 + IoU), and DSC is Fβ Score with β = 1.Source: §2.1 · p. 73

2.2 Recommendations for Image-level Classification

  • Confusion-matrix metrics differ in which entries they combine; per-class validation is recommended because a single multi-class score can hide poor performance in individual classes.Multi-class metrics such as Accuracy, MCC and Expected Cost use the full confusion matrix, while per-class metrics such as the Fβ Score validate a class of interest.Source: §2.2 · p. 77
  • If calibration is also being assessed, select calibration metrics: the Brier Score measures discrimination and calibration together, while Expected Calibration Error assesses calibration.Counting and multi-threshold metrics do not establish whether predicted scores reflect true probabilities.Source: §2.2 · pp. 78–80
  • Choose multi-class metrics according to costs and prevalence: use Expected Cost for unequal error costs, and consider Balanced Accuracy when dataset prevalences differ from the target population.Accuracy is recommended only when dataset prevalences reflect the target population and imbalance should not be compensated for; MCC is another general option, and the confusion matrix should also be reported when feasible.Source: §2.2 · pp. 78–79
  • Always select a per-class multi-threshold metric: AUROC is prevalence-independent, whereas AP is prevalence-dependent and can be useful when rare-class performance matters.Multi-threshold metrics assess discrimination across thresholds, avoiding reliance on a potentially arbitrary single decision threshold.Source: §2.2 · pp. 78–79

2.3 Recommendations for Semantic segmentation

  • When same-type instances may be confused or missed, reconsider semantic segmentation as instance segmentation; if semantic segmentation remains, avoid distance-based metrics.Such metrics can compare a structure boundary with the wrong instance’s boundary and are not designed for that situation.Source: §2.3 · p. 82
  • For semantic segmentation, use overlap metrics such as DSC by default and generally complement them with boundary metrics to capture shape or contour errors.Use Normalized Surface Distance when compensating for annotation imprecision; otherwise, choose Boundary IoU for existence-based penalties or Mean Average Surface Distance for distance-based penalties.Source: §2.3 · pp. 82–83

2.3 Recommendations for Semantic segmentation; 2.4 Recommendations for Object detection

  • For semantic segmentation, application-specific metrics can address interests such as object volume, connectivity, shape, or hierarchical labels; handle missing values as the worst possible value when aggregating.For object detection, common reference-based metrics may be complemented by application-specific or non-reference-based metrics, such as runtime or carbon footprint. Multi-class segmentation scores may use macro averaging for equal class importance or weighted averaging for unequal interests or prevalence.Source: §2.3 · pp. 84–85, §2.4 · p. 86

2.4 Recommendations for Object detection

  • Object detection validation requires localization, class agreement, and one-to-one assignment, while the absence of true negatives makes Accuracy, Specificity, and AUROC invalid.The framework therefore treats object matching as a prerequisite to computing object-level classification metrics.Source: §2.4 · pp. 86–87
  • Choose localization criteria and assignment strategies for the domain and outputs, then use FβScore for counting and AP or FROC Score for multi-threshold assessment.Center Distance suits position-focused interests, whereas Box/Approximation IoU suits rough-outline localization; Greedy (by Score) Matching is the default when class scores are available.Source: §2.4 · pp. 87–88
  • Object-level aggregation may be per data set or per image; per-image metrics can reflect hierarchical or image-level interests, but require explicit NaN handling.Exclude NaN cases except when empty predictions match empty references, where PPV and FβScore should be set to 1; stratify by size when variability is high.Source: §2.4 · pp. 90–91

2.5 Recommendations for Instance segmentation

  • Instance segmentation combines detection and boundary delineation while distinguishing same-class instances; PQ combines detection and segmentation quality, and segmentation metrics are computed per matched instance.The recommended detection process selects localization, assignment, and classification metrics, with PQ as an alternative to FβScore.Source: §2.5 · pp. 93, 95, 96
  • Standard reference-based metrics may fail for dense, complex instance shapes; pair-counting or information-theoretic metrics and split/merge errors are alternatives.Split and merge errors can be standalone metrics or components of compound metrics such as F1 Score.Source: §2.5 · p. 95

2.6 Recommendations for Calibration of Predicted Class Scores

  • Calibration guidance remains provisional because the field is evolving; calibration means predicted scores match empirical success rates, not necessarily patient-specific posterior risk.Strong calibration concerns correct posteriors, whereas common research assesses moderate calibration because individual posteriors are unknown.Source: §2.6 · p. 97
  • Canonical, class-wise, and top-label calibration assess progressively weaker score-matching conditions; they are equivalent in binary tasks but may differ in multiclass settings.Canonical calibration assesses full distributions, class-wise calibration assesses each class marginal, and top-label calibration considers only the maximum score.Source: §2.6 · pp. 97–98
  • Calibration Error measures distance from a chosen calibration condition, and its values can differ across conditions in multiclass settings [56].The described relation is CE_canonical ≥ CE_class-wise ≥ CE_top-label; weaker conditions can indicate zero error even when canonical calibration is not met.Source: §2.6 · p. 98
  • Common calibration-error estimators are sample-size biased; density-based ECEKDE tends to have lower bias than binning, while unbiased KCE sacrifices interpretability and requires kernel configuration [56].The passages describe ongoing work on canonical calibration errors that do not use the ℓp norm.Source: §2.6 · pp. 99–100
  • Proper scoring rules such as BS and NLL combine discrimination and calibration, making them useful for comparisons but unable to isolate calibration error; RBS is a robust upper bound on canonical CE [56].That bound can matter in safety-critical settings where calibration error must not be underestimated.Source: §2.6 · p. 100
  • Calibration metric choice follows the goal: use KCE or ECEKDE for recalibration comparisons, CE estimates for classifier calibration comparisons, and proper scoring rules for joint performance rankings.For output interpretation, the framework recommends interpretable CE estimates and, where appropriate, a guaranteed upper bound such as RBS.Source: §2.6 · pp. 102–103
  • Because calibration measures depend on prevalence, U2 and U3 comparisons are generally restricted to one data set, and nonrepresentative prevalence requires revalidation on each new cohort.The same metric may be selected for both comparative assessment and output interpretability.Source: §2.6 · p. 103
  • Calibration recommendations currently focus on image-level classification; for object-level tasks, they apply only to classification outputs, discard background scores, and exclude NLL while retaining BS.The framework notes that its recommendations may theoretically be traversed for object detection and segmentation, despite the limited research basis for those categories.Source: §2.6 · pp. 103–104

DG2.1: Weighted Cohen’s Kappa (WCK) versus Expected Cost (EC)

  • When metric choice is ambiguous, decision guides support preference-aware decisions; for unequal confusion costs, EC is generally preferred to WCK because validation is asymmetric and EC supports decision-rule analysis [51].WCK originated in symmetric comparisons between raters, whereas validation compares predictions against a reference approximating truth.Source: §2.7 · p. 105, DG2.1: Weighted Cohen’s Kappa (WCK)… · pp. 105–106

DG2.2: Balanced Accuracy (BA) versus Expected Cost (EC)

  • Choose EC when target-population prevalences can be specified despite dataset mismatch; choose BA when each class should contribute equally, since BA was designed for that purpose.EC can reflect expected class prevalences directly, but this approach requires those prevalences to be known upfront.Source: DG2.2: Balanced Accuracy (BA) versus… · pp. 106–107

DG2.3: Balanced Accuracy (BA) versus Matthews Correlation Coefficient (MCC) versus normalized EC (ECN)

  • For imbalanced classification when prevalence reflects the target population and compensation is desired, MCC, BA, and ECN provide fixed naive-classifier scores of 0, 1/C, and 1, respectively.The guide contrasts these alternatives with Accuracy, which is subject to all three identified pitfalls in imbalanced settings.Source: DG2.3: Balanced Accuracy (BA) versus… · pp. 108–109
  • BA 0.99, MCC 0.7, and ECN 1 assess the same system as near-perfect, fairly good, and random/naive, respectively, reflecting differences in the effects they compensate for.BA does not compensate for Effect 3, whereas ECN does not compensate for Effect 2.Source: DG2.3: Balanced Accuracy (BA) versus… · p. 109
  • BA averages class Sensitivities to give classes equal contribution under imbalance, whereas ECN only rescales EC; when class interests differ, ECN is the correct choice.MCC also establishes equal class contribution by assessing individual class Sensitivities.Source: DG2.3: Balanced Accuracy (BA) versus… · p. 109
  • BA (0.95) calls the system near-perfect despite PPV of 0.09, while MCC (0.29) rates it better than random but not useful and ECN (9.2) much worse than random.In the screening example, more than 90% of biopsies would be unnecessary false positives.Source: DG2.3: Balanced Accuracy (BA) versus… · p. 110

DG3.2: Net Benefit (NB) versus Expected Cost (EC)

  • For multiclass tasks, the guide recommends skipping NB-versus-EC selection and using metrics selected in Subprocess S2, because one-versus-rest cost-benefit analysis is not intuitive.Both metrics are suited to cost-benefit-based decision rules when task-dependent tradeoffs are known.Source: DG3.2: Net Benefit (NB) versus… · pp. 111–112
  • EC requires explicit costs for false positives and false negatives, while NB uses predicted scores as risks and sets a cutoff from task interest or a benefit-harm tradeoff.Decision curves compare NB across a reasonable range of risk thresholds when a single cutoff cannot be specified confidently.Source: DG3.2: Net Benefit (NB) versus… · p. 112
  • When class prevalences shift, EC can update its cutoff analytically if new prevalences are known, whereas NB requires recalibrated class scores using a labeled validation set.EC may alternatively optimize a data-based cutoff on a dedicated data split.Source: DG3.2: Net Benefit (NB) versus… · pp. 112–113

DG3.3: Positive Likelihood Ratio (LR+) versus Sensitivity

  • LR+ equals TPR divided by 1 minus TNR; it conveys both class Sensitivities in one score for binary tasks, while Sensitivity is generally easier to interpret in multiclass settings.With optimization-based decision rules, LR+ avoids the per-class Sensitivity cutoff of 1 that results from optimizing that Sensitivity alone.Source: DG3.3: Positive Likelihood Ratio (LR+)… · p. 114
  • F𝛽Score is the only compared metric where high scores ensure high PPV; LR+ and Sensitivity are insensitive to PPV, making F𝛽Score useful when imbalance and false positives matter.In the example, Sensitivities were (90%, 99%), LR+ was (90, 90), and F1 Scores were (0.165, 0.995), indicating low PPV for the rare positive class.Source: DG3.3: Positive Likelihood Ratio (LR+)… · p. 115
  • Setting β to 1 weights false-positive and false-negative penalties equally; higher β gives greater weight to false negatives and emphasizes Sensitivity over PPV.The most common choice is β = 1.Source: DG3.3: Positive Likelihood Ratio (LR+)… · p. 116
  • Fβ is a pure detection metric—the harmonic mean of PPV and Sensitivity—while instance segmentation enters through a prior pixel-level localization cutoff.For example, the localization criterion can require Intersection over Union (IoU) > 0.5.Source: DG3.3: Positive Likelihood Ratio (LR+)… · p. 116
  • To assess segmentation, researchers can add a per-instance metric such as DSC per true-positive instance, or use PQ, which combines detection and segmentation quality.PQ replaces each true-positive instance’s unit contribution in F1 with its DSC score, but the combined score makes the two aspects harder to distinguish.Source: DG3.3: Positive Likelihood Ratio (LR+)… · p. 117
  • Two predictions with different detection and segmentation quality can receive the same PQ score, illustrating how a combined score can obscure the source of performance.One prediction has high segmentation but low detection quality; the other has only true-positive instances but low segmentation quality.Source: DG3.3: Positive Likelihood Ratio (LR+)… · p. 117

DG4.1: Area under the Receiver Operating Characteristic Curve (AUROC) versus Average Precision (AP)

  • AUROC is interpretable as a positive-negative score-ranking probability and has a fixed random-classifier reference of 0.5; AP’s random reference is positive-class prevalence and its implementations vary.The variety of AP implementations results from the greater complexity of interpolating the precision-recall curve compared with the ROC curve.Source: DG4.1: Area under the Receiver… · pp. 118–119
  • In the class-imbalanced example, AUROC is 0.95 while AP is 0.32: AUROC neglects PPV, whereas AP reflects the positive class’s low PPV and utility.Because AP focuses on the positive class, its score depends on which class is defined as positive; the rare class should generally be designated positive in imbalanced settings.Source: DG4.1: Area under the Receiver… · pp. 118–119

DG4.2: Average Precision (AP) versus Free-Response Receiver Operating Characteristic (FROC) Score

  • AP is the computer-vision standard, while FROC is often favored clinically for interpretability; global AP ignores added empty images, whereas FROC accounts for dataset size.Adding empty images improves FROC but does not change global AP; AP can instead be configured to aggregate per image.Source: DG4.2: Average Precision (AP) versus… · p. 120
  • FROC can naturally exclude low-confidence predictions, but its results depend on nonstandardized FPPI choices that should be reported and justified.For AP, filtering low-confidence predictions requires a confidence cutoff or a limit on predictions; one suggested FROC default uses FPPI values 1/8, 1/4, 1/2, 1, 2, 4, and 8.Source: DG4.2: Average Precision (AP) versus… · p. 121

Prediction

The section examines how dataset size and metric configuration affect validation scores, and compares calibration, segmentation, and boundary-distance metrics by what they measure, their suitability, and their limitations.

Prediction

  • AP gives D1 and D2 the same score despite different image counts, while FROC accounts for images through FPPI; D2’s lower FPPI yields a higher FROC score.FROC scores also change when the FPPI range used to draw the curve changes.Source: Prediction · pp. 121–122

KCE ECEKDE

  • Because the U2 task requests only relative calibration comparisons, KCE’s lack of interpretability may be acceptable, while ECEKDE could become a viable alternative if debiasing reduces its bias.Both measures are recently proposed and not widely known in the biomedical community.Source: KCE ECEKDE · p. 123

DG5.1: Kernel Calibration Error (KCE) versus Expected Calibration Error Kernel Density Estimate (ECEKDE)

  • For comparing calibration across classifiers on one task, KCE is unbiased but hard to interpret and configure, whereas ECEKDE is interpretable but biased, with proposed debiasing schemes.KCE estimates canonical calibration error using maximum mean discrepancy; ECEKDE uses the ℓ_p norm.Source: KCE ECEKDE · p. 123

BS KCE ECEKDE

  • For recalibration comparisons, BS combines calibration and discrimination, while KCE and ECEKDE isolate calibration and can compare methods regardless of discrimination changes.BS suits accuracy-preserving recalibration when calibration is the focus; it can also be useful when assessing changes in discrimination is desired. KCE and ECEKDE support comparisons of non-accuracy-preserving methods on calibration alone.Source: BS KCE ECEKDE · pp. 124–125

DG5.3: Brier Score (BS) versus Negative Log Likelihood (NLL)

  • Because NLL’s logarithm more heavily penalizes tail probabilities, it can penalize naive systems in class-imbalanced settings, while BS may favor dominant-class predictions; NLL is generally not recommended for calibration assessment.For a true class of 1, predicting 0.001 rather than 0.01 increases BS by ≈2% and NLL by ≈230% for that single entry.Source: BS KCE ECEKDE · p. 125, DG5.3: Brier Score (BS) versus… · p. 126
  • Choose calibration metrics by task interest: CWCE suits unequal class importance, ECE suits decision-focused top-label calibration, and ECEKDE suits interest in all predicted scores.Top-label calibration can be biased toward prevalent classes and depends on binning; for all-score assessment, per-class CWCE may also be reported, while RBS is recommended as an upper bound on calibration error.Source: DG5.3: Brier Score (BS) versus… · pp. 127–128

DG6.1: Dice Similarity Coefficient (DSC) versus Intersection over Union (IoU)

  • DSC is the pixel-level F1 Score and is closely related to IoU, which equals the Jaccard Index; the metrics usually rank aggregated results alike, so combining them adds no value.The medical image community commonly favors DSC, while computer vision commonly favors IoU.Source: DG6.1: Dice Similarity Coefficient… · p. 128

DG7.1: Normalized Surface Distance (NSD) versus Boundary Intersection over Union (IoU)

  • NSD tolerates boundary deviations below a chosen threshold, making it suitable for annotation imprecision; Boundary IoU measures contour overlap within a chosen width without tolerance.Both require a manually set parameter; the NSD threshold can reflect inter-rater variability, while smaller Boundary IoU widths increase sensitivity to contour errors.Source: DG7.1: Normalized Surface Distance… · p. 129

DG7.2: Mean Average Surface Distance (MASD) versus Average Symmetric Surface Distance (ASSD)

  • MASD can be much lower than ASSD when a one-pixel prediction lies close to the reference boundary.The figure describes ASSD and MASD as distance-based metrics, defining d(a,b) as the Euclidean distance between boundary pixels and considering only TPs.Source: DG7.2: Mean Average Surface Distance… · pp. 130–131

DG7.3: Hausdorff Distance (HD) versus X𝑡ℎPercentile Hausdorff Distance (X𝑡ℎPercentile HD)

  • HD uses the maximum shortest boundary distance and is sensitive to spatial outliers, whereas Xth-percentile HD (e.g., HD95) can disregard them.The comparison identifies HD as sensitive to outliers and Xth-percentile HD as compensating for them.Source: DG7.2: Mean Average Surface Distance… · p. 131

2.7.7 Decision guide S8.

For instance segmentation, choose a localization criterion that reflects the target segmentation metric when feasible, then weigh Mask IoU, Boundary IoU and IoR against their distinct benefits and pitfalls.

DG8.1: Mask Intersection over Union (IoU) versus Boundary IoU versus Intersection over Reference (IoR)

  • When feasible, define the localization criterion from the target segmentation metric; metrics without a fixed upper bound, such as HD, make suitable cutoffs challenging.A common object-detection localization criterion is an alternative when a custom criterion is not feasible.Source: DG8.1: Mask Intersection over Union… · p. 133
  • Boundary IoU focuses on boundary correctness and mitigates Mask IoU’s over-penalization of small structures, but its sensitive distance d requires setting and it can score imperfect predictions 1.0.The distance d can be determined based on inter-rater variability; unlike established Mask IoU, Boundary IoU [28] is a recently proposed modification that may need introduction.Source: DG8.1: Mask Intersection over Union… · p. 134
  • For frequent non-split errors with touching reference objects, IoR can reduce IoU’s potential heavy penalization by allowing multiple true-positive matches to one prediction.Penalize merge errors separately or use additional segmentation metrics; IoR can be deceived by large predictions and shares Mask IoU’s behavior on boundaries and small structures.Source: DG8.1: Mask Intersection over Union… · p. 134

Small structure

Boundary IoU penalizes boundary errors and is more invariant to structure size than Mask IoU.

Small structure

  • Boundary IoU specifically penalizes boundary errors and is more invariant to structure size than Mask IoU.The figure compares two Boundary IoU thresholds and shows large structures above small structures.Source: Small structure · p. 135

Overlapping boundary pixels

The guidance shows how overlap, localization thresholds, instance matching, and global decision-threshold application shape evaluation and can expose misleading performance.

Overlapping boundary pixels

  • Boundary IoU can score 1.00 for a prediction with a central hole when distance=2 covers all mask pixels, while Mask IoU gives a lower score.The example shows that Boundary IoU alone can miss a defect that Mask IoU detects.Source: Overlapping boundary pixels · p. 135

DG8.2: Mask Intersection over Union (IoU) > 0 versus Center Distance versus Point inside Mask/Box/Approx

  • For localization-only tasks, loose IoU>0 risks oversized ambiguous predictions, center distance ignores overlap and misfires on tubular structures, while Point inside Mask suits tubular or disconnected objects but has no tunable strictness.Point inside Mask/Box/Approx is suggested when annotations are masks and a center point poorly represents the structure.Source: DG8.2: Mask Intersection over Union… · pp. 137–139

DG8.3: Choose localization threshold

  • Average cutoff-based metrics across IoU 0.5–0.9 in 0.05 steps, but use lower thresholds for existence interest or small, variable, 3D, or uncertain cases and higher thresholds for precise localization or dense objects; Point inside Mask and IoU>0 need none.The threshold guidance does not apply to Point inside Mask/Box/Approx criteria.Source: DG8.3: Choose localization threshold · p. 140

DG9.1: Assignment without predicted class probabilities on instance level

  • When overlap-based matching above 0.5 is unsuitable, greedy localization-score matching is an option, while Hungarian matching can optimistically validate ambiguous outputs without reflecting realistic performance.Matching via overlap > 0.5 inherently avoids assignment ambiguity when overlapping predictions are not possible.Source: DG9.1: Assignment without predicted… · p. 141

SUPPL. NOTE 3 STEP 3 - METRIC APPLICATION

  • For fixed-cutoff score metrics, set one global decision threshold across all classes and metrics to avoid overestimating performance, then aggregate and report raw metric values.The guidance also recommends using metric cheat sheets to find reference implementations and avoid metric-specific pitfalls.Source: SUPPL. NOTE 3 STEP 3 - METRIC… · p. 142

3.1 Metrics Cheat Sheets

The section’s cheat sheets describe selected metrics and their formulas, value ranges, characteristics, and recommendations; they also cover calibration, localization, matching, and instantiated use cases.

Counting metrics

  • For multi-class counting, the framework recommends Balanced Accuracy, Accuracy, Matthews Correlation Coefficient, and Weighted Cohen’s Kappa; Expected Cost is recommended for per-class counting.These recommendations map to Subprocess S2 for multi-class metrics and per-class Expected Cost.Source: Counting metrics · pp. 144, 145, 148, 152, 160
  • For overlap-based evaluation, recommended metrics include clDice, Dice Similarity Coefficient, and Intersection over Union; AUROC, Average Precision, and FROC are recommended for multi-threshold evaluation.Boundary-based recommendations include ASSD, Boundary IoU, Hausdorff Distance, MASD, NSD, and Xth Percentile HD.Source: Counting metrics · pp. 146, 147, 151, 161, 162, 163
  • Per-class counting recommendations include Fβ Score, False Positives per Image, Net Benefit, Negative Predictive Value, Positive Predictive Value, and Sensitivity.FPPI, NPV, PPV, and Sensitivity can be used for target-value optimization, such as evaluating a metric at a specified sensitivity.Source: Counting metrics · pp. 149, 150, 153, 154, 157, 158

3.1.2 Calibration metrics.

  • The calibration cheat sheets recommend BS/Brier Skill Score, CWCE, ECE, ECEKDE, KCE, and NLL as calibration metrics.The section points readers to Suppl. Note 2.6 for an introduction to calibration and related terminology.Source: §3.1.2 · pp. 170–175

3.1.3 Localization criteria.

  • Localization recommendations include Boundary IoU, Center Distance, IoR, Mask/Box/Approx IoU, Mask IoU > 0, and Point inside Mask/Box/Approximation.Source: §3.1.3 · pp. 177–182

3.1.4 Assignment strategies.

  • The recommended assignment strategies are Greedy (by Score) Matching, Greedy (by Localization Criterion) Matching, Optimal (Hungarian) Matching, and Matching via Overlap > 0.5.Source: §3.1.4 · pp. 183–186

SUPPL. NOTE 4 RECOMMENDATIONS FOR SELECTED USE CASES

4.1 Image-level classification

This section demonstrates Metrics Reloaded through seven biomedical image-level classification use cases and their metric recommendations.

4.1 Image-level classification

  • The framework was instantiated for biomedical image-level classification, with recommendations in Fig. SN 4.1 and detailed S2–S5 recommendations in Figs. SN 4.5–SN 4.7.Source: §4.1 · pp. 187–188
  • Use cases span sperm motility, dermoscopic disease [33], autophagy-stage, and ultrasound standard-plane classification [11].Source: §4.1 · p. 187
  • Additional cases cover new-lesion identification in MS MRI [79], mammography breast cancer, and multi-class cardiac disease in MRI.Source: §4.1 · p. 187

4.2 Semantic segmentation

The framework is instantiated for five biomedical semantic-segmentation use cases, with metric recommendations shown for each and further detail provided for selection subprocesses S6 and S7.

4.2 Semantic segmentation

  • The examples include embryo segmentation from microscopy images and liver segmentation in CT images [2] [141].Source: §4.2 · pp. 189–190
  • They also cover lesion labeling in breast whole-slide imaging [3] and cortical structure segmentation from 3D MRI [23].Source: §4.2 · pp. 189–190
  • The fifth use case is aneurysm segmentation in TOF-MRA; recommendations appear in Extended Data Fig. SN 4.2, with S6/S7 details in Figs. SN 4.10–SN 4.11.Source: §4.2 · pp. 189–190

4.3 Object detection

The framework is instantiated for five biomedical object-detection use cases, with metric recommendations summarized in Fig. SN 4.3 and detailed subprocess recommendations in Figs. SN 4.6–SN 4.9.

4.3 Object detection

  • ObD-1 covers cell detection and tracking during autophagy in time-lapse microscopy, and ObD-2 covers MS lesion detection in multi-modal brain MRI [79].Source: §4.3 · pp. 191–192
  • ObD-3 addresses polyp detection in colonoscopy videos at predefined sensitivity 0.95, ObD-4 mitosis detection in histopathology [8], and ObD-5 lung nodule detection in CT.Source: §4.3 · pp. 191–192

4.4 Instance segmentation

The framework is instantiated for four biomedical instance-segmentation problems, with recommendations spanning counting, thresholding, overlap, boundary, localization, and assignment subprocesses.

4.4 Instance segmentation

  • Fig. SN 4.4 presents the recommendations, with detailed subprocess overviews in Figs. SN 4.6–SN 4.11.Source: §4.4 · pp. 193–194
  • The four use cases are neuron segmentation in fruit-fly microscopy, surgical-instrument segmentation in colonoscopy, cell-nuclei segmentation for tracking, and MS-lesion segmentation in brain MRI [79].Source: §4.4 · p. 194
  • Per-class counting (S3) recommendations cover InS-1–4, while multi-threshold selection (S4) covers InS-1, InS-2, and InS-4.Source: §4.4 · pp. 196–197
  • Overlap-based (S6) and boundary-based (S7) metric recommendations cover all four instance-segmentation use cases.Source: §4.4 · pp. 199–200
  • Localization-criteria (S8) and assignment-strategy (S9) recommendations likewise cover all four instance-segmentation use cases.Source: §4.4 · pp. 201–202

5.1 Symbol References

The section identifies the symbols used in the process diagrams and notes that their notation originates from Business Process Model and Notation (BPMN).

5.1 Symbol References

  • The process diagrams use symbols based on Business Process Model and Notation (BPMN).An overview of the symbols is provided in Extended Data Fig. SN 5.1.Source: §5.1 · p. 203

5.2 Expected formats of reference and algorithm output

The metric mapping specifies task-specific reference and algorithm-output formats for image-level classification, semantic segmentation, object detection, and instance segmentation.

5.2 Expected formats of reference and algorithm output

  • Image-level classification uses a class label or multi-label vector per image; predictions use the matching format or continuous class scores when available.Reference labels are y_I ∈ {1, ..., C} or y_I ∈ {0, 1}^C; predicted scores are ŷ_I ∈ [0, 1]^C.Source: §5.2 · p. 204
  • Semantic segmentation requires reference and prediction labels per pixel in the same coordinate system and spacing, with boundary-pixel lists when metrics require them.Labels may be single classes or multi-label vectors; boundaries are supplied separately for each class.Source: §5.2 · p. 204
  • Object detection represents each reference and prediction as a class-location tuple, with an optional predicted class score and location information in a similar format.Locations may be boxes, center points, or radii; background-class scores are typically discarded because there are no background objects, and rough-outline shapes should adequately represent the object, especially for tubular or potentially disconnected objects.Source: §5.2 · p. 204
  • Instance segmentation represents each object with a class and binary per-instance pixel map, optionally adding a predicted class score and providing boundary-pixel lists for each instance.Semantic annotations can be converted by connected-component analysis when instances are non-touching and connected.Source: §5.2 · p. 204

5.3 Acronyms

This section defines abbreviations used for image-analysis tasks, validation metrics, and biomedical imaging and research concepts.

5.3 Acronyms

  • ImLC, ObD, SemS, and InS stand for image-level classification, object detection, semantic segmentation, and instance segmentation, respectively.The list also defines AI as artificial intelligence and ML as machine learning.Source: §5.3 · pp. 205–206
  • The glossary defines validation measures including AP (Average Precision), mAP (Mean Average Precision), DSC (Dice Similarity Coefficient), IoU (Intersection over Union), and AUROC (Area under the Receiver Operating Characteristic Curve).It also lists surface-distance, calibration, classification, and agreement measures, including HD95, ECE, BA, and Cohen’s Kappa.Source: §5.3 · p. 205
  • Biomedical imaging terms include CT (computed tomography), MRI (magnetic resonance imaging), WSI (whole slide imaging), and TOF-MRA (time-of-flight magnetic resonance angiography).The list also defines ROI as Region of Interest and MS as multiple sclerosis.Source: §5.3 · pp. 205–206
  • Other defined terms include EC (Expected Cost), NB (Net Benefit), PHI (Protected Health Information), MONAI (Medical Open Network for Artificial Intelligence), and EQUATOR (Enhancing the QUAlity and Transparency Of health Research).The glossary also covers confusion-matrix rates, proper scoring rules, and data or algorithm terminology.Source: §5.3 · pp. 205–206

5.4 Glossary

The glossary defines terminology for biomedical image-analysis tasks, algorithm outputs, performance metrics, and validation. It distinguishes task types and clarifies how data, predictions, and metric choices are described.

5.4 Glossary

  • Metrics quantify and validate algorithm performance; reference-based metrics compare outputs with a reference, while non-reference metrics assess properties such as runtime, memory use, or carbon footprint.Macro averaging computes a metric per class and averages the scores; micro averaging aggregates a metric score over all classes.Source: §5.4 · pp. 207–208
  • Predicted class scores indicate class membership for images, objects, or pixels; calibration assesses confidence reliability, and a decision rule converts scores into discrete classifications.A target-metric formulation fixes a cutoff to achieve a specified target value, then obtains other metrics from the resulting confusion matrix.Source: §5.4 · pp. 207–208
  • The four problem categories are image-level classification, semantic segmentation, object detection, and instance segmentation, which combine detection and delineation while distinguishing structures of the same class.Image-level classification assigns labels to an image or predefined regions; semantic segmentation labels pixels; object detection localizes and categorizes structures.Source: §5.4 · pp. 207–208
  • The glossary distinguishes training cases with reference annotations from test cases used for performance assessment, and defines validation as checking whether an algorithm does what it was developed to do.It also defines type 1 errors as false positives and type 2 errors as false negatives, and distinguishes validation from assessing clinical added value, termed evaluation.Source: §5.4 · p. 209

Related papers

Loading 2206.01653v8…