PaperlayerView exact revision on arXiv

Source-linked AI summary

MID: Mask-Image Distributional Divergence for Evaluating Medical Image Segmentation

Vincenzo Marcianò, Xiaoming Zhang, Gianluca Guglielmo, Sebastien Ourselin, Michela Antonelli, Maria A. Zuluaga

arXiv:2610.17354v1cs.CVcs.AI

TL;DR

Paired segmentation metrics require ground-truth masks and do not compare the distributional structure of prediction sets. MID compares joint image-mask distributions without sample-wise correspondence. Across corruption tests and datasets, it tracks paired metrics and supports shift, misalignment, and model-ranking analyses.

  • Problem

    Dice and Hausdorff distance require paired ground-truth masks, and sample-wise metrics do not assess whether prediction sets match the ground-truth population distribution.

    Source: §1 · pp. 1–2

  • Method

    MID embeds images and masks jointly with a pretrained medical vision encoder, then compares Gaussian approximations of reference and prediction distributions using squared 2-Wasserstein distance.

    Source: §1 · p. 2

  • Results

    Across 56 organ groups, MID correlates with mean Dice at Spearman ρ = −0.55 and with HD95 at ρ = 0.72, without per-sample ground-truth pairing.

    Source: §1 · p. 2, §4.2 · p. 6

  • Takeaways & Limitations

    The image, mask, and residual correspondence decomposition provides a diagnostic for shifts that alter image-mask dependence while preserving marginal feature distributions.

    Source: §3.4 · p. 4

    MID's Gaussian approximation and anatomy-dependent absolute scale mean anatomy-specific references may be preferable for within-anatomy monitoring.

    Source: §4.2 · p. 6, §5 · p. 9

Abstract

from arXiv · show

Current evaluation protocols for medical image segmentation remain largely anchored in sample-wise comparisons, e.g., Dice and Hausdorff distance, which require each prediction to be paired with a ground-truth mask. Yet two limitations arise in practice: they are inapplicable when predictions are made on unlabeled data; and even when annotations are available, they cannot assess whether a collection of predictions matches the distributional structure of the ground-truth population. We introduce MID (Mask-Image Distributional Divergence), a distribution-level evaluation metric for medical image segmentation. MID builds upon the observation that segmentation quality is a property of the joint image-mask distribution: a mask may be anatomically plausible in isolation yet incorrect for the image it accompanies. MID compares the joint distribution of images and masks between a reference set and a set of predictions, without requiring sample-wise correspondence. This captures both mask realism and image-mask consistency. We evaluate MID across 13 publicly available datasets spanning CT, MRI, and MRA. Under anatomically realistic corruptions across 56 organ groups, MID tracks mean Dice and Hausdorff distance without per-sample ground-truth pairing. MID also detects distributional shifts across modalities and anatomies, identifies image-mask misalignment through controlled shuffling experiments, and produces reliable model rankings across real segmentation models. Official implementation available at github.com/erc-caravel/pytorch-mid

1 Introduction

Sample-wise segmentation metrics require paired ground truth, while unlabeled evaluation also lacks scalable surrogate references. MID instead compares joint image-mask distributions, capturing both mask realism and image-mask consistency without sample-wise correspondence.

1 Introduction

  • MID formulates evaluation as distribution matching over joint image-mask pairs, comparing reference and prediction sets without sample-wise correspondence.It uses a pretrained medical vision encoder and Gaussian approximations of the embedded distributions.Source: §1 · p. 2
  • Joint image-mask evaluation captures failures that image-only or mask-only metrics miss, including realistic masks misaligned with their images.The paper also notes that a general metric should cover volumetric and 2-D modalities without modality-specific pipelines.Source: §1 · p. 2
  • Across 56 organ groups, MID correlates with mean Dice at Spearman ρ = −0.55 without per-sample ground-truth pairing.The comparison uses five signed-distance-transform corruption types at three severity levels; paired HD95 remains more precise when ground truth is available.Source: §1 · p. 2
  • Across thirteen datasets, MID tracks corruption severity, detects distribution shifts and misalignment, and ranks models consistently with paired metrics.These contributions address evaluation settings where annotations are unavailable or sample-wise matching is not possible.Source: §1 · p. 2

2 Related Work

Prior segmentation metrics assess paired masks, while distributional approaches compare collections in feature space. Medical image representation methods also use feature distances, but do not model image-mask correspondence.

2 Related Work

  • Dice, HD95, NSD, and related per-sample metrics require each prediction to be aligned with ground-truth at test time.The cited framework catalogs overlap, surface-distance, and detection metrics; topologypreserving variants are also listed.Source: §2 · p. 2
  • FID [16] compares fitted Gaussian distributions of Inception-v3 image features using the squared Fréchet distance, equivalent to W 2 between the Gaussians [9].KID is described as an unbiased MMD-based alternative.Source: §2 · pp. 2–3
  • Joint image-caption embeddings extend distributional evaluation, but the cited approach is limited to 2-D natural images and does not model segmentation masks or volumetric data.MID instead jointly encodes images and masks and supports 2-D and 3-D inputs.Source: §2 · p. 3
  • Medical vision encoders support feature-space distances for out-of-distribution detection, but these image-only approaches do not model image-mask correspondence.The cited work describes learned medical feature spaces as organizing by anatomy and generalizing across datasets and modalities.Source: §2 · p. 3

3 Method

MID compares joint image-mask feature distributions through Gaussian approximations and a squared 2-Wasserstein distance. The method uses class-specific embeddings, reference-fitted PCA, and a residual term to analyze image-mask correspondence.

3 Method

  • MID compares reference and candidate joint image-mask distributions by projecting their embeddings and computing squared 2-Wasserstein distance between Gaussian approximations.This distribution-level construction does not require sample-by-sample alignment.Source: §3 · p. 3

3.1 Setup and Joint Embedding

  • For each foreground class present, the method pairs an image embedding with a class-specific binary-mask embedding; the image feature can represent 2-D and 3-D samples in one shared space.Each image-mask pair contributes an embedding for every foreground class present in the mask.Source: §3.1 · p. 3

3.2 MID

  • MID pools per-class embeddings from reference and candidate pairs, fits Gaussian approximations, and compares them with squared 2-Wasserstein distance without sample alignment.The paper relates this closed-form comparison to the one used in FID.Source: §3.2 · p. 4

3.3 Projection Design

  • PCA fitted on the reference set reduces embedding dimension to stabilize covariance estimates; the 128 components used retain more than 94% of reference variance.Under a Gaussian assumption, the approximation preserves MID up to discarded variance.Source: §3.3 · p. 4

3.4 Sensitivity Decomposition

  • MIDcorr is a diagnostic residual after accounting for image-only and mask-only shifts; shuffling image-mask pairs can increase it while leaving those marginals approximately unchanged.The residual captures the part of the joint score not explained by image-only or mask-only shifts.Source: §3.4 · p. 4

4 Experiments

The experiments evaluate MID on controlled mask corruptions, cross-dataset and acquisition shifts, metric comparisons, real model predictions, and image-mask correspondence. Across these settings, they test how MID reflects segmentation quality and distributional change without sample-wise pairing.

4.1 Setup

  • The experiments span 13 public CT, MRI, and MRA datasets, using a fixed 10-dataset reference pool and candidate sets from corruptions or model predictions.Comparisons include FID variants, Mahalanobis distance, KID, Dice, and HD95; the main implementation uses MedSigLIP embeddings, incremental PCA with 128 components, and Ledoit-Wolf covariance shrinkage.Source: §4.1 · p. 5

4.2 Response to Controlled Segmentation Corruptions

  • Across 840 corruption-sweep cells, MID correlates with mean Dice at ρ = −0.55 and HD95 at ρ = 0.72, while increasing monotonically with severity.Correlations with Dice are stronger for shape-altering corruptions (ρ = −0.58 to −0.65) than for blob false positives (−0.45) or missing slices (−0.42). MID's scale varies by roughly an order of magnitude across anatomies, so scores should be interpreted against anatomy-specific reference ranges.Source: §4.2 · p. 6, Table 1 · p. 6

4.3 Distributional Shift Detection

  • Against a TotalSegmentator CT train reference, MID is 0.011 on same-distribution held-out CT, 0.278 on TotalSegmentator MRI, and 0.932 and 0.949 on prostate and ACDC cardiac MRI, respectively.In Task05_Prostate with SegResNet-300, MID tracks degradation monotonically across all three artefact types and severity levels, capturing combined image-appearance and predicted-mask shifts.Source: §4.3 · pp. 6–7, Figure 2 · p. 7

4.4 Metric Comparison

  • Joint MID correlates with Dice at ρ = −0.55 and HD95 at ρ = 0.72, exceeding distributional baselines on Dice correlation.Its Dice correlation is 0.08 higher than FID-mask, with overlapping confidence intervals.Source: §4.4 · pp. 7–8

4.5 Real Segmentation Models

  • On IXI MRA, MID ranks nnUNet, A2V, and SPOCKMIP in the same order as Dice and HD95; across CT groups, MID correlates with Dice at ρ = −0.45.ACDC's elevated MID against the global reference reflects cardiac-MRI/reference mismatch as well as segmentation quality.Source: §4.5 · p. 8, Figure 3 · p. 8

4.6 Image-Mask Correspondence

  • With shuffled image-mask pairs, the image and mask terms stay near zero while the residual correspondence term carries the signal.The shuffle preserves both marginal feature distributions but disrupts their dependence; for a CT-to-MRI shift, the image term instead dominates.Source: §4.6 · p. 9, Figure 4 · p. 9

5 Discussion

MID provides a consistent distribution-level quality signal across controlled corruptions, real model outputs, and cross-dataset shifts. Its interpretation is constrained by anatomy-dependent scales and the Gaussian approximation.

5 Discussion

  • MID gives a consistent distribution-level quality signal across controlled corruptions, real model outputs, and cross-dataset shifts.Under controlled corruptions, it correlates with Dice and HD95 at ρ = −0.55 and 0.72 without per-sample ground-truth pairing.Source: §5 · p. 9
  • MID’s graded shift response ranges from 0.011 for same-distribution held-out data to 0.278–0.949 for modality shift.The paper reports that MID is sensitive to shift type as well as magnitude.Source: §5 · p. 9
  • The Gaussian approximation is the main limitation; anatomy-specific references may be preferable for within-anatomy monitoring.Absolute MID values also vary by anatomy and should be interpreted against anatomy-specific reference ranges.Source: §5 · p. 9
  • MID depends on encoder quality, but the framework is encoder-agnostic and can adopt stronger medical vision encoders without changing its definition or estimation procedure.Feature extraction is the computational bottleneck; extracting features takes approximately 11 seconds per volume on one GPU, while PCA fitting and MID computation take under one minute on CPU for the full reference set.Source: §5 · p. 9

6 Conclusion

Across 13 datasets spanning CT, MRI, and MRA, MID tracks geometric metrics under controlled corruptions, detects distributional shifts, and recovers model rankings without per-case annotations at inference time. It compares joint image-mask distributions and offers a diagnostic decomposition for failure-mode analysis.

6 Conclusion

  • MID embeds image-mask pairs jointly, fits Gaussian approximations to reference and candidate distributions, and reports their squared 2-Wasserstein distance.The metric is non-negative, symmetric, and consistent.Source: §6 · p. 9
  • A diagnostic decomposition into image, mask, and residual correspondence terms is available for failure-mode analysis.Source: §6 · p. 9
  • Across 13 datasets spanning CT, MRI, and MRA, MID tracks per-sample geometric metrics under controlled corruptions, detects distribution shifts, and recovers correct model rankings without per-case annotations at inference time.Source: §6 · p. 10

A Dataset and Implementation Details

The appendix documents the datasets, feature processing, controlled corruptions, and distribution-shift experiments. Its additional analyses show that MID should be interpreted against anatomy-specific reference ranges.

A Dataset and Implementation Details

  • Thirteen dataset entries span the experiments; ten form the reference pool, while marked datasets are held out as candidate sets for cross-dataset evaluation.Table 3 reports modality, foreground-structure count, and per-class image-mask feature-representation count; multimodal datasets are listed once per modality.Source: Table 3 · p. 14, A Dataset and Implementation Details · p. 14

B Controlled Corruption Protocol

  • Five SDT-based corruption types are applied at mild, moderate, and severe levels to mimic boundary bias, over-segmentation, false positives, and incomplete volumetric predictions.Figure 5 illustrates the corruption types and uses green for preserved regions, red for false positives, and blue for false negatives.Source: B Controlled Corruption Protocol · p. 14, Figure 5 · p. 15

C Additional Results for Controlled Corruptions

C.1 Clean-Reference Variability

  • In clean-vs-clean comparisons, MID spread reflects natural anatomical variability and finite-sample estimation noise, despite Dice being 1.0 by construction.Figure 6 reports comparisons within each dataset-organ group, with n = 56 per group.Source: §C.1 · p. 15, Figure 6 · p. 16

C.2 Per-Anatomy Reference Index

  • Absolute MID values vary by roughly an order of magnitude across anatomies, supporting interpretation within an anatomy or against its specific reference range.Source: Figure 7 · p. 16, §C.2 · p. 17

D Distributional Shift and ROOD-MRI Details

D.1 Cross-Dataset Shift Setup

  • The cross-dataset setup uses TotalSegmentator CT training cases as reference and compares held-out CT, TotalSegmentator MRI, Task05 Prostate MRI, and ACDC cardiac MRI candidates.The design orders shifts from same dataset and modality to modality shift and combined anatomy-modality shift.Source: §D.1 · p. 17

D.2 ROOD-MRI Prostate Robustness

In the prostate MRI artefact experiment, MID compares corrupted-image and prediction distributions with a clean reference without using candidate ground-truth masks. MID rises with artefact severity and follows Dice and HD95 degradation trends.

D.2 ROOD-MRI Prostate Robustness

  • The experiment applies ghosting, Rician noise, and random motion at three severity levels to the T2 channel, while keeping ADC clean.The data are Task05_Prostate T2-weighted MRI with peripheral-zone and transition-zone labels, and the model is SegResNet trained for 300 epochs on Task05_Prostate [30].Source: §D.2 · p. 17
  • For each artefact-severity condition, candidate distributions pair corrupted T2 images with SegResNet predictions; candidate ground-truth masks are not used to compute MID.The clean reference pairs T2 images with ground-truth masks; Dice and HD95 use paired ground truth only as reference-quality indicators.Source: §D.2 · p. 17
  • MID increases with artefact severity and follows the degradation trends measured by Dice and HD95.This supports monitoring distribution-level segmentation quality when per-case annotations are unavailable.Source: §D.2 · p. 17

E Metric Comparison: Full Correlation Results

MID is the strongest unpaired distributional correlate of paired segmentation metrics in the corruption sweep. Its decomposition separates image and mask marginals from a residual term that reflects image–mask coupling.

E Metric Comparison: Full Correlation Results

  • The comparison evaluates distributional metrics against Dice and HD95 cell-by-cell across 840 corruption-sweep cells, with bootstrap confidence intervals.The decomposition terms are examined separately rather than included as rows in this metric comparison.Source: E Metric Comparison: Full Correlation… · p. 17
  • MID is the strongest unpaired distributional correlator with Dice (ρ = −0.55) and HD95 (ρ = +0.72), although its Dice and HD95 bootstrap CIs overlap those of FID-mask.HD95 correlations have uniformly larger absolute values than Dice correlations across metrics; the authors relate this to boundary-shape errors in the sweep.Source: Table 4 · p. 18, E Metric Comparison: Full Correlation… · p. 18

F.1 Covariance Block Structure

  • The appendix derives the diagnostic decomposition from block covariances of joint image-mask embeddings, with mean and additive trace terms separating linearly by block.The principal matrix square-root term generally does not decompose linearly because it mixes diagonal and off-diagonal blocks.Source: F Image-Mask Correspondence… · p. 18, §F.1 · p. 18

F.2 Image, Mask, and Residual Correspondence Terms

  • MIDcorr is the residual contribution not captured by image-only or mask-only Fréchet distances, and it depends on off-diagonal covariance blocks that quantify image–mask coupling.When the off-diagonal blocks are zero, MID equals the sum of its image and mask marginal terms and MIDcorr is zero.Source: §F.2 · p. 19

F.3 Experimental Scenarios

Pair shuffling tests whether MIDcorr responds to disrupted image-mask dependence, while real-model experiments assess rankings against paired metrics and the effect of the global reference pool. The appendix also examines projection and sample-size estimator choices.

F.3 Experimental Scenarios

  • Pair shuffling preserves image and mask marginals but disrupts their dependence; the experiment supports interpreting MIDcorr as a diagnostic residual for correspondence changes, not a standalone metric.The experiment also tests mask corruption with pairing preserved and a CT-to-MRI shift.Source: §F.3 · p. 19

G Real-Model Ranking Details

  • Using a global reference pool puts candidates from different models, datasets, and anatomies on a common scale, but the predominantly CT reference creates a modality-mismatch offset for ACDC.That offset reflects cardiac-MRI/reference mismatch rather than poor cardiac segmentation quality.Source: G Real-Model Ranking Details · pp. 19–20
  • For IXI MRA vessels, MID ranks nnUNet, A2V, and SPOCKMIP in the same order as Dice and HD95, with nnUNet ranked first.All methods are evaluated on the same anatomy and modality, avoiding cross-modality reference mismatch.Source: G Real-Model Ranking Details · p. 20

H Projection and Sample-Size Analyses

  • The appendix analyzes how PCA projection choice and candidate sample size affect MID estimate stability.The passage introduces these analyses but does not report their outcomes.Source: H Projection and Sample-Size Analyses · p. 20

H.1 Dataset-Specific Projections

Dataset-specific PCA projections produce nearly the same MID values as the universal projection, while candidate sample size affects estimate stability. The feature-space visualizations show modality and anatomy structure in MedSigLIP embeddings.

H.1 Dataset-Specific Projections

  • Organ-specific PCA reduces MID by at most 1.5% relative to universal PCA, and the projection choice has negligible effect on relative rankings in single-organ evaluations.Liver tumour scores exceed the healthy cross-dataset class, whereas pancreas shows the reverse ordering because tumour masks are smaller and more variable.Source: §H.1 · p. 20, Table 5 · p. 20
  • For single-dataset evaluation, target-data PCA can improve within-domain sensitivity; universal PCA is preferable for cross-dataset comparisons because it preserves a common coordinate system.The experiment fits organ-specific PCA with k = 128 on the TotalSegmentator CT training split and compares it with the universal projection.Source: §H.1 · p. 20

H.2 Sample-Size Sensitivity

  • Candidate-set MID stabilizes around ≈0.30 for n ≳250, while its coefficient of variation decreases roughly as n^−1/2.The mean decreases monotonically with sample size, consistent with finite-sample positive bias; comparisons are most reliable with similar candidate sizes or a fixed size.Source: §H.2 · p. 21, Table 6 · p. 21

I.1 PCA Projection Structure

  • Joint MedSigLIP embeddings place CT datasets in a common PCA region and displace MRI datasets along the dominant modality-related component.This modality axis contributes much less to MID in within-modality evaluations than in cross-modality shifts.Source: §I.1 · pp. 21–22, Figure 8 · p. 22

I.2 MedSigLIP Feature Clustering

  • In t-SNE projections, anatomical structures cluster coherently in mask-only and joint embeddings, while MRI separates from CT mainly in the joint space.The appendix presents these visualizations as qualitative evidence about the feature space, not as inputs to MID computation.Source: I Latent Space Geometry · p. 21, Figure 9 · p. 22, §I.2 · p. 22

J Encoder Backbone Ablation

The encoder ablation finds that MedSigLIP has the highest mean per-organ correlation between MID and Dice among five tested backbones. Joint image-mask encoding outperforms mask-only FID across encoders, while domain-specific pretraining does not guarantee stronger correlations.

J Encoder Backbone Ablation

  • MedSigLIP achieves the highest mean per-organ |ρ| = 0.833 [0.742, 0.918] across five encoder backbones.BiomedCLIP and DINOv2-L follow at 0.796; CLIP-L/14 reaches 0.755 and MAISI obtains 0.739.Source: Figure 10 · p. 23, J Encoder Backbone Ablation · p. 23
  • Joint image-mask encoding outperforms mask-only FID for every tested encoder, indicating that the image component contributes information beyond mask geometry alone.Source: J Encoder Backbone Ablation · p. 23
  • Domain-specific encoders do not uniformly outperform natural-image encoders: DINOv2-L matches BiomedCLIP, while MAISI has lower correlation consistent with its VAE-style generative objective.The authors suggest representational capacity may partly compensate for domain gap in this corruption setting.Source: J Encoder Backbone Ablation · p. 23

Related papers

Loading 2610.17354v1…